Metrics & Time Series
Metrics are numeric measurements captured over time — request rate, error count, latency, CPU usage, queue depth. They're the time-series backbone of observability: cheap to store, fast to query, and ideal for dashboards and alerting because you can aggregate them across thousands of instances and watch trends at a glance. When you ask "is the system healthy right now, and how does that compare to an hour ago?", you're asking a metrics question.
Metrics trade detail for efficiency. Unlike a log line (which records one event with full context) or a trace (which follows one request across services), a metric is an aggregate — "5,000 requests in the last minute, p99 latency 250ms." That aggregation is exactly what makes them scalable to monitor, and exactly why they can't tell you why a specific request failed. Knowing what metrics are good (and bad) at is the foundation of using them well.
TL;DR
- Metrics are numeric measurements over time (time series) — efficient to store and query.
- Core types: counter (only goes up), gauge (up/down), histogram (distribution).
- Track the four golden signals: latency, traffic, errors, saturation.
- Watch cardinality — too many label combinations explode storage and cost.
Quick Example
A metric is a named measurement with labels, sampled over time:
Metric Types
- Counter — a value that only increases (total requests, errors). You graph its rate (requests/sec), not the raw total.
- Gauge — a value that goes up and down (current memory, active connections, queue depth) — a snapshot.
- Histogram — buckets observations into a distribution, enabling percentiles (p50/p95/p99 latency) — essential because averages hide the slow tail.
- Summary — similar to a histogram, computing quantiles client-side.
The Four Golden Signals
Google's SRE practice recommends watching four signals for any user-facing system:
- Latency — how long requests take (track percentiles, separate success from error latency).
- Traffic — demand on the system (requests/sec).
- Errors — rate of failed requests.
- Saturation — how "full" the system is (CPU, memory, queue depth) — the leading indicator of trouble.
Instrument these first; they catch most problems.
Cardinality: The Key Pitfall
Cardinality is the number of unique label combinations a metric has. Each combination is a separate time series to store and index:
⚠️ Never put high-cardinality values (user ids, request ids, emails, full URLs) in metric labels — each unique value spawns a new time series, blowing up storage and cost. High-cardinality detail belongs in logs and traces, not metrics.
Metrics vs Logs vs Traces
The three pillars of observability answer different questions:
Use metrics for dashboards and alerting; drill into logs and traces when a metric tells you something is wrong.
Best Practices
- Instrument the four golden signals first — latency, traffic, errors, saturation.
- Use histograms for latency — percentiles, not averages (the tail is what hurts users).
- Keep label cardinality bounded — no user/request ids in metric labels.
- **Graph counter rates** (per second), not raw totals.
- Pair metrics with logs and traces — metrics detect, the others diagnose.
Common Mistakes
High-cardinality labels
Alerting on averages instead of percentiles
FAQ
What's the difference between a counter, a gauge, and a histogram?
A counter only ever increases — total requests, total errors — and you typically graph its rate (per second) rather than the cumulative value. A gauge can go up or down and represents a current snapshot — memory in use, active connections, queue length. A histogram records the distribution of observed values into buckets, which lets you compute percentiles (p50/p95/p99). The type matters because it determines what the metric can answer: counters for "how often," gauges for "how much right now," histograms for "how is this distributed" (essential for latency).
What are the four golden signals?
From Google's SRE practice, they're the four metrics most worth watching for any user-facing service: latency (how long requests take, tracked as percentiles), traffic (demand, e.g. requests/sec), errors (rate of failures), and saturation (how full the system is — CPU, memory, queue depth, the leading indicator of impending trouble). Instrumenting these four catches the majority of problems with minimal effort, which is why they're the recommended starting point before adding more specialized metrics. They give a high-level health picture that points you toward where to dig deeper.
What is cardinality and why does it matter so much?
Cardinality is the number of unique combinations of label values a metric has — each combination is a distinct time series the system must store and index. High-cardinality labels (user ids, request ids, email addresses, full URLs) can generate millions of series, exploding storage and query costs and potentially overwhelming the metrics backend. It's the single most common way teams break their metrics system. The rule: keep metric labels low-cardinality and bounded (route, status code, region), and put high-cardinality, per-event detail in logs and traces instead.
When should I use metrics versus logs or traces?
Use metrics to answer aggregate, trend questions cheaply — "is the error rate up? how does latency compare to yesterday?" — which makes them ideal for dashboards and alerting. When a metric tells you something is wrong, drill into logs for the detailed record of specific events ("what error did request X hit?") and traces to see where a request spent its time across services. The pattern is: metrics detect the problem, logs and traces diagnose it. They're complementary pillars, not competitors — mature observability uses all three.
Related Topics
- Observability — The hub tying the pillars together
- Distributed Tracing — Following requests across services
- Logging — Detailed event records
- Prometheus — The de facto metrics system
- Alerting — Acting on metric thresholds