Prometheus: Metrics, PromQL & the Pull Model

Prometheus

Prometheus is the de facto open-source standard for cloud-native metrics monitoring. It pulls (scrapes) numeric metrics from your applications and infrastructure at intervals, stores them as time series, and lets you query and alert on them with PromQL.

It's purpose-built for metrics — numbers over time like request rate, error rate, and latency — not for logs or traces. Paired with Alertmanager (routing alerts) and Grafana (dashboards), it forms the backbone of most open-source observability stacks.

TL;DR

For metrics (numbers over time), not logs or traces.
Pull model: Prometheus scrapes a /metrics endpoint you expose (or an exporter).
Query with PromQL; route alerts via Alertmanager.
Avoid high-cardinality labels (user IDs, request IDs) — they explode storage.

Quick Example

A PromQL query computes the error rate from your request counter over a 5-minute window:

Core Concepts

Scrape — Prometheus periodically pulls metrics from target endpoints.
Time series — a metric name plus a unique set of labels (dimensions like service, route, status).
PromQL — the query language for selecting and aggregating series.
Exporters — adapters that expose metrics for systems you can't instrument directly (node_exporter, postgres_exporter).

Typical architecture

Best Practices

Keep cardinality low — labels should be bounded. Use route templates (/users/:id), status-code classes, and service/env — never raw IDs or full URLs.
Alert on symptoms (SLO burn, error rate), and attach runbooks.
Pick scrape intervals that balance resolution against target load.
Use recording rules to precompute expensive queries used in dashboards.

Common Mistakes

High-cardinality labels

Scraping too aggressively

FAQ

Why does Prometheus pull instead of push?

Pulling lets Prometheus control scrape timing, easily detect when a target is down (the scrape fails), and avoid apps needing to know where to send data. For short-lived jobs that can't be scraped, a Pushgateway bridges the gap, but pull is the default for good reasons.

What is PromQL?

Prometheus's query language for time series. It selects series by metric name and labels, then applies functions and aggregations — e.g. rate(...) for per-second rates and histogram_quantile(...) for percentiles. It's how you build dashboards and alert expressions.

Why are high-cardinality labels a problem?

Each unique label combination is a separate time series stored in memory. Unbounded labels (user IDs, request IDs, raw URLs) create millions of series, exhausting memory and storage. Keep labels bounded and low-cardinality.

Does Prometheus handle logs and traces?

No — it's metrics-only. Pair it with a log system (Loki, ELK) and a tracing system (OpenTelemetry + a backend) for full observability. Grafana can visualize all three together.