Thanos: Long-Term, Global Prometheus Metrics

Thanos

A single Prometheus server is simple and reliable, but it has limits: it keeps data on local disk for weeks rather than years, each server only knows about its own targets, and a highly available pair of servers produces duplicate series that dashboards must choose between. Thanos, a CNCF project, extends existing Prometheus deployments with three capabilities: long-term storage in cheap object storage, a global query view across many Prometheus servers and clusters, and deduplication of high-availability pairs — while keeping PromQL and the Prometheus data format.

Thanos is a set of components you assemble. Understanding what each does, and which must run exactly once, is the key to operating it well. See Prometheus Long-Term Storage for how Thanos compares with other options.

TL;DR

Every Prometheus needs unique external labels (for example cluster and replica).
The sidecar uploads Prometheus TSDB blocks to object storage and serves recent data; Receive is the alternative remote-write path.
The store gateway serves historical blocks from object storage.
The querier fans PromQL out to all sources and deduplicates HA replicas by a replica label; the query frontend splits and caches queries.
The compactor compacts blocks, downsamples to 5-minute and 1-hour resolutions, and applies retention — run one per bucket.
The ruler evaluates recording and alerting rules against the global view.

Quick Example

Give each Prometheus unique external labels:

Describe the bucket once:

Run the components:

Point Grafana at the querier as a Prometheus data source; dashboards now see every cluster and years of history.

Core Concepts

Components

Sidecar Versus Receive

External Labels and Deduplication

External labels identify each Prometheus. HA pairs scrape the same targets and differ only in a replica label; the querier's --query.replica-label merges their series so a gap in one replica is filled by the other. Labels that collide between unrelated Prometheus servers make data indistinguishable — keep them unique and stable.

Compaction, Downsampling, and Retention

The compactor merges small blocks into larger ones, creates 5-minute and 1-hour downsampled copies of older data, and deletes data past its retention per resolution. Downsampling does not save storage — it adds blocks — but it makes queries over months or years fast. Keep raw data long enough for detailed investigations, and downsampled data for capacity planning and trends.

Operating Thanos

Run one compactor per bucket: concurrent compactors corrupt or duplicate data. Monitor it closely; a stopped compactor lets blocks pile up and queries slow down.
Cache aggressively: index and chunk caches for store gateways, and results caching in the query frontend, reduce object storage requests.
Shard store gateways by time or label for very large buckets.
Set query limits: maximum samples, series, and concurrency protect queriers from expensive dashboards.
Monitor Thanos itself: upload failures, compaction backlog, store gateway latency, and query errors.

Best Practices

Decide Retention per Resolution

Define how long raw, 5-minute, and 1-hour data are needed based on how teams actually investigate and plan, then set compactor flags accordingly.

Use Recording Rules for Expensive Queries

Pre-aggregate high-cardinality series into recording rules, so long-range dashboards query small series instead of millions.

Keep Prometheus Local and Simple

Let each Prometheus scrape and alert locally for fast, independent alerting; use Thanos for global and historical views.

Control Cardinality at the Source

Thanos stores whatever Prometheus collects. Drop or aggregate high-cardinality labels before they reach long-term storage.

Common Mistakes

Missing or Duplicate External Labels

Two Compactors on One Bucket

Leaving Prometheus Local Compaction On

FAQ

What is Thanos used for?

Extending Prometheus with long-term metric storage in object storage, a single query interface across many Prometheus servers, and deduplication of high-availability Prometheus pairs.

How is Thanos different from Prometheus federation?

Federation scrapes selected series from other Prometheus servers into one server, which limits scale and detail. Thanos queries all servers and object storage directly with full-resolution data and adds long-term retention.

Do I need the sidecar or Receive?

Use the sidecar when you run Prometheus with persistent storage and queriers can reach it. Use Receive when Prometheus instances are ephemeral, numerous, or behind networks where remote write is easier than inbound access.

Does downsampling reduce storage costs?

No. Downsampled blocks are stored in addition to raw data. Storage savings come from shorter retention for raw data; downsampling makes long-range queries fast.

Thanos or Grafana Mimir or VictoriaMetrics?

All provide scalable Prometheus-compatible storage. Thanos builds around existing Prometheus servers and object storage; Mimir is a horizontally scalable remote-write backend; VictoriaMetrics emphasizes efficiency and simple operations. See VictoriaMetrics.