Grafana Mimir

Grafana Mimir provides distributed metrics storage and querying for Prometheus-oriented monitoring systems. Start with the metrics you need to retain, their active-series count, the write path, and the queries operators must complete during an incident. Keeping more samples is useful only if the monitoring path remains usable.

Choose and document the Mimir version and architecture. Ingest-storage and classic deployments have different write-path responsibilities, so an architecture diagram from another version is not a recovery procedure.

TL;DR

Quick Example

This complete Prometheus configuration fragment describes one local exporter and an HTTPS remote-write gateway. The .invalid hostname is reserved for illustration and must be replaced only in an authorized pilot. The gateway is expected to authenticate the token and derive the tenant ID; this file supplies no client-selected tenant header.

The secret file, exporter, TLS gateway, tenant mapping, and Mimir service are prerequisites, not resources created by the example. The Mimir authentication guide documents using an authenticating proxy and appropriate tenant injection. Local parsing checks configuration structure only; it does not establish delivery, token permissions, or query isolation.

For a fictional sizing exercise, 20,000 active series scraped every 15 seconds produce about 20,000 / 15 = 1,333 samples per second before accounting for changes in series population or additional streams. This is sample-rate arithmetic, not a storage-size estimate or a capacity claim.

Core Concepts

Writes and reads cross several components

Distributors handle incoming writes, while ingesters serve recent data and move blocks toward object storage; query components combine the needed data paths. The ingester reference distinguishes ingest-storage and classic architectures. In ingest-storage mode, the documented acknowledgment boundary involves committing series to Kafka. Record the actual mode before diagnosing durability or startup lag.

Tenant identifiers are routing information

Mimir requests use X-Scope-OrgID for tenant selection. A supplied header alone does not authenticate the caller. Protect the service behind a suitable identity layer that maps authorized callers to their tenant and prevents arbitrary header spoofing. Verify the read and write paths independently, including any intended federation.

Series growth differs from sample growth

A fixed active-series population with a shorter scrape interval increases sample rate. A new unbounded label value can also create more series. Review both dimensions and the queries that touch them. Request IDs and user IDs are poor default metric labels when each distinct value expands the series set.

Verify the Monitoring Path Under Failure

Define a pilot with a known labeled metric and time window, then query that evidence through the same path operators will use. Interrupt only the test destination and observe the collector's actual buffering and retry limits. A later query gap can come from missing collection, dropped remote writes, retention, or the query scope; preserve enough evidence to distinguish them.

Plan retention through Mimir's documented Compactor controls and the actual object-store lifecycle. Deletion must not remove blocks still needed for the intended history. Review capacity for active data, long-term storage, query concurrency, and recovery; increasing retention is not merely a dashboard setting.

Choose among Mimir and alternatives such as Thanos from the existing monitoring architecture, tenancy needs, query behavior, availability objectives, and operating effort. This article supplies no benchmark or universal winner. A small standalone Prometheus may be simpler when the distributed requirements do not justify another system.

Best Practices

Record workload assumptions

Keep active series, scrape intervals, ingestion peaks, retention, and representative queries with a sizing result. Revisit the estimate after adding labels or onboarding tenants. Samples per second alone do not capture the cost of a broad query.

Monitor rejection and lag

Treat rejected samples and growing remote-write queues as first-class signals. Verify which configured limit was reached and whether data is recoverable. A retry policy cannot recover data that the sender has already discarded.

Keep an independent observation route

Decide how operators will inspect the monitoring service if its own query path is unavailable. Include credentials, logs, health signals, and a bounded recovery runbook. Rehearse that route rather than assuming the usual dashboard will be available.

Common Mistakes

Bad: Trust X-Scope-OrgID from any client.

Correct: Authenticate and authorize the caller and inject the permitted tenant at a controlled boundary.

Bad: Estimate capacity from retained disk alone.

Correct: Include active series, sample rate, write buffering, query concurrency, and failure recovery.

Bad: Use classic-mode assumptions for an ingest-storage deployment.

Correct: Read the version and architecture configuration before interpreting acknowledgment or component dependencies.

FAQ

Is a tenant header an access token?

No. It selects tenant context; a suitable protection layer must authenticate and authorize it.

Does remote-write acknowledgment prove every future query will work?

It establishes the documented write boundary for the selected architecture, not the entire read, retention, or recovery path.

Can the sizing arithmetic determine storage bytes?

No. Compression, labels, data distribution, index, retention, replication, and operating configuration require separate measurement.

What should a Mimir pilot test?

Remote-write delivery from Prometheus, the storage backend, authentication and tenant query isolation, per-tenant limits, retention, and recovery. A local configuration check can't exercise any of these.

Related Topics

References