Log Aggregation
Log aggregation centralizes logs from every service and host into one searchable system. In a distributed application, the logs you need during an incident are scattered across dozens of containers — aggregation pulls them together so you can search, correlate, and alert across the whole system.
The discipline that makes it work is structured logging: emit machine-parseable JSON with consistent fields and a correlation/trace ID, so a central system can index it and you can follow one request across services.
TL;DR
- Centralize logs from all services into one searchable store.
- Emit structured (JSON) logs with a correlation/trace ID.
- Set retention tiers to control cost (hot → warm → cold → archive).
- Build dashboards and alerts on log patterns (e.g. error-rate spikes).
Quick Example
A structured log line carries the context that makes centralized search and correlation possible:
Core Concepts
The pipeline
Common stacks
Best Practices
- Structured logging with consistent field names across services.
- Include a trace/correlation ID so logs link to traces (see OpenTelemetry).
- Tier retention by value: hot (days) → warm (weeks) → cold/archive (compliance).
- Never log secrets or PII; redact at the source. See Logging.
Common Mistakes
Unstructured logs
No retention or cost control
FAQ
ELK or Loki?
ELK (Elasticsearch-based) indexes full log content, giving powerful full-text search at higher storage and operational cost. Loki indexes only labels (like Prometheus) and stores raw logs cheaply, querying by label + grep — lighter and cost-effective, especially with Grafana. Choose by your search needs and budget.
Why is structured logging important for aggregation?
Structured (JSON) logs have consistent, machine-readable fields the aggregator can index and filter on. Free-text logs force fragile regex parsing and make reliable querying, dashboards, and alerts nearly impossible. Structure is what turns a pile of lines into a queryable dataset.
How long should I retain logs?
Tier it by value: keep recent logs hot for fast search (days), older logs warm (weeks), and archive what compliance requires cheaply (cold storage). Full-fidelity indefinite retention is the main driver of runaway logging cost.
How do logs relate to metrics and traces?
They're complementary: metrics tell you something's wrong, traces tell you where, and logs give the detailed context of what happened. Sharing a trace/correlation ID across all three lets you jump from a metric alert to the exact logs for the failing request.
Related Topics
- Logging — Application logging practices
- Monitoring & Alerting — Metrics and the bigger picture
- Grafana — Visualizing logs (Loki)
- Elasticsearch — The store behind ELK
- OpenTelemetry — Correlating logs with traces