Observability & Monitoring

Observability is the ability to understand what's happening inside a system from the outside — to ask new questions about its behavior without shipping new code to answer them. It's what lets you move from "the site is slow" to "the payment service's database query is the bottleneck for EU users" using data the system already emits. As systems grow distributed and complex, observability shifts from a nice-to-have to the difference between debugging in minutes and flying blind for hours.

The discipline rests on three pillars — logs, metrics, and traces — plus the practices that turn signals into action: alerting and SLOs. Each pillar answers a different question, and mature observability uses all three together.

TL;DR

• Observability = understanding a system's internal state from its outputs.
• Three pillars: metrics (aggregate health), logs (event detail), traces (request flow).
• SLOs define "reliable enough"; alerting pages humans on real user impact.
• Monitoring ⊂ observability — monitoring watches known signals; observability lets you ask new questions.

The Three Pillars

Each pillar answers a different question — and you need all three:

Featured Topics

Logging

• Logging — Detailed, structured event records
• Log Aggregation — Centralizing and searching logs at scale

Metrics & Time Series

• Metrics & Time Series — Numeric measurements, golden signals, cardinality
• Prometheus — The de facto metrics system

Distributed Tracing

• Distributed Tracing — Following requests across services
• OpenTelemetry — The vendor-neutral instrumentation standard

APM & Visualization

• Monitoring — The overall practice
• Grafana — Dashboards and visualization
• Datadog — Full-stack APM and observability platform
• Sentry — Error monitoring and performance tracking

Reliability & Response

• Site Reliability Engineering (SRE) — Error budgets, toil, and the reliability discipline
• SLOs, SLIs & Error Budgets — Defining "reliable enough"
• Alerting & On-Call — Notifying humans on real impact

Common Questions

What's the difference between monitoring and observability?

Monitoring is watching known signals for known failure modes — dashboards and alerts on metrics you've decided matter ("is CPU high? is the error rate up?"). Observability is broader: instrumenting the system richly enough (metrics, logs, traces) that you can ask and answer new questions about its behavior — including failures you didn't anticipate — without shipping new code. Monitoring tells you that something is wrong; observability helps you understand why, even for novel problems. Monitoring is a subset of observability.

What are the three pillars of observability?

Metrics, logs, and traces. Metrics are numeric time series — cheap, aggregate, ideal for dashboards and alerting ("is it healthy? what's the trend?"). Logs are detailed records of individual events ("what exactly happened here?"). Traces follow a single request across services ("where did it spend its time?"). They're complementary: metrics detect that something's wrong and trigger alerts, then logs and traces diagnose the specific cause. Mature observability uses all three, increasingly unified through standards like OpenTelemetry.

How do I decide what to alert on?

Alert on symptoms that affect users, not internal causes. Page when error rates climb, latency spikes, or you're burning your error budget fast — not on every high-CPU reading or pod restart, which often don't matter and create noise. Tie alerts to SLOs and burn rate so urgency reflects real reliability risk, make every page actionable (with a runbook), and prune alerts that never lead to action. Noisy alerting causes fatigue and missed incidents, so fewer, higher-signal alerts is the goal. See Alerting.

Where should I start with observability?

Start with the four golden signals — latency, traffic, errors, saturation — as metrics, since they catch most problems cheaply and feed dashboards and alerts. Add structured logging so you can investigate specific events, and distributed tracing once you're running multiple services and need to see cross-service request flow. Define SLOs for your key user journeys to set reliability targets, and build alerting on those rather than on raw thresholds. Instrument with OpenTelemetry to stay vendor-neutral. Build incrementally; you don't need everything at once.

Related Hubs

• DevOps — Infrastructure and the monitoring toolchain
• Backend Development — Instrumenting services
• Performance & Optimization — Acting on what you observe