Monitoring & Alerting
Monitoring gives you visibility into whether your system is healthy and, when it isn't, what's wrong. Good monitoring is the difference between learning about an outage from your dashboards versus from angry users — and between an alert that tells you to act and noise everyone ignores.
The craft is twofold: instrument the right signals, and alert only on things that are actionable and user-impacting. An alert that fires constantly trains people to ignore alerts, which is worse than no alert at all.
TL;DR
- Watch the four golden signals: latency, traffic, errors, saturation.
- Alert on symptoms (users affected), not internal causes.
- Use the three pillars: metrics, logs, and traces together.
- Define SLOs and error budgets, and attach runbooks to alerts.
Quick Example
Instrument the signals that matter — a request counter and a latency histogram:
Core Concepts
The three pillars
- Metrics — numeric measurements over time (cheap, aggregable).
- Logs — discrete, contextual events (detailed, searchable).
- Traces — a request's path across services (where latency went).
The four golden signals
Alert levels
- Critical → page someone (immediate action).
- Warning → ticket (attention soon).
- Info → dashboard only.
Best Practices
- Alert on symptoms — "error rate > 1% for 5 minutes," not "CPU > 80%."
- Define SLOs (e.g. 99.9% availability) and track the error budget burn rate.
- Attach a runbook link to every alert so responders know what to do.
- Layer dashboards: business metrics on top, service health in the middle, infrastructure below.
Common Mistakes
Alerting on causes, not symptoms
Alert fatigue
FAQ
What are the four golden signals?
Latency, traffic, errors, and saturation — Google SRE's recommended starting set for monitoring any user-facing system. If you watch these four well, you'll catch most problems that actually affect users without drowning in metrics.
Why alert on symptoms instead of causes?
Symptom-based alerts (high error rate, slow responses) fire when users are actually affected and are inherently actionable. Cause-based alerts (high CPU, low disk) fire constantly without necessarily meaning anything is wrong, producing noise that erodes trust in alerting.
What's an SLO and an error budget?
An SLO (Service Level Objective) is a target like "99.9% of requests succeed." The error budget is the allowed failure (0.1% — about 43 minutes/month). Tracking how fast you're "burning" the budget tells you when to slow down and fix reliability versus ship features.
How do metrics, logs, and traces fit together?
Metrics tell you that something is wrong (error rate up); logs tell you what happened (the specific errors); traces tell you where it happened (which service/call was slow). Used together, they take you from alert to root cause.
Related Topics
- Prometheus — Metrics collection
- Grafana — Visualization and dashboards
- OpenTelemetry — Traces and instrumentation
- Log Aggregation — Centralized logs
- Logging — Application logging