Alerting & On-Call: Symptom-Based Alerts & Reducing Noise

Alerting & On-Call

Alerting is how a system tells humans "something needs your attention now." It's the bridge between observability (knowing what's happening) and action (someone fixing it). Done well, alerts catch real problems early and route them to the right person; done poorly, they bury the team in noise until people stop trusting — and reading — them. The hardest part of alerting isn't detecting problems; it's deciding what's worth waking someone up for.

The single most important principle: alert on symptoms, not causes. Page when users are affected (errors are up, the site is slow), not on every internal blip (one server's CPU spiked, a disk is 80% full) that may not matter. Symptom-based alerting tied to SLOs catches what counts while dramatically cutting the noise that causes alert fatigue and burned-out on-call engineers.

TL;DR

Alert on symptoms (user impact), not every internal cause — page on what matters.
Noise is the enemy — too many alerts cause fatigue and ignored pages.
Tie alerts to SLOs and burn rate for signal over noise.
Make on-call humane — actionable alerts, runbooks, sustainable rotations.

Quick Example

The difference between a good and bad alert:

Symptom-Based Alerting

Alert on what the user experiences, not on internal mechanics:

Symptoms — elevated error rate, high latency, the service is down, a queue is backing up to the point of harm. These mean users are (or soon will be) affected.
Causes — high CPU, a restarted pod, a disk filling — might indicate a problem, but often self-resolve or don't affect users. Monitor them for diagnosis, but don't page on them.

💡 If an alert fires and the answer is "nothing's actually wrong / nothing to do," it shouldn't have paged. Every page should be actionable and tied to real impact.

Reducing Noise & Alert Fatigue

Noisy alerting is worse than none — people learn to ignore it, and real incidents get missed:

Every alert must be actionable — if there's nothing to do, it's a dashboard, not a page.
Tune thresholds and durations — require a condition to persist (e.g. 5 minutes) to avoid flapping on transient blips.
Alert on burn rate, not blips — page when you're consuming your error budget fast, not on every spike.
Deduplicate and group — one incident shouldn't fire fifty pages.
Prune relentlessly — delete alerts that never lead to action.

Severity & Routing

Severity levels — distinguish "wake someone up now" (page) from "look at it during business hours" (ticket) from "FYI" (notification).
Routing — send alerts to the right team/person based on the service and severity (PagerDuty, Opsgenie).
Escalation — if the first responder doesn't ack in N minutes, escalate to the next person.

Humane On-Call

On-call is sustainable only if alerting is good:

Actionable alerts + runbooks — every page should link to "what to do about it."
Sustainable rotations — reasonable frequency, follow-the-sun where possible, time off after incidents.
Blameless postmortems — learn from incidents without punishing people (see Monitoring).
Reduce the page count — the best on-call improvement is fewer, better alerts.

Best Practices

Alert on user-facing symptoms tied to SLOs, not internal causes.
Make every page actionable — with a runbook; delete non-actionable alerts.
Use burn-rate alerts to catch real SLO threats without noise.
Set severity and routing so the right person gets the right alert.
Protect on-call humans — sustainable rotations and continuous alert pruning.

Common Mistakes

Alerting on causes instead of symptoms

Tolerating noisy, non-actionable alerts

FAQ

Why alert on symptoms instead of causes?

Because symptoms (errors, slow responses, outages) reflect actual user impact, while causes (high CPU, a restarted process, a filling disk) often don't — they may self-resolve or never affect users at all. Alerting on every internal cause generates huge noise, most of it not actionable, which trains on-call engineers to ignore pages and miss the real incidents. Alerting on symptoms, ideally tied to SLOs, pages humans only when something users care about is actually wrong. You still monitor causes — they're invaluable for diagnosis once a symptom alert fires — you just don't page on them.

How do I reduce alert fatigue?

Make every alert actionable and delete the rest. Concretely: page only on user-facing symptoms (not internal causes), require conditions to persist for a few minutes to avoid flapping on transient spikes, alert on error-budget burn rate rather than every blip, deduplicate so one incident doesn't fire dozens of pages, and continuously prune alerts that never lead to action. The mindset shift is that a noisy alerting system is worse than a quiet one — once people learn alerts are usually noise, they stop responding, and the system fails exactly when it matters. Fewer, higher-signal alerts is the goal.

What is burn-rate alerting?

Burn-rate alerting pages based on how fast you're consuming your error budget rather than on raw thresholds. If a 99.9% monthly SLO gives ~43 minutes of allowed failure, a "burn rate" of 14× means you'd exhaust the whole month's budget in a few hours — clearly urgent and worth a page. A slow burn that would take weeks to exhaust the budget can be a low-priority ticket instead. This ties alert urgency directly to real reliability risk and user impact, cutting noise (no paging on harmless blips) while still catching genuine fast-moving incidents. It's a core SRE alerting technique.

How do I make on-call sustainable?

Good on-call rests on good alerting: if every page is actionable and noise is minimal, the human load drops dramatically. Beyond that, give each alert a runbook ("here's what to do"), keep rotations reasonable in frequency and length (consider follow-the-sun across time zones), give people recovery time after major incidents, and run blameless postmortems so the team learns without fear. Crucially, treat reducing the number of pages as an ongoing engineering priority — the most effective on-call improvement is usually fixing or deleting the alerts that wake people for no good reason.