Site Reliability Engineering (SRE): Error Budgets & Reliability Practice

Q: SRE vs DevOps — what's the difference?

Same destination, different specificity: DevOps is the broad cultural movement (dev and ops collaborate, automate the pipeline, own what you ship); SRE is one concrete, opinionated implementation with named mechanisms — SLOs, error budgets, toil caps. The quip "class SRE implements DevOps" is roughly right.

Q: Do we need an SRE team, or SRE practices?

Under ~50 engineers: practices, embedded in product teams. A dedicated SRE/platform function starts paying off when multiple teams share infrastructure and someone needs to own the paved road — and even then, product teams keep on-call for their own services in most modern setups.

Q: What's a reasonable first SLO?

Availability of your most important user-facing endpoint: non-5xx responses / total responses ≥ 99.9% over 30 days, measured at the load balancer (closest to the user's truth). It's simple, hard to game, and the burn-rate alerting recipe for it is textbook.

Q: How does SRE relate to observability?

Observability (metrics, logs, traces) is the sensory system; SRE is the decision framework consuming it. SLIs are computed from observability data; postmortems are only as good as the telemetry that lets you reconstruct what happened.

Q: Where do error budgets break down?

Where the SLI doesn't capture what users feel (budget healthy, users furious), where a single dependency's outage eats a quarter's budget in an hour (address with design, not freezes), and where leadership overrides the policy the first time it's inconvenient — which retroactively deletes it.

Site Reliability Engineering (SRE)

Site Reliability Engineering is the discipline of running production systems with an engineering mindset — Google's formulation: "what happens when you ask a software engineer to design an operations function." Instead of ops as reactive firefighting, SRE treats reliability as a measurable product feature with explicit targets, and operations problems as software problems to be automated away.

Its exportable core is a handful of ideas — SLOs, error budgets, toil reduction, blameless postmortems — that give teams a shared, numerical language for the eternal argument between shipping fast and staying up. You don't need Google's org chart to use them; you need the mindset.

TL;DR

100% is the wrong reliability target — users can't tell 99.99% from 99.999%, but your feature velocity can. Pick the cheapest target users are happy with.
An SLO (e.g. "99.9% of requests succeed, measured over 30 days") defines that target on top of a measured SLI; the SLO gap is your error budget.
The error budget is a decision tool: budget remaining → ship freely; budget burned → reliability work takes priority over features. It converts culture wars into arithmetic.
Toil — manual, repetitive, automatable ops work — is tracked and capped (Google's rule: <50% of SRE time), with the remainder spent engineering it away.
Blameless postmortems treat incidents as system failures, not people failures — because punished engineers stop reporting near-misses, and the system stops learning.
Small teams adopt SRE as practices, not a department: a few SLOs, alerting on burn rate, a postmortem habit, and an automation budget.

The Core Ideas

SLIs, SLOs, and Error Budgets

The measurement stack:

The error budget reframes reliability from a virtue into a resource you spend — on deploys, risky migrations, chaos experiments. The accompanying policy is what gives it teeth:

Now "should we ship Friday afternoon?" isn't an opinion contest between product and ops — it's a lookup. See SLOs for choosing indicators and windows, and Metrics for the measurement plumbing.

Alerting on Burn Rate, Not Blips

SRE's biggest practical gift to on-call sanity: page on error-budget burn rate, not on instantaneous symptoms.

Multi-window burn-rate alerts fire when users are actually being hurt at a rate that threatens the objective — and stay silent for self-healing blips. This single change eliminates most 3 a.m. pages that end in "it recovered on its own." Details in Alerting.

Toil

Toil is operational work that is manual, repetitive, automatable, reactive, and scales with the service — cert renewals by hand, restarting the flaky worker, provisioning via ticket queue. It's distinct from real engineering and from legitimate incident response.

The SRE discipline: measure it (track time spent), cap it (the 50% rule — beyond that you've become a ticket queue with a fancy title), and spend the reclaimed time automating its sources. Every quarter, the top toil item becomes an engineering project: the manual runbook becomes a script, the script becomes a self-healing check, the check becomes "why does this even happen?" fixed at the root.

Incidents and Blameless Postmortems

SRE's incident practice, condensed:

During: clear roles (incident commander ≠ the person typing), a comms channel, severity levels agreed before the incident, and runbooks linked from the alerts themselves.
After, for significant incidents, a postmortem: timeline, user impact quantified against the SLO, contributing causes (plural — real incidents are never one cause), and tracked action items with owners.
Blameless is load-bearing, not politeness: if the engineer who fat-fingered the config gets named and shamed, the next near-miss goes unreported, and the actual defect — a production config change with no validation, no review, no rollback — stays unfixed. The question is never "who?"; it's "why did the system let this happen, and how does it not happen again?"

A postmortem whose action items quietly die in the backlog is theater; review them like any other engineering commitment.

Reducing the Cost of Failure

Since failure is inevitable (that's why the budget exists), SRE invests in making failures small and short: progressive deployment strategies (canaries catch bugs at 1% of traffic), fast rollbacks as a first response ("roll back, then debug"), load shedding and graceful degradation, capacity planning, and — for the mature — chaos engineering to verify resilience assumptions before production does it for you. High availability design is the structural half of the same goal.

SRE for Normal-Sized Teams

You are not Google; the ideas still work. A right-sized adoption path:

Pick 2–3 SLOs for what users feel — API availability, checkout success rate, p95 latency on the core flow. Instrument the SLIs (Prometheus/Grafana or your APM).
Rewrite paging alerts as burn-rate alerts; demote everything not user-visible to tickets/dashboards. Measure pages-per-week before and after.
Adopt the postmortem template (timeline, impact, causes, actions) for every SEV-worthy incident; review action items monthly.
Give toil a budget: one automation task per person per sprint, sourced from a visible "most annoying manual work" list.
Write the error-budget policy down — one page, agreed with product — so the freeze conversation is precedent, not war.

What to avoid: renaming the ops team "SRE" and changing nothing; SLOs on internal metrics nobody feels; hiring "an SRE" to absorb all reliability responsibility while everyone else ships (Google's model embeds shared production ownership — you ship it, you help run it).

Common Mistakes

Reliability Theater

Five nines in the deck, no SLI actually measured, alerts on CPU. If you can't state today's remaining error budget, you have vibes, not SRE.

SLOs Without a Policy

An SLO that triggers nothing when violated is a dashboard decoration. The entire mechanism is the agreed consequence — the freeze/prioritization rule signed by product and engineering both.

Aiming Too High

99.99% on an internal tool means paying weekend-pager costs for reliability nobody asked for — and your dependencies (cloud load balancers, DNS) may not even offer it. Set targets from user need, and remember each added nine costs roughly 10× more.

On-Call Without Investment

Rotations where every shift means multiple night pages, with no time budgeted to fix the causes, burn out exactly the people who know the system best. Page volume is itself an SLO for the team: track it, cap it, engineer it down.

Blame Wearing a Blameless Mask

"Blameless" postmortems where everyone silently knows Marcus is in trouble teach the same lesson as open blame. Leadership behavior in the first three incidents sets the real policy.

FAQ

SRE vs DevOps — what's the difference?