Multiwindow Burn-Rate Alerts for SLOs in Prometheus

Implementing Multiwindow Burn-Rate Alerts

Threshold alerts — "page if the error rate is above 1% for five minutes" — are either too sensitive or not sensitive enough. They page at 3 a.m. for a two-minute blip that healed itself, and they stay silent while a steady 0.5% error rate quietly consumes a month of reliability. Burn-rate alerting fixes this by asking a better question: at the current error rate, how fast are we spending the error budget our SLO allows?

A multiwindow, multi-burn-rate setup, popularized by Google's SRE Workbook, pages quickly for severe problems, opens a ticket for slow leaks, ignores blips, and stops firing soon after recovery. This guide implements it from a user-facing SLI, with Prometheus rules, unit tests, and the edge cases that break naive versions.

TL;DR

Burn rate = observed error ratio ÷ error budget ratio. A burn rate of 1 spends exactly the whole budget over the SLO period.
Page when a significant share of the budget is burning fast: 2% in an hour (burn rate 14.4) or 5% in six hours (burn rate 6).
Open a ticket, not a page, for slow burns: 10% over three days (burn rate 1).
Pair each long window with a short window so alerts fire only while the problem is still happening and reset quickly after it ends.
Test the rules with synthetic series, and add alerts for missing telemetry — a silent SLI never pages.
Route every page to a runbook and dashboard; handle low-traffic services explicitly.

Quick Example

For a 99.9% availability SLO over 30 days, the error budget is 0.1% of requests — about 43 minutes of total outage. Three alert conditions cover fast and slow problems:

The fast-burn page in Prometheus, using recorded error ratios for each window:

During a total outage, the one-hour ratio crosses 1.44% in under a minute, so the page arrives almost immediately; after recovery, the five-minute window drops below the threshold within minutes, so the alert resolves instead of firing for the rest of the hour.

Core Concepts

From SLO to Error Budget

An SLO states the share of events that must be good — for example, 99.9% of checkout requests succeed over 30 days. The error budget is the remainder: 0.1% of requests may fail. Alerts should protect the budget, not an arbitrary threshold.

Burn Rate

A burn rate of 14.4 sustained for a month would spend the budget fourteen times over; sustained for a day, it spends nearly half of it. That is worth waking someone up.

Why Two Windows

A long window provides significance: one hour of data is not fooled by a ten-second spike. But a long window also has memory — after a 20-minute outage ends, the one-hour ratio stays above threshold for a long time, and the alert keeps firing while everything is fine. The short window, typically one-twelfth of the long one, must also be above threshold, so the alert resets within minutes of recovery.

Why Several Burn Rates

A single burn rate forces a bad trade-off: a high threshold misses slow leaks, a low one pages for problems that could wait until morning. Fast-burn conditions page; slow-burn conditions create tickets for working hours.

Building the Rules

Start From a User-Facing SLI

Measure what users experience, as close to them as practical — at the load balancer or API gateway rather than inside one service, so failures in routing and TLS count too. Availability SLIs count failed requests; latency SLIs count requests slower than a threshold as "bad":

Recording Rules

Precompute the error ratio for each window. Compute the ratio of sums, not the average of per-instance ratios, or quiet instances count as much as busy ones.

Add the fast-burn alert from the Quick Example and a slow-burn ticket:

Long-window rates over raw counters are expensive to evaluate. Generators such as Sloth and Pyrra produce these rule sets from a short SLO definition, including cheaper long-window calculations, and avoid copy-paste mistakes across services.

Validating the Alerts

Unit-Test the Rules

promtool test rules evaluates rules against synthetic series, so you can prove that a fast burn pages and a brief blip does not:

Run these tests in CI whenever rules change. Then confirm end to end in a test environment: inject errors with a fault-injection flag and check that the page reaches the right person with a working runbook link.

Missing Telemetry

If the exporter dies or a relabeling change renames the metric, the recorded ratios disappear and every burn-rate alert goes silent — exactly when something is wrong. Alert on absence:

Zero traffic is a different case: the ratio becomes NaN (0 ÷ 0), which never crosses a threshold. If an unexpected drop to zero traffic matters, add a separate traffic alert.

Low-Traffic Services

With 20 requests an hour, one failure is a 5% error ratio and a fast-burn page. Options:

Require a minimum volume before paging, for example and sum by (job) (increase(http_requests_total{job="checkout"}[1h])) > 500.
Generate synthetic traffic with probes so the SLI has enough events.
Combine related low-traffic endpoints into one SLO.
Use longer windows or ticket-only alerts for genuinely quiet services.

Routing and Runbooks

Route by the severity label in Alertmanager: page to the on-call rotation, ticket to the team's queue. Every page should link to a runbook and a dashboard showing the SLI, the budget remaining, and recent deploys — the first question in most burn-rate pages is "what changed?".

Best Practices

One SLO, One Alert Set

Replace older threshold alerts for the same symptom with the burn-rate set. Keeping both means duplicate pages and arguments about which one to trust.

Generate Rules From SLO Definitions

Define SLOs declaratively — in a tool such as Sloth or Pyrra, or in OpenSLO format — and generate recording rules, alerts, and dashboards. Hand-maintained rules drift between services.

Review Alert Quality Regularly

Track pages per week, the share that needed action, and time to acknowledge. A burn-rate page that was not actionable means the SLO, the SLI, or the window choice needs adjusting.

Show the Budget, Not Just the Alert

A dashboard with remaining error budget and burn rate over time turns alerts into decisions: whether to slow releases, prioritize reliability work, or accept the risk. See Site Reliability Engineering (SRE).

Common Mistakes

A Single Window

Averaging Per-Instance Ratios

Paging on Slow Burns

Measuring Far From Users

FAQ

What is a burn rate?

The speed at which a service spends its error budget, relative to the speed that would spend it exactly over the SLO period. A burn rate of 1 uses the whole budget in 30 days; a burn rate of 14.4 would use it in about two days.

Why 14.4?

It is the burn rate that consumes 2% of a 30-day error budget in one hour: 0.02 × 720 hours ÷ 1 hour. The SRE Workbook suggests 2% in an hour and 5% in six hours as page-worthy, and 10% in three days as ticket-worthy. Adjust the percentages to your tolerance and SLO period.

Why does each alert need two windows?

The long window makes the alert significant, so short blips do not page. The short window makes it reset quickly, so the alert stops firing once the problem is over instead of lingering until the long window forgets the incident.

Does this work for latency SLOs?

Yes. Define the SLI as the share of requests slower than a threshold — using histogram buckets — and treat that share as the error ratio. The burn-rate math and windows are identical.

What about services with very little traffic?

Low volume makes ratios noisy. Add a minimum-volume condition, generate synthetic traffic, combine endpoints into one SLO, or use ticket-only alerts with longer windows. Deciding nothing pages for a service is also legitimate when the business impact is low.