On-Call Operations & Incident Command for Engineering Teams

On-Call Operations and Incident Command

When production breaks, two things decide how long users suffer: whether the right person is reached quickly, and whether the people responding work as a coordinated team or as a crowd. On-call operations handle the first — rotations, escalation, and alerts worth waking up for. Incident command handles the second — clear roles, one person coordinating, regular communication, and a written record of what was decided and why.

Both are learnable practices rather than talents. Teams that define them in calm times resolve incidents faster, communicate better, and burn out less than teams that rely on whoever happens to be online and heroic.

TL;DR

Design sustainable rotations with a primary and a secondary, enough people to share the load, and clear escalation when a page is not acknowledged.
Page only for actionable, user-affecting conditions, and give every page a runbook.
Declare incidents early and assign an incident commander who coordinates rather than debugs.
Separate roles: operations lead, communications lead, and scribe, combined only in small incidents.
Communicate on a fixed cadence, keep a decision log, and hand off explicitly during long incidents.
Support responders: rest after night pages, blameless reviews, and on-call load tracked as a health metric.

Quick Example

The opening of a well-run incident channel. Roles are named, severity and impact are stated, the next update time is promised, and decisions are logged with reasons:

Anyone joining late can read the channel top to bottom and know what is happening, who owns what, and what has already been tried.

Core Concepts

Rotations

Escalation Policies

An escalation policy defines what happens when a page is not acknowledged: primary, then secondary after a few minutes, then a team lead or manager. Escalation to other teams depends on knowing who owns each service — a service catalog with owners and on-call schedules makes "who do we page for the payments API?" a lookup rather than a search.

Severity Levels

Define severities by user and business impact, not by how hard the problem looks. Severity can change during an incident; upgrading is normal.

Incident Command Roles

The model, adapted from emergency services' Incident Command System, separates coordination from hands-on work:

In a small incident one person may hold several roles, but the incident commander should still avoid debugging: someone needs to watch the whole picture while everyone else is heads-down.

Running an Incident

Declare it — anyone can, and the threshold should be low. Declaring early costs little; declaring late costs a lot.
Assign roles and open a dedicated channel or bridge.
Assess impact: who is affected, how badly, since when. Set severity.
Mitigate first: roll back, fail over, disable a feature flag, add capacity. Understanding can wait until users are safe — see Release Rollback and Database Compatibility.
Communicate on the promised cadence, even when there is no news.
Resolve when impact has ended and the system is stable, then confirm with monitoring.
Hand over to review: schedule the postmortem and preserve the timeline.

Communication Cadence

Each update says what is known, what is being done, the current impact, and when the next update will come. "No new information; next update at 15:00" is a valid and useful update.

The Decision Log

Record each significant decision with time, owner, and reasoning: "14:09 — roll back release 3 (IC). Errors began four minutes after deploy." The log keeps a crowded channel coherent, lets new responders catch up without interrupting, and becomes the backbone of the postmortem.

Handoffs

Long incidents outlast human concentration. Rotate the incident commander and responders every few hours, and make each handoff explicit:

The incoming commander announces "I am now IC" in the channel so nobody is left unsure who is in charge.

Supporting Responders

Make Pages Worth Answering

Every page should signal user impact and be actionable. Burn-rate alerts on SLOs page for real problems and ignore blips — see Implementing Multiwindow Burn-Rate Alerts. Every page links to a runbook.

Protect Recovery Time

After a night of pages, the responder should start late or hand the next shift to someone else. During long incidents, the incident commander schedules breaks and swaps people out before judgment degrades.

Train Before It Counts

New responders shadow experienced ones, then lead with an experienced shadow. Game days and incident drills — staged failures with real roles — let people practice incident command when stakes are low. See Chaos Engineering.

Measure On-Call Health

Track pages per shift, pages outside working hours, the share of pages that needed action, time to acknowledge, and repeat incidents. Rising numbers are a reliability problem and a retention problem, and they justify investment in fixes.

Best Practices

Use Tooling to Remove Friction

Incident bots that create channels, assign roles, start timelines, and post status-page drafts let responders focus on the problem. The process should be easier to follow than to skip.

Keep Security Incidents Connected

Some incidents turn out to be security incidents. Agree in advance how to bring in the security team and when to switch to the security incident process — see Security Incident Response.

Review Every Significant Incident

Run blameless postmortems for SEV-1 and SEV-2 incidents, track their action items to completion, and look for patterns across incidents rather than treating each as unique.

Common Mistakes

The Incident Commander Debugging

Declaring Too Late

Silence Between Updates

Heroics as a Staffing Model

FAQ

What does an incident commander do?

Coordinates the response. The incident commander assigns roles, sets priorities, makes or delegates decisions, makes sure communication happens on schedule, and decides when the incident is resolved. They deliberately avoid hands-on debugging so they can keep track of the whole response.

How many people should be in an on-call rotation?

Enough that each person is on call only occasionally and incidents per shift stay manageable. Very small rotations mean constant on-call duty, which is unsustainable; combining teams' rotations, follow-the-sun schedules, or reducing alert volume are the usual fixes.

When should we declare an incident?

As soon as users are likely affected or a problem needs more than one person to coordinate. A low threshold is healthy: an incident that turns out minor is cheap to close, while a late declaration costs response time and communication.

How often should we send updates?

On a fixed cadence matched to severity — every 30 minutes for the most severe incidents — and always with the time of the next update. Regular updates stop stakeholders from interrupting responders to ask for status.

Should developers be on call?

Usually, for the services they build. Teams that carry the pager for their own software fix operational problems at the source. That works only with good alerts, runbooks, fair rotations, and time allocated to fix what pages them.