Failure Injection and Recovery Exercises
Every system has failure modes nobody has tested: the dependency that becomes slow rather than down, the availability zone that disappears, the certificate that expires, the retry storm that follows a brief outage. Failure injection — chaos engineering in practice — introduces such faults deliberately, in controlled conditions, to find weaknesses before they cause incidents. Recovery exercises test the human side: whether alerts fire, runbooks work, and responders can restore service within targets.
Done carelessly, failure injection is just an outage you caused. Done well, it is a scientific experiment: a hypothesis about how the system behaves under a fault, a measured steady state, a deliberately small blast radius, explicit stop conditions, and evidence that turns into fixes. This page describes how to design, run, and learn from such experiments.
TL;DR
- Start with a hypothesis framed around a steady-state metric users care about, such as checkout success or SLO compliance.
- Pick a realistic fault: instance loss, latency, errors, dependency outage, zone loss, resource exhaustion, clock or DNS problems.
- Limit the blast radius: a fraction of instances, one region, short duration, and an announced time window.
- Define stop conditions that abort automatically, and a rollback that removes the fault immediately.
- Progress from local and staging to production as confidence grows; run game days for people and processes.
- Capture recovery evidence — time to detect, time to mitigate, alerts, runbook accuracy — and track fixes to completion.
Quick Example
An experiment plan reviewed before anything is injected:
The fault itself, with Chaos Mesh on Kubernetes:
Core Concepts
Hypotheses and Steady State
A useful hypothesis names a fault, an expected outcome in terms of a steady-state metric, and the mechanism you believe provides resilience: "If one database replica fails, read latency stays within SLO because the connection pool fails over within 10 seconds." Measuring steady state before, during, and after the fault distinguishes real impact from noise. Metrics tied to user outcomes — success rates, SLO burn — are better than CPU graphs. See SLOs.
Fault Types
Slow dependencies are often more damaging than dead ones, because they hold threads and connections; see Reliability Testing and Dependency Failures.
Blast Radius and Safety
Start small and expand: one pod, then a percentage, then a zone. Run in business hours with the on-call team informed, avoid peak traffic and change freezes, and make faults time-bounded so they expire on their own. Stop conditions should be automated where possible — AWS Fault Injection Service, for example, can stop experiments when CloudWatch alarms fire. Everyone involved must be able to abort instantly.
Progression of Environments
Staging rarely reproduces production load, data, and configuration exactly, which is why carefully scoped production experiments find issues staging misses.
Tools
Game Days and Recovery Evidence
A game day is a scheduled exercise in which a team responds to injected failures — or a simulated scenario — as if it were real. It tests alerts, dashboards, runbooks, escalation, and communication, not only the software. Capture:
- Time to detect: did an alert fire, and how quickly?
- Time to mitigate and recover, compared with recovery objectives.
- Runbook accuracy: which steps were missing, wrong, or slow?
- Data integrity: were writes lost, duplicated, or corrupted?
- Communication: were status updates and escalations timely?
Write a short report like a blameless postmortem, with owners and due dates for every finding. See Postmortems and On-Call Operations and Incident Command.
Best Practices
Write the Hypothesis Before Injecting
If you cannot state the expected outcome and mechanism, you are not ready to run the experiment.
Automate Stop Conditions
Tie experiments to alarms that end them automatically when user-facing metrics degrade.
Expand Scope Gradually
Increase blast radius only after smaller experiments confirm the hypothesis.
Close the Loop
Track findings as work items, fix them, and rerun the experiment to confirm the fix.
Common Mistakes
Chaos Without a Hypothesis
No Way to Stop
Findings That Never Get Fixed
FAQ
What is failure injection?
Deliberately introducing faults — such as terminated instances, network latency, or dependency errors — into a system under controlled conditions to verify that it behaves as expected and to find weaknesses.
Is it safe to run chaos experiments in production?
It can be, with a clear hypothesis, a small and time-bounded blast radius, automated stop conditions, informed on-call staff, and confidence gained from earlier experiments in lower environments.
What is a game day?
A planned exercise in which a team responds to injected or simulated failures to test systems, alerts, runbooks, and incident response processes together.
What should I measure during an experiment?
User-facing steady-state metrics such as success rates and latency percentiles, plus recovery evidence: time to detect, time to mitigate, alert behavior, and data integrity.
Which faults should I start with?
Those most likely to happen or most damaging in your system: instance loss, dependency latency and errors, and zone failures are common starting points.
Related Topics
- Chaos Engineering — The discipline behind failure injection
- SLOs — Steady-state metrics for experiments
- Reliability Testing and Dependency Failures — Testing resilience to dependency problems
- On-Call Operations and Incident Command — Responding during exercises
- Postmortems — Learning from incidents and exercises
- Kubernetes — A common target for fault injection tools