Mutation Testing: Do Your Tests Catch Real Bugs?

Mutation Testing

Code coverage tells you which lines your tests execute. It says nothing about whether the tests would notice if those lines were wrong. A test that calls a function and asserts nothing reaches 100% coverage and catches no bugs. Mutation testing measures what coverage cannot: it makes small deliberate changes to the code — flipping >= to >, replacing + with -, deleting a function call — and runs the tests against each changed version. If a test fails, the mutant is killed. If every test still passes, the mutant survives, revealing behavior your tests do not check.

Mutation testing is the most direct measure of test effectiveness available, and also one of the most expensive to run. Used selectively — on critical code and on changed lines — it is an excellent way to find weak assertions and missing boundary tests.

TL;DR

A mutant is your code with one small change; tests should fail against it.
Killed mutants mean tests detect the change; surviving mutants show untested behavior.
Mutation score = killed ÷ (total − equivalent); use it as a guide, not a target.
Some mutants are equivalent — they behave identically — and can never be killed.
Tools: Stryker (JavaScript, TypeScript, C#, Scala), PIT (Java), mutmut (Python), cargo-mutants (Rust), Infection (PHP).
Keep runs affordable with incremental and changed-code mutation in CI.

Quick Example

A function and a test with full line coverage:

A mutation tool changes >= to > and runs the test. It still passes — can_vote(18) is never checked — so the mutant survives. The fix is a boundary assertion:

Coverage was 100% both times; only mutation testing showed the gap.

Core Concepts

Mutation Operators

Outcomes

Equivalent Mutants

Some mutations change code without changing behavior — for example, mutating a loop bound in a way the surrounding logic makes irrelevant. They cannot be killed, inflate the survivor count, and must be identified by a human. This is why a 100% mutation score is neither realistic nor a sensible goal.

Tools

Stryker for a TypeScript project:

Keeping It Affordable

Every mutant means another test run, so naive mutation testing can take hours. Practical strategies:

Mutate changed code only in pull requests, with full runs nightly or weekly.
Use incremental modes (Stryker's incremental runs, PIT's history files) that reuse previous results.
Run only the tests that cover each mutant — most tools do this by default from coverage data.
Exclude generated code, logging, and trivial getters from mutation.
Focus on critical modules: pricing, permissions, validation, financial calculations.

Using the Results

Treat surviving mutants as questions, not failures: "Should a test notice this change?" Often the answer is a missing boundary or assertion. Sometimes the mutated code is dead, redundant, or genuinely irrelevant — and the right fix is to delete it. Mutation testing is also a strong way to evaluate tests written for legacy code before refactoring — see Characterization Tests for Legacy Code.

Best Practices

Start Small

Run mutation testing on one important module, review the survivors with the team, and improve tests there before expanding.

Prefer Trends and Thresholds on New Code

Gate pull requests on the mutation score of changed code, or on not adding new survivors, rather than on a whole-codebase percentage that old code drags down.

Review Survivors, Do Not Just Count Them

The value is in the specific gaps survivors reveal. A slightly lower score with every survivor understood is better than a higher score reached by weak tests targeting mutants.

Combine With Other Test Techniques

Boundary survivors often suggest property-based tests, which cover many boundaries at once.

Common Mistakes

Treating Coverage as Proof

Chasing 100% Mutation Score

Running Full Mutation on Every Commit

FAQ

What is mutation testing?

A technique that measures test effectiveness by introducing small changes (mutants) into the code and checking whether the test suite fails. Surviving mutants reveal behavior the tests do not verify.

What is a good mutation score?

There is no universal number. Many teams find 70–90% achievable on well-tested critical code. More useful than a target is ensuring new code adds no unexamined survivors and that critical modules score well.

Why is mutation testing slow?

Because the test suite runs once per mutant, and a codebase can produce thousands of mutants. Incremental runs, mutating only changed code, and running only covering tests make it practical.

How is mutation score different from code coverage?

Coverage measures which code runs during tests. Mutation score measures whether the tests detect changes to that code. High coverage with a low mutation score means tests execute code without checking its results.

What are equivalent mutants?

Mutants whose change does not alter observable behavior, so no test can kill them. They have to be identified and excluded manually, which is one reason 100% mutation scores are unrealistic.