Coverage, honestly

Read a coverage number for what it is, and use the one measurement that tells you whether a test would catch anything.

13of 16
≈ 12Min read
5Parts

What the percentage actually measures

Every one of them measures execution, not assertion. A test that calls a function and asserts nothing gives exactly the same coverage as one that checks every result. This is not a subtle limitation; it is the whole story.

100% coverage, zero value
it("works", () => {
  quote({ lines: [{ qty: 2, price: 890 }] });      // every line executed
});                                                 // …and nothing asserted
MetricCounts
Line / statementlines executed at least once
Brancheach side of each if, ?: and &&
Functionfunctions called at least once
Pathcombinations of branches — nobody reports this, it explodes

What it is good for

  • Finding what is never run at all. A module at 0% is either dead code or a gap somebody should know about.
  • A ratchet. "Do not let it fall below where it is" is a reasonable rule; "get it to 100" is not.
  • Branch coverage on a specific function you are about to change — it tells you which case you have not thought about.
A target turns into a game

Mandate 90% and people write tests that execute code without asserting, test getters, and add / istanbul ignore / to whatever is awkward. The number goes up and the suite gets worse. Goodhart’s law, with a CI badge.

The measurement that does mean something

A mutant that makes the suite fail is killed — some test noticed. A mutant that survives is a change to your production code that nobody would have caught, which is a precise description of a missing test.

Mutation score is expensive to compute (the suite runs once per mutant) and far more honest than coverage. Run it on the parts that matter — the pricing, the permissions, the thing that moves money — not on the whole repository.

mutation testing: change the code, see whether the tests notice
// the original
if (order.qty > 0) { … }

// mutants
if (order.qty >= 0) { … }      // boundary
if (order.qty < 0)  { … }      // negated
if (true)           { … }      // removed

The equivalent mutant

Those two lines behave identically: when qty is zero, adding 0 × price changes nothing. No test can distinguish them, so the mutant survives for ever and the score never reaches 100%.

Equivalent mutants are the reason mutation score is read as a direction rather than a target — and the reason every tool lets you mark one as "known, and impossible". Recognising them is part of using the technique.

a change that cannot be caught, because it changes nothing
if (line.qty > 0) total += line.qty * line.price;
if (line.qty >= 0) total += line.qty * line.price;   // a mutant nobody can kill

What no measurement will tell you

Whether the behaviour is the one the business asked for. Whether the error message is usable. Whether the thing is fast enough, or accessible, or secure. Coverage and mutation score both measure the relationship between your tests and your code — neither has any opinion about whether the code is right.

Practice