Back to Notes

Coverage Fell Quietly in One Sprint

Coverage Fell Quietly in One Sprint

In one sprint, coverage quietly slipped from the 80% we had worked to reach and enforced in CI toward 72% over the next four sprints. New code shipped without tests, and nobody noticed until review. We had treated coverage as a milestone to reach once, not a living metric to maintain.

This pattern is nearly universal. When teams treat coverage as a one-time goal, it decays in proportion to deployment frequency. Faster shipping means faster decline, because new code outpaces later test writing unless tests are created in the same workflow as the code.

Why Coverage Decays Despite Careful Teams

The mechanism is predictable. Coverage is a ratio of tested lines to total lines. Each feature branch adds lines, and unless its author tests them in the same pull request, coverage falls a little. Repeat that across 20 feature branches in a quarter and the combined decline is substantial, even when each individual drop looks trivial.

New code usually lands in the areas that change most often. Those same areas are therefore most likely to have their tests broken by later changes. Low coverage in high-churn code is the worst setup for detecting regressions.

Coverage can also decay through test deletion instead of code addition. When selector drift or changing integration behavior makes tests fail repeatedly, teams often skip or remove them rather than repair them. Coverage falls, yet the CI gate stays green because the tested-code denominator has shrunk. Sometimes a team sees coverage rise after deleting flaky tests. It tested less, but the ratio increased because tests and code were removed together.

Coverage as a Rate, Not a Point

A more useful view treats coverage as a rate of change, not a point-in-time value. Ask not, "what is our coverage today?" but, "how does coverage change for each unit of new code shipped?" A team holding 80% across 50 consecutive deploys has healthier test infrastructure than one that reached 85% once and has declined ever since.

Coverage velocity, or the change per sprint or deploy, helps predict a regression spike in three months. Even a small negative velocity compounds into a meaningful quarterly gap. A team at 80% that loses 0.5 points per sprint reaches 74% in 12 weeks without deliberately choosing to test less.

A trend view exposes this better than a snapshot. Most CI reporters feature the current number and bury the trend in a small chart. Make the trend the headline and the current number the supporting detail, and the discussion of coverage health changes.

What Automatic Coverage Maintenance Requires

Maintaining coverage as a living number requires two activities that teams now handle manually to happen automatically.

First, tests for new paths must be written when code ships, not two weeks afterward. Manual follow-up testing is the main source of decay in active codebases. The dependable prevention is to create coverage for new code as it is committed. That could mean generated tests, a PR rule requiring tests for every new function, or a CI gate that fails when coverage drops past a threshold. Whatever the method, it must occur with the code change, not as a later task.

Second, tests broken by code changes should be repaired automatically, not skipped or deleted. Removing tests lowers the numerator without a matching denominator reduction, hiding real coverage gaps. Repair preserves the test and makes it valid again, so the coverage figure reflects reality.

After we added coverage gap detection to TestSprighte, we found that teams' real coverage was consistently below their reported coverage. Their reports counted tests exercising dead paths from earlier product iterations. Removing those tests and replacing them with tests for current live paths caused some teams' figures to fall before recovering. That is the correct result: accurate coverage is more useful than inflated coverage.

The Threshold Debate

Teams disagree about the threshold to enforce. 80% is the most common gate, some use 70%, and others aim for 90%+. The exact figure matters less than whether it applies consistently to new code and covers meaningful paths rather than total lines.

Total line coverage gives every line equal weight, so a 100-line utility packed with error-handling edge cases weighs the same as a 100-line critical payment flow. Path coverage, branch coverage, and mutation testing offer sharper signals about whether tests catch failure modes. Line coverage is common because it is easiest to calculate, but teams focused on coverage quality eventually move toward branch or mutation metrics.

We are not claiming 80% is a bad target. The important distinction is between 80% maintained as a living constraint checked continuously and 80% reached once, then watched passively. The same figure means something different when the team actively defends it.

Practical Setup

The minimum viable setup for a small team is to enforce a coverage floor on the --changed-files scope in CI, covering new code rather than only total codebase coverage. Each PR is then judged on whether it preserves coverage for its own changes, a more precise rule that is harder to game by deleting old test files. Add a dashboard trend line showing coverage velocity per sprint, and you have enough signal to catch decay before it enters the backlog.

Coverage is not, and never will be, a proxy for test quality. It is a proxy for maintenance discipline, which separates a test suite that survives 18 months of active development from one that becomes a liability after six.