What flakiness costs
A suite that fails at random stops being evidence. People re-run it until it is green, which means a real failure is re-run until it is green too — and the day the build goes red for a genuine reason, nobody believes it.
One flaky test in a thousand, run on every commit, fails somebody's build several times a week. It is not a small problem, and it is never worth "looking at later".
The five causes
Almost every flaky test is one of those five. The diagnosis is usually quick once you know the list.
| Cause | Looks like | Fix |
|---|---|---|
| Time | fails at midnight, at month end, or on a slow machine | inject the clock; fake timers |
| Randomness | fails one run in twenty | seed it, or pass it in |
| Order and shared state | passes alone, fails in the suite | fresh state per test; no module-level mutables |
| Real waiting | passes locally, fails in CI | poll for a condition; never sleep |
| Concurrency | two tests using the same row, port or file | a namespace per test — its own record, its own database |
Finding it
Run the single test fifty times. If it fails alone, the cause is inside it — time, randomness or a real wait. If it only fails in the suite, the cause is outside it, and shuffling the order will usually reproduce it within a few runs.
vitest run --repeat 50 path/to/test # is it random, or time?
vitest run --sequence.shuffle # is it order?
vitest run path/to/one.test.js # does it pass alone? then it is shared stateRetries are a decision, not a fix
A retry on an end-to-end test that talks to a real network is defensible: the network genuinely is unreliable, and the retry is documenting that. A retry on a unit test is hiding a bug — quite possibly a race in the production code, which will happen to a user and not be retried.
If you do retry, make it visible: report the tests that needed one, and treat a rising number as a failing build. A quiet retry is how a suite rots.
Quarantine, honestly
- A flaky test that cannot be fixed today goes in a quarantine list — still run, not blocking, and on somebody's name with a date.
- A quarantine with no owner and no expiry is a deleted test with extra steps. Delete it instead; at least then the gap is visible.
- Track the rate. "Two flaky tests, both owned, both due next week" is a healthy suite; "forty skipped" is not a suite at all.