TheSkillz TeamFlaky Test Triage
A flaky test does more damage than a failing one. A failing test stops the line; a flaky test teaches the team that red means "hit retry", and from then on every real failure is discovered in production. The cost is not the test — it is the habit it creates.
The mistake almost everyone makes is fixing the symptom: adding a sleep, widening a timeout, retrying the assertion. That converts a fast flaky test into a slow flaky test.
1. Prove it, with a number
"It fails sometimes" is a claim, not a measurement. Get the pass rate before touching anything:
for i in $(seq 1 50); do <run one test> >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
- 50 passes out of 50 — not flaky under these conditions. It fails in CI and not locally, so the difference is the bug: parallelism, machine speed, timezone, or ordering. Reproduce the CI conditions rather than the CI failure.
- A rate between 1% and 99% — genuinely flaky. Record the number; it is your before-measurement and the only way to know the fix worked.
- Fails only in a full run, never alone — ordering or shared state. Jump to that class.
2. Classify the cause
Each class has its own fix. They are not interchangeable, and this is the step that prevents the sleep-and-hope patch.
| Class | Signals | Fix |
|---|---|---|
| Time and timezone | Fails near midnight, month end, or only in CI's UTC | Inject a clock. Never call now() in the code under test. |
| Ordering and shared state | Passes alone, fails in a suite; passes in one order only | Find the leaked state — module singleton, shared DB row, env var — and reset it per test. |
| Unawaited async | Passes when slow, fails when fast; sometimes passes after the test "ends" | Await the promise. Assert on a condition, never after a fixed delay. |
| Network and ports | Fails in parallel, or on a colleague's machine | Bind port 0 and read the assigned port. Stub external calls. |
| Real concurrency | Rare, load-dependent, worse on more cores | A genuine race, usually in the code and not the test. Take it seriously — it is a production bug. |
| Resource exhaustion | Later tests fail, failures cluster at the end of a run | Leaked connections, handles, listeners. Close in a cleanup hook. |
Verify the classification before fixing it. Run the suite with a fixed seed and a shuffled seed; run the single test in isolation; run with concurrency at one. Each result eliminates classes.
3. Fix the class, not the instance
- Never add a bare sleep. If you must wait, wait on a condition with a timeout.
- Never retry an assertion to make it pass. Retry hides the race and keeps it in production.
- If the same class appears in several tests, fix the shared cause — the fixture, the helper, the bootstrap — rather than patching each test.
- If the flakiness is in the code rather than the test, say so loudly. A concurrency bug found by a flaky test is the test doing its job, and closing it as "flaky" throws away the finding.
4. Quarantine, with an expiry
Sometimes you cannot fix it today. Quarantine honestly:
- Move it to a non-blocking job. Do not delete it and do not silently
skipit. - Every quarantine entry carries an owner and a date. A quarantine with no expiry is a deletion with extra steps, and the file grows forever.
- Fail the build if the quarantine list grows, or if an entry is past its date.
5. Wire the signal into CI
One-off cleanups do not hold. Record per-test outcomes over time — most runners emit JUnit XML that a short script can accumulate — and surface the ten lowest pass rates. That list is the work queue, and it turns flakiness from a feeling into a number that trends.
Verification gate
- A before pass rate exists, from a stated number of runs.
- The cause was classified, and the classification was tested rather than assumed.
- No sleep, retry or widened timeout was added to reach green.
- An after pass rate was measured over at least as many runs as the before. 50/50 after a single green run proves nothing.
- Every quarantined test has an owner and an expiry date.
- Paste both pass rates. Do not describe them from memory.