---
name: flaky-test-triage
description: Proves a test is flaky with a measured pass rate before anyone touches it, classifies the cause as time, ordering, async, network, concurrency or resource exhaustion, and applies the fix pattern for that class. Includes a quarantine policy with an expiry and CI wiring to track per-test pass rates. Use for intermittent failures, a red build that passes on retry, or a suite nobody trusts.
license: MIT
compatibility: Any test runner (Vitest, Jest, pytest, go test, RSpec, JUnit, Playwright). Requires permission to run the suite repeatedly.
metadata:
  category: testing
  version: "1.0.0"
---

# Flaky Test Triage

A flaky test does more damage than a failing one. A failing test stops the line; a flaky test teaches
the team that red means "hit retry", and from then on every real failure is discovered in production.
The cost is not the test — it is the habit it creates.

The mistake almost everyone makes is fixing the symptom: adding a sleep, widening a timeout, retrying
the assertion. That converts a fast flaky test into a slow flaky test.

## 1. Prove it, with a number

"It fails sometimes" is a claim, not a measurement. Get the pass rate before touching anything:

```
for i in $(seq 1 50); do <run one test> >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
```

- **50 passes out of 50** — not flaky under these conditions. It fails in CI and not locally, so the
  difference *is* the bug: parallelism, machine speed, timezone, or ordering. Reproduce the CI
  conditions rather than the CI failure.
- **A rate between 1% and 99%** — genuinely flaky. Record the number; it is your before-measurement
  and the only way to know the fix worked.
- **Fails only in a full run, never alone** — ordering or shared state. Jump to that class.

## 2. Classify the cause

Each class has its own fix. They are not interchangeable, and this is the step that prevents the
sleep-and-hope patch.

| Class | Signals | Fix |
| --- | --- | --- |
| **Time and timezone** | Fails near midnight, month end, or only in CI's UTC | Inject a clock. Never call `now()` in the code under test. |
| **Ordering and shared state** | Passes alone, fails in a suite; passes in one order only | Find the leaked state — module singleton, shared DB row, env var — and reset it per test. |
| **Unawaited async** | Passes when slow, fails when fast; sometimes passes after the test "ends" | Await the promise. Assert on a condition, never after a fixed delay. |
| **Network and ports** | Fails in parallel, or on a colleague's machine | Bind port 0 and read the assigned port. Stub external calls. |
| **Real concurrency** | Rare, load-dependent, worse on more cores | A genuine race, usually in the code and not the test. Take it seriously — it is a production bug. |
| **Resource exhaustion** | Later tests fail, failures cluster at the end of a run | Leaked connections, handles, listeners. Close in a cleanup hook. |

**Verify the classification before fixing it.** Run the suite with a fixed seed and a shuffled seed;
run the single test in isolation; run with concurrency at one. Each result eliminates classes.

## 3. Fix the class, not the instance

- **Never** add a bare sleep. If you must wait, wait on a condition with a timeout.
- **Never** retry an assertion to make it pass. Retry hides the race and keeps it in production.
- If the same class appears in several tests, fix the shared cause — the fixture, the helper, the
  bootstrap — rather than patching each test.
- If the flakiness is in the *code* rather than the test, say so loudly. A concurrency bug found by a
  flaky test is the test doing its job, and closing it as "flaky" throws away the finding.

## 4. Quarantine, with an expiry

Sometimes you cannot fix it today. Quarantine honestly:

- Move it to a non-blocking job. Do not delete it and do not silently `skip` it.
- **Every quarantine entry carries an owner and a date.** A quarantine with no expiry is a deletion
  with extra steps, and the file grows forever.
- Fail the build if the quarantine list grows, or if an entry is past its date.

## 5. Wire the signal into CI

One-off cleanups do not hold. Record per-test outcomes over time — most runners emit JUnit XML that a
short script can accumulate — and surface the ten lowest pass rates. That list is the work queue, and
it turns flakiness from a feeling into a number that trends.

## Verification gate

- [ ] A before pass rate exists, from a stated number of runs.
- [ ] The cause was classified, and the classification was tested rather than assumed.
- [ ] No sleep, retry or widened timeout was added to reach green.
- [ ] An after pass rate was measured over **at least as many runs** as the before. 50/50 after a
      single green run proves nothing.
- [ ] Every quarantined test has an owner and an expiry date.
- [ ] Paste both pass rates. Do not describe them from memory.