---
name: evidence-first-fixes
description: Governs slow, flaky and timing-out systems by requiring a measurement before a diff, reconciling it against the configured threshold, and classifying the result as a false alarm, genuine work, a wedged process or a regression — each of which has a different correct fix. Use for performance problems, timeout alerts, slow builds or queries, and anything described as hanging.
license: MIT
compatibility: Any stack. Requires permission to run the code under test and read its configuration.
metadata:
  category: coding
  version: "1.0.0"
---

# Evidence-First Fixes

Someone reports that a step is slow. Within a minute the timeout is raised from 30 seconds to 120 and
the ticket is closed. Nobody measured anything, the step now takes two minutes to fail instead of
thirty seconds, and the actual cause is untouched.

**If you cannot state the measured duration of the thing you are calling slow, any fix you write is a
guess in the shape of a patch.**

This skill covers the *slow, flaky and timing-out* class. For a system producing the *wrong answer*,
root-cause debugging is the right tool — the failure mode there is a wrong hypothesis, not a missing
measurement.

## 1. Take the report apart before believing it

An alert names a cause. That name is a hypothesis written by whoever configured the threshold, often
years ago, and it is frequently wrong.

Separate what was **observed** ("the job exceeded 30s and was killed") from what was **concluded**
("the database is slow"). Only the first is evidence. Find the specific entity — this query, this
endpoint, this file — not the aggregate. An average across a thousand requests hides the fifty that
are actually broken.

## 2. Measure the named step in isolation

Reproduce it and put a number on it, with the cheapest instrument that works:

- `time` on the command, or a timer around the block.
- `EXPLAIN ANALYZE` for a query — the plan matters as much as the total.
- A profiler for CPU-bound work; identify whether it is CPU, I/O or lock contention before optimising
  the wrong one.
- Server timing or a trace for a request path, so you learn *which* segment holds the time.

Run it more than once. The first run is often cold cache and is not the number you want — but note it,
because if the cold number is the one users get, then cold is the problem.

**Record the units and the conditions.** "2.3s, warm cache, 10k rows, local Postgres 16" is evidence.
"It was slow" is not.

## 3. Reconcile against the configured threshold

Now find the number the system was actually checking against, and read it in the config rather than
recalling it. Compare the two. The ratio is what tells you what kind of problem you have.

## 4. Classify — this is the step that prevents the wrong fix

| Measurement vs threshold | Diagnosis | Correct fix |
| --- | --- | --- |
| Comfortably under, yet it alerted | **False alarm** — the threshold or the alert's scope is wrong | Fix the alert. Change no application code. |
| Slightly over, and the work genuinely is that size | **Needs headroom** | Raise the threshold *deliberately*, with the measured number recorded next to it. |
| No progress at all, not merely slow | **Wedged** — a deadlock, an unclosed connection, a lost lock | Speed is irrelevant. Find what it is waiting on. |
| Far over, and it used to be fine | **Regression** | `git bisect` on a benchmark. Do not optimise; find the commit. |

Conflating rows two and four is how "just raise the timeout" happens. They look identical from the
alert and could not be more different underneath.

## 5. Size the fix to the measured gap

A 40% overshoot is not a rewrite. A 40× overshoot is not a timeout bump — at that ratio something is
algorithmically wrong, usually a query in a loop or an unbounded fetch, and no amount of headroom will
save it.

State the expected improvement **before** making the change: "batching should cut this from 2.3s to
roughly 200ms." A prediction you can check is what separates engineering from rearranging.

## 6. Measure again, in the same units

Same conditions, same instrument, same measurement. Report before and after together.

If the improvement is well short of the prediction, you fixed something that was not the bottleneck —
which is useful information and means you are not finished. Say so rather than shipping and hoping.

## Judgement calls

- **Slow is not always wrong.** Sometimes the honest answer is that the work takes that long and the
  expectation was miscalibrated. That is a real finding; deliver it rather than manufacturing an
  optimisation.
- **Do not optimise what you did not measure**, even when the cause looks obvious. The obvious
  candidate is wrong often enough that checking is cheaper than the wasted change.
- **A flaky test is a timing problem with a pass rate.** Measure the rate over many runs before
  theorising; "it fails sometimes" is not a measurement.

## Verification gate

- [ ] A measured duration exists for the step, with units and conditions stated.
- [ ] The configured threshold was read from config, not recalled.
- [ ] The problem was placed in one of the four classes, and the fix matches that class.
- [ ] An expected improvement was stated before the change.
- [ ] An after-measurement was taken in the same units, and both numbers appear in the report.
- [ ] Paste both measurements. Do not describe them from memory.