Evidence-First Fixes

Someone reports that a step is slow. Within a minute the timeout is raised from 30 seconds to 120 and the ticket is closed. Nobody measured anything, the step now takes two minutes to fail instead of thirty seconds, and the actual cause is untouched.

If you cannot state the measured duration of the thing you are calling slow, any fix you write is a guess in the shape of a patch.

This skill covers the slow, flaky and timing-out class. For a system producing the wrong answer, root-cause debugging is the right tool — the failure mode there is a wrong hypothesis, not a missing measurement.

1. Take the report apart before believing it

An alert names a cause. That name is a hypothesis written by whoever configured the threshold, often years ago, and it is frequently wrong.

Separate what was observed ("the job exceeded 30s and was killed") from what was concluded ("the database is slow"). Only the first is evidence. Find the specific entity — this query, this endpoint, this file — not the aggregate. An average across a thousand requests hides the fifty that are actually broken.

2. Measure the named step in isolation

Reproduce it and put a number on it, with the cheapest instrument that works:

• time on the command, or a timer around the block.
• EXPLAIN ANALYZE for a query — the plan matters as much as the total.
• A profiler for CPU-bound work; identify whether it is CPU, I/O or lock contention before optimising the wrong one.
• Server timing or a trace for a request path, so you learn which segment holds the time.

Run it more than once. The first run is often cold cache and is not the number you want — but note it, because if the cold number is the one users get, then cold is the problem.

Record the units and the conditions. "2.3s, warm cache, 10k rows, local Postgres 16" is evidence. "It was slow" is not.

3. Reconcile against the configured threshold

Now find the number the system was actually checking against, and read it in the config rather than recalling it. Compare the two. The ratio is what tells you what kind of problem you have.

4. Classify — this is the step that prevents the wrong fix

Conflating rows two and four is how "just raise the timeout" happens. They look identical from the alert and could not be more different underneath.

5. Size the fix to the measured gap

A 40% overshoot is not a rewrite. A 40× overshoot is not a timeout bump — at that ratio something is algorithmically wrong, usually a query in a loop or an unbounded fetch, and no amount of headroom will save it.

State the expected improvement before making the change: "batching should cut this from 2.3s to roughly 200ms." A prediction you can check is what separates engineering from rearranging.

6. Measure again, in the same units

Same conditions, same instrument, same measurement. Report before and after together.

If the improvement is well short of the prediction, you fixed something that was not the bottleneck — which is useful information and means you are not finished. Say so rather than shipping and hoping.

Judgement calls

• Slow is not always wrong. Sometimes the honest answer is that the work takes that long and the expectation was miscalibrated. That is a real finding; deliver it rather than manufacturing an optimisation.
• Do not optimise what you did not measure, even when the cause looks obvious. The obvious candidate is wrong often enough that checking is cheaper than the wasted change.
• A flaky test is a timing problem with a pass rate. Measure the rate over many runs before theorising; "it fails sometimes" is not a measurement.

Verification gate

• A measured duration exists for the step, with units and conditions stated.
• The configured threshold was read from config, not recalled.
• The problem was placed in one of the four classes, and the fix matches that class.
• An expected improvement was stated before the change.
• An after-measurement was taken in the same units, and both numbers appear in the report.
• Paste both measurements. Do not describe them from memory.