Systematic Debugging

The rule: no fix without a reproduction, and no reproduction without a failing check you can re-run. Guess-and-patch cycles burn time and hide the real cause.

The loop

1. Capture the symptom precisely

Write down, verbatim, before touching anything:

• Expected vs actual behaviour, with the exact error text and stack trace.
• Where it happens (env, OS, browser, version, user role, data) and where it does not.
• When it started: last known good commit, deploy or dependency bump (git log --since, deploy history).

If the report is vague, get the exact input. "Sometimes the export is wrong" is not debuggable; "export of order 8813 shows 0 tax" is.

2. Reproduce

• Build the smallest deterministic reproduction: a failing unit test, a script, a curl command or a recorded browser session.
• If it is intermittent, loop it (for i in $(seq 50); do ...; done) and record the failure rate; reduce the loop while keeping the failure.
• Save the reproduction; it becomes the regression test.

If you cannot reproduce, do not proceed to fixes. Add instrumentation (step 4) in the environment where it happens and wait for the next occurrence.

3. Form ranked hypotheses

List three to five candidate causes, each with a cheap test that would distinguish it. Rank by prior probability × cheapness of the test. Common priors:

• Recent change in this area (check git blame on the failing lines).
• State that differs between working and failing cases (env vars, cache, DB rows, feature flags, time zone, locale).
• Boundaries: empty input, off-by-one, null, max size, Unicode, DST transitions.
• Concurrency: retries, double submission, missing await, shared mutable state.
• Environment: dependency version drift (lockfile diff), Node/Python version, missing build step.

4. Instrument, do not stare

• Add targeted logging at the boundaries of the suspect path with correlation ids and the values that matter; log at entry and exit.
• Use the debugger for local reproductions; use structured logs or tracing for remote ones.
• Assert your assumptions in code (assert amount >= 0) so violations surface immediately.
• Diff the working and failing runs side by side.

5. Bisect when the cause is a change

git bisect start && git bisect bad HEAD && git bisect good <last-good>
git bisect run ./repro.sh     # exit 0 = good, 1 = bad

Bisect over commits, over dependencies (pin versions one at a time), or over config (toggle flags one at a time). Change exactly one variable per experiment.

6. Confirm the root cause

You have the cause when you can (a) explain every observed symptom, including why it was intermittent or env-specific, and (b) predict a way to make it happen and not happen on demand. If any symptom is unexplained, keep going.

7. Fix at the cause, then prove it

• Fix the root cause, not the place where the error surfaced.
• Run the reproduction: it must now pass. Run the full suite: nothing else may break.
• Convert the reproduction into a permanent regression test with a comment referencing the issue.
• Remove temporary instrumentation, keep any logging that would have made this faster next time.
• Search for the same pattern elsewhere in the codebase (rg) and fix siblings.

Output format

## Symptom
## Reproduction (command/test, failure rate)
## Hypotheses tried
1. X — ruled out by ...
2. Y — confirmed by ...
## Root cause
## Fix (files)
## Proof (test output before/after, full suite result)
## Follow-ups (sibling bugs, monitoring, docs)

Checklist

• A failing, re-runnable reproduction exists before any code change.
• Each experiment changed one variable and its result was recorded.
• The root cause explains all symptoms.
• A regression test is committed with the fix.
• Instrumentation noise is removed.

Pitfalls

• "It works on my machine" means the environment is a variable; diff it rather than dismissing it.
• Adding a retry, a try/catch or a sleep to make a symptom go away is not a fix; note it explicitly as a mitigation if you must ship it.
• Do not trust a fix that you could not make fail first.
• Beware of fixing the reproduction rather than the bug (e.g., special-casing the test input).
• Two bugs can produce one symptom; after the first fix, re-run the original report, not just the reduced case.
• Time-based and ordering bugs need the loop repeated many times after the fix, not once.