---
name: systematic-debugging
description: Drives a disciplined hypothesis-driven debugging loop — reproduce, isolate, instrument, bisect, fix, prove — instead of guessing at patches. Use when a bug is hard to reproduce, a test fails intermittently, behaviour differs between environments, or a previous quick fix did not hold.
license: MIT
compatibility: Any language or stack. Works best with access to logs, a test runner and git history.
metadata:
  category: coding
  version: "1.0.0"
---

# Systematic Debugging

The rule: **no fix without a reproduction, and no reproduction without a failing check you can re-run.** Guess-and-patch cycles burn time and hide the real cause.

## The loop

### 1. Capture the symptom precisely
Write down, verbatim, before touching anything:
- Expected vs actual behaviour, with the exact error text and stack trace.
- Where it happens (env, OS, browser, version, user role, data) and where it does not.
- When it started: last known good commit, deploy or dependency bump (`git log --since`, deploy history).

If the report is vague, get the exact input. "Sometimes the export is wrong" is not debuggable; "export of order 8813 shows 0 tax" is.

### 2. Reproduce
- Build the smallest deterministic reproduction: a failing unit test, a script, a curl command or a recorded browser session.
- If it is intermittent, loop it (`for i in $(seq 50); do ...; done`) and record the failure rate; reduce the loop while keeping the failure.
- Save the reproduction; it becomes the regression test.

If you cannot reproduce, do not proceed to fixes. Add instrumentation (step 4) in the environment where it happens and wait for the next occurrence.

### 3. Form ranked hypotheses
List three to five candidate causes, each with a cheap test that would distinguish it. Rank by prior probability × cheapness of the test. Common priors:
- Recent change in this area (check `git blame` on the failing lines).
- State that differs between working and failing cases (env vars, cache, DB rows, feature flags, time zone, locale).
- Boundaries: empty input, off-by-one, null, max size, Unicode, DST transitions.
- Concurrency: retries, double submission, missing await, shared mutable state.
- Environment: dependency version drift (`lockfile` diff), Node/Python version, missing build step.

### 4. Instrument, do not stare
- Add targeted logging at the boundaries of the suspect path with correlation ids and the values that matter; log at entry and exit.
- Use the debugger for local reproductions; use structured logs or tracing for remote ones.
- Assert your assumptions in code (`assert amount >= 0`) so violations surface immediately.
- Diff the working and failing runs side by side.

### 5. Bisect when the cause is a change
```bash
git bisect start && git bisect bad HEAD && git bisect good <last-good>
git bisect run ./repro.sh     # exit 0 = good, 1 = bad
```
Bisect over commits, over dependencies (pin versions one at a time), or over config (toggle flags one at a time). Change exactly one variable per experiment.

### 6. Confirm the root cause
You have the cause when you can (a) explain every observed symptom, including why it was intermittent or env-specific, and (b) predict a way to make it happen and not happen on demand. If any symptom is unexplained, keep going.

### 7. Fix at the cause, then prove it
- Fix the root cause, not the place where the error surfaced.
- Run the reproduction: it must now pass. Run the full suite: nothing else may break.
- Convert the reproduction into a permanent regression test with a comment referencing the issue.
- Remove temporary instrumentation, keep any logging that would have made this faster next time.
- Search for the same pattern elsewhere in the codebase (`rg`) and fix siblings.

## Output format

```
## Symptom
## Reproduction (command/test, failure rate)
## Hypotheses tried
1. X — ruled out by ...
2. Y — confirmed by ...
## Root cause
## Fix (files)
## Proof (test output before/after, full suite result)
## Follow-ups (sibling bugs, monitoring, docs)
```

## Checklist

- [ ] A failing, re-runnable reproduction exists before any code change.
- [ ] Each experiment changed one variable and its result was recorded.
- [ ] The root cause explains all symptoms.
- [ ] A regression test is committed with the fix.
- [ ] Instrumentation noise is removed.

## Pitfalls

- "It works on my machine" means the environment is a variable; diff it rather than dismissing it.
- Adding a retry, a `try/catch` or a `sleep` to make a symptom go away is not a fix; note it explicitly as a mitigation if you must ship it.
- Do not trust a fix that you could not make fail first.
- Beware of fixing the reproduction rather than the bug (e.g., special-casing the test input).
- Two bugs can produce one symptom; after the first fix, re-run the original report, not just the reduced case.
- Time-based and ordering bugs need the loop repeated many times after the fix, not once.