Pin the Fault Matrix Before You Refactor a Messy Module

Consider an illustrative composite scenario rather than one specific incident, because the shape repeats across review threads. A service module grows past nine hundred lines, a coding agent returns a four hundred line refactor diff, and the existing suite turns green on the first run. The diff merges, and days later a partner integration fails, because one validation path now raises a bare ValueError where callers expected a typed error with a stable code.

Recent community threads argue loudly about how fast agent-authored diffs arrive, and much less about what those diffs quietly change. The piece that changes first is usually not the happy path but the error surface: exception types, error codes, and the order in which validation rejects a payload. This post pins that surface as data before a messy refactor starts, then constrains the change to the smallest seam available.

Happy paths get covered because they are easy to observe, cheap to assert, and pleasant to demo in a pull request. Error paths stay thin because each one needs a deliberate fault injection, and their assertions tend to encode prose instead of structure. When a refactor moves validation across layers, the happy path usually survives the move and the exception taxonomy does not.

Three patterns account for most of the drift reviewers see:

ValueError or Exception once a helper is extracted.code attribute that callers branch on. All three pass a suite whose only error assertion is "something raised".

Turn the current error surface into a list you can diff, using nothing beyond the standard library. ast.unparse requires Python 3.9 or newer.

"""Print every raise site in a package as file:line:expression."""
import ast
import sys
from pathlib import Path

def raise_sites(path: Path):
    tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))
    for node in ast.walk(tree):
        if isinstance(node, ast.Raise):
            if node.exc is None:
                name = 'bare-raise'
            elif isinstance(node.exc, ast.Call):
                name = ast.unparse(node.exc.func)
            else:
                name = ast.unparse(node.exc)
            yield path, node.lineno, name

def main(root: str) -> int:
    rows = []
    for path in sorted(Path(root).rglob('*.py')):
        if '.venv' in path.parts:
            continue
        rows.extend(raise_sites(path))
    for path, line, name in rows:
        print(f'{path}:{line}:{name}')
    print(f'\n{len(rows)} raise sites')
    return 0

if __name__ == '__main__':
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else 'src'))

Run it once and commit the output as a baseline artifact, then run it again after the refactor.

python tools/raise_sites.py src > .refactor/raise_before.txt
grep -c . .refactor/raise_before.txt   # example baseline: 47

The raw count is not a quality score, and it should never be optimized for its own sake. What matters is that the diff tells you exactly which raises moved, appeared, or disappeared during the change.

Forty-seven raise sites normally collapse into a handful of distinct faults, and the faults are what tests should pin. A shipping rate card gives a compact example.

Four rules keep the matrix worth maintaining over several refactors:

Keep the tests shaped like the table, so adding one fault stays a one-line change.

import pytest
from pricing.rate_card import (
    CurrencyMismatchError,
    MissingWeightError,
    OverweightError,
    RateCard,
    UnknownZoneError,
)

FAULTS = [
    ('Z1', {'zone': 'ZZ', 'weight': 1.0, 'currency': 'USD'}, UnknownZoneError, 'unknown_zone'),
    ('W1', {'zone': 'US', 'weight': None, 'currency': 'USD'}, MissingWeightError, 'missing_weight'),
    ('C1', {'zone': 'US', 'weight': 1.0, 'currency': 'EUR'}, CurrencyMismatchError, 'currency_mismatch'),
    ('O1', {'zone': 'US', 'weight': 99.0, 'currency': 'USD'}, OverweightError, 'overweight'),
]

@pytest.mark.parametrize('case_id,payload,exc,code', FAULTS, ids=[f[0] for f in FAULTS])
def test_error_surface(case_id, payload, exc, code):
    with pytest.raises(exc) as caught:
        RateCard.load().quote(payload)
    assert caught.value.code == code

Parametrized ids keep the output readable, so pytest -q tests/test_error_surface.py reports Z1, W1, C1, and O1 as separate lines. Add one structural guard as well: assert that every exception class in the matrix exposes a non-empty code attribute, which stops a new raise site from shipping without a machine-readable identifier.

With the matrix frozen, give the refactor an explicit churn budget and pick exactly one seam inside it.

git diff --numstat origin/main...HEAD | awk '{a+=$1; d+=$2} END {print a+d " changed lines"}'

A seam worth taking has few call sites, no I/O, and error paths already represented in the matrix. Everything else waits for a second pass with its own baseline.

Wrap the workflow in one script that CI or a human can run against the branch.

#!/usr/bin/env bash
set -euo pipefail
BASE=${1:-origin/main}
BUDGET=${2:-40}

python tools/raise_sites.py src > /tmp/raise_after.txt
diff -u .refactor/raise_before.txt /tmp/raise_after.txt || true

pytest -q tests/test_error_surface.py tests/test_value_characterization.py

CHURN=$(git diff --numstat "$BASE"...HEAD | awk '{a+=$1; d+=$2} END {print a+d}')
echo churn=$CHURN budget=$BUDGET
test "$CHURN" -le "$BUDGET"

The raise-site diff stays informational, while the two hard gates are the error-surface tests and the churn limit. Treat any new raise site as a contract change: either add a matrix row in the same commit, or move the raise instead of introducing it.

Drafting matrix rows across dozens of raise sites is mechanical work, which is the kind of task a coding agent can genuinely accelerate. This workflow uses MonkeyCode for that step, where the free model access covers the drafting and the free server option gives the gate somewhere to run without local provisioning. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Scope the prompt tightly, and ask the agent to read .refactor/raise_before.txt, cluster the raise sites by exception name and payload condition, propose one matrix row per cluster with a candidate code, and then stop. Verify every proposed code against one real runtime call before freezing it, because the agent cannot know your taxonomy intent.

Three things the agent should never be trusted with:

If you want to try the gate end to end without provisioning a machine, the free model access and free server option are enough to run the script, the tests, and the churn check.

Messy refactors rarely fail because the happy path changed, and they fail quietly because the error contract moved. Freeze the fault matrix first, take one seam with a churn budget, and let the gate make "the tests still pass" mean something specific about behavior.