{"slug": "pin-the-fault-matrix-before-you-refactor-a-messy-module", "title": "Pin the Fault Matrix Before You Refactor a Messy Module", "summary": "A developer outlines a technique for protecting a module's error surface during agent-assisted refactors by extracting every raise site with a standard-library Python AST script (tools/raise_sites.py) and committing the output as a diffable baseline before the change. The approach maps the roughly 47 raise sites in a typical module onto a small fault matrix of injected faults, expected exception types, and stable error codes, so reviewers can see exactly which raises moved, appeared, or disappeared.", "body_md": "Consider an illustrative composite scenario rather than one specific incident, because the shape repeats across review threads. A service module grows past nine hundred lines, a coding agent returns a four hundred line refactor diff, and the existing suite turns green on the first run. The diff merges, and days later a partner integration fails, because one validation path now raises a bare `ValueError` where callers expected a typed error with a stable code.\n\nRecent community threads argue loudly about how fast agent-authored diffs arrive, and much less about what those diffs quietly change. The piece that changes first is usually not the happy path but the error surface: exception types, error codes, and the order in which validation rejects a payload. This post pins that surface as data before a messy refactor starts, then constrains the change to the smallest seam available.\n\nHappy paths get covered because they are easy to observe, cheap to assert, and pleasant to demo in a pull request. Error paths stay thin because each one needs a deliberate fault injection, and their assertions tend to encode prose instead of structure. When a refactor moves validation across layers, the happy path usually survives the move and the exception taxonomy does not.\n\nThree patterns account for most of the drift reviewers see:\n\n`ValueError` or `Exception` once a helper is extracted.`code` attribute that callers branch on.\nAll three pass a suite whose only error assertion is \"something raised\".\n\nTurn the current error surface into a list you can diff, using nothing beyond the standard library. `ast.unparse` requires Python 3.9 or newer.\n\n```\n# tools/raise_sites.py\n\"\"\"Print every raise site in a package as file:line:expression.\"\"\"\nimport ast\nimport sys\nfrom pathlib import Path\n\ndef raise_sites(path: Path):\n    tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))\n    for node in ast.walk(tree):\n        if isinstance(node, ast.Raise):\n            if node.exc is None:\n                name = 'bare-raise'\n            elif isinstance(node.exc, ast.Call):\n                name = ast.unparse(node.exc.func)\n            else:\n                name = ast.unparse(node.exc)\n            yield path, node.lineno, name\n\ndef main(root: str) -> int:\n    rows = []\n    for path in sorted(Path(root).rglob('*.py')):\n        if '.venv' in path.parts:\n            continue\n        rows.extend(raise_sites(path))\n    for path, line, name in rows:\n        print(f'{path}:{line}:{name}')\n    print(f'\\n{len(rows)} raise sites')\n    return 0\n\nif __name__ == '__main__':\n    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else 'src'))\n```\n\nRun it once and commit the output as a baseline artifact, then run it again after the refactor.\n\n```\npython tools/raise_sites.py src > .refactor/raise_before.txt\ngrep -c . .refactor/raise_before.txt   # example baseline: 47\n```\n\nThe raw count is not a quality score, and it should never be optimized for its own sake. What matters is that the diff tells you exactly which raises moved, appeared, or disappeared during the change.\n\nForty-seven raise sites normally collapse into a handful of distinct faults, and the faults are what tests should pin. A shipping rate card gives a compact example.\n\n| ID | Injected fault | Expected exception | Stable code | Prose fragment | \n|---|---|---|---|---|\n| Z1 | zone missing from the rate card | `UnknownZoneError` | `unknown_zone` | \"unknown zone\" | \n| W1 | weight absent or `None` | `MissingWeightError` | `missing_weight` | \"weight\" | \n| C1 | parcel currency differs from card | `CurrencyMismatchError` | `currency_mismatch` | \"currency\" | \n| O1 | weight above the carrier maximum | `OverweightError` | `overweight` | \"maximum\" | \n\nFour rules keep the matrix worth maintaining over several refactors:\n\nKeep the tests shaped like the table, so adding one fault stays a one-line change.\n\n``` python\n# tests/test_error_surface.py\nimport pytest\nfrom pricing.rate_card import (\n    CurrencyMismatchError,\n    MissingWeightError,\n    OverweightError,\n    RateCard,\n    UnknownZoneError,\n)\n\n# (id, payload, exception class, stable code)\nFAULTS = [\n    ('Z1', {'zone': 'ZZ', 'weight': 1.0, 'currency': 'USD'}, UnknownZoneError, 'unknown_zone'),\n    ('W1', {'zone': 'US', 'weight': None, 'currency': 'USD'}, MissingWeightError, 'missing_weight'),\n    ('C1', {'zone': 'US', 'weight': 1.0, 'currency': 'EUR'}, CurrencyMismatchError, 'currency_mismatch'),\n    ('O1', {'zone': 'US', 'weight': 99.0, 'currency': 'USD'}, OverweightError, 'overweight'),\n]\n\n@pytest.mark.parametrize('case_id,payload,exc,code', FAULTS, ids=[f[0] for f in FAULTS])\ndef test_error_surface(case_id, payload, exc, code):\n    with pytest.raises(exc) as caught:\n        RateCard.load().quote(payload)\n    assert caught.value.code == code\n```\n\nParametrized ids keep the output readable, so `pytest -q tests/test_error_surface.py` reports `Z1`, `W1`, `C1`, and `O1` as separate lines. Add one structural guard as well: assert that every exception class in the matrix exposes a non-empty `code` attribute, which stops a new raise site from shipping without a machine-readable identifier.\n\nWith the matrix frozen, give the refactor an explicit churn budget and pick exactly one seam inside it.\n\n```\ngit diff --numstat origin/main...HEAD | awk '{a+=$1; d+=$2} END {print a+d \" changed lines\"}'\n```\n\n| Candidate seam | Call sites | In/out types | Side effects | Error paths | Verdict | \n|---|---|---|---|---|---|\n| `quote()` body | 3 | dict to Money | none | 4 | extract one helper | \n| currency formatting | 11 | str to str | none | 1 | defer | \n| carrier client wrapper | 6 | network | I/O | 6 | do not touch | \n| rate cache | 2 | dict | TTL clock | 2 | defer | \n\nA seam worth taking has few call sites, no I/O, and error paths already represented in the matrix. Everything else waits for a second pass with its own baseline.\n\nWrap the workflow in one script that CI or a human can run against the branch.\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\nBASE=${1:-origin/main}\nBUDGET=${2:-40}\n\npython tools/raise_sites.py src > /tmp/raise_after.txt\ndiff -u .refactor/raise_before.txt /tmp/raise_after.txt || true\n\npytest -q tests/test_error_surface.py tests/test_value_characterization.py\n\nCHURN=$(git diff --numstat \"$BASE\"...HEAD | awk '{a+=$1; d+=$2} END {print a+d}')\necho churn=$CHURN budget=$BUDGET\ntest \"$CHURN\" -le \"$BUDGET\"\n```\n\nThe raise-site diff stays informational, while the two hard gates are the error-surface tests and the churn limit. Treat any new raise site as a contract change: either add a matrix row in the same commit, or move the raise instead of introducing it.\n\nDrafting matrix rows across dozens of raise sites is mechanical work, which is the kind of task a coding agent can genuinely accelerate. This workflow uses MonkeyCode for that step, where the free model access covers the drafting and the free server option gives the gate somewhere to run without local provisioning. *Disclosure: This article was prepared as part of MonkeyCode's product outreach.*\n\nScope the prompt tightly, and ask the agent to read `.refactor/raise_before.txt`, cluster the raise sites by exception name and payload condition, propose one matrix row per cluster with a candidate code, and then stop. Verify every proposed code against one real runtime call before freezing it, because the agent cannot know your taxonomy intent.\n\nThree things the agent should never be trusted with:\n\nIf you want to try the gate end to end without provisioning a machine, the free model access and free server option are enough to run the script, the tests, and the churn check.\n\nMessy refactors rarely fail because the happy path changed, and they fail quietly because the error contract moved. Freeze the fault matrix first, take one seam with a churn budget, and let the gate make \"the tests still pass\" mean something specific about behavior.", "url": "https://wpnews.pro/news/pin-the-fault-matrix-before-you-refactor-a-messy-module", "canonical_source": "https://dev.to/webx_2736/pin-the-fault-matrix-before-you-refactor-a-messy-module-3a1f", "published_at": "2026-10-15 11:16:22+00:00", "updated_at": "2026-10-15 11:19:24.605673+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Python", "ast.unparse"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pin-the-fault-matrix-before-you-refactor-a-messy-module", "markdown": "https://wpnews.pro/news/pin-the-fault-matrix-before-you-refactor-a-messy-module.md", "text": "https://wpnews.pro/news/pin-the-fault-matrix-before-you-refactor-a-messy-module.txt", "jsonld": "https://wpnews.pro/news/pin-the-fault-matrix-before-you-refactor-a-messy-module.jsonld"}}