Golden-Master a Tangled Script Before the Cleanup Diff

A developer detailed a protocol for safely refactoring legacy scripts with AI assistance, emphasizing the creation of characterization tests and golden-master oracles before making any code changes. The approach involves recording current outputs, including stdout, stderr, exit codes, and file writes, to serve as a behavior contract. The developer provided a worked example with a Python script and proposed tools for capturing these golden outputs.

Do not clean a tangled script before you freeze outputs. Write characterization tests against today's actual messy behavior. Only then apply the smallest safe change. AI diffs make local edits cheap and frequent. They do not make observable behavior cheap to verify. A messy repo hides effects in prints, files, and globals. This article is a labeled worked example, not production history. The protocol stays useful without any coding assistant. Cheap model output does not replace a checked-in oracle. A typical messy script mixes calculation, I/O, and formatting. Helpers share one mutable dictionary across branches. Exit codes often depend on print order. Tests are missing, or they mock every call. An assistant then rewrites the whole file. The diff looks small and deceptively tidy. Downstream jobs then break on whitespace, paths, or codes. The missing artifact is a behavior oracle. The oracle is current output, not intended design. Work on one entrypoint at a time. Do not start the cleanup inside helpers. Capture four facts before any source edit. Record the entrypoint in a checked-in text file. Keep that file next to the characterization tests. Do not trust chat memory for this inventory. Proposed inventory commands for the sample script: python mkdir -p tests/golden git rev-parse HEAD git status --short python3 -c "import sys; print sys.version " Proposed tests/inventory/score jobs.txt contents: entrypoint: python3 score jobs.py jobs.csv reads: jobs.csv writes: $SCORE OUT/summary.txt, $SCORE OUT/failures.json stdout: one status line per job stderr: empty on the happy path exit: 0 if any job scored, 2 if none scored env: SCORE STRICT=1 treats unknown status as failure env: SCORE OUT selects the output directory Copy a tiny input next to the test. Do not copy production exports into the fixture. Keep the fixture rows ugly on purpose. Proposed fixtures/jobs.csv : id,status,weight a1,done,2 b2,UNKNOWN,1 c3,done,0 d4,failed,4 Ugly rows are the contract, not noise. Zero weights and unknown statuses encode real branches. Clean sample data hides the mess you must pin. Write a recorder before you write assertions. The recorder must run the real entrypoint. It dumps stdout, stderr, exit code, and files. Proposed tools/record score jobs oracle.py : """Record golden outputs for score jobs.py. Proposed example only.""" from future import annotations import os import subprocess import sys from pathlib import Path ROOT = Path file .resolve .parents 1 SCRIPT = ROOT / "score jobs.py" FIXTURE = ROOT / "fixtures" / "jobs.csv" GOLDEN = ROOT / "tests" / "golden" / "score jobs" def record label: str, env extra: dict str, str - None: tmp = ROOT / ".oracle-tmp" / label if tmp.exists : for child in tmp.rglob " " : if child.is file : child.unlink out dir = tmp / "out" out dir.mkdir parents=True, exist ok=True env = os.environ.copy env "SCORE OUT" = str out dir env.update env extra proc = subprocess.run sys.executable, str SCRIPT , str FIXTURE , cwd=tmp, env=env, text=True, capture output=True, check=False, dest = GOLDEN / label dest.mkdir parents=True, exist ok=True dest / "exit" .write text str proc.returncode dest / "stdout.txt" .write text proc.stdout dest / "stderr.txt" .write text proc.stderr summary = out dir / "summary.txt" failures = out dir / "failures.json" dest / "summary.txt" .write text summary.read text if summary.exists else "" dest / "failures.json" .write text failures.read text if failures.exists else "" if name == " main ": record "default", {} record "strict", {"SCORE STRICT": "1"} print "wrote", GOLDEN Run the recorder on an unchanged tree. Read every golden file by hand before commit. Commit those golden files as the behavior contract. python3 tools/record score jobs oracle.py git add tests/golden/score jobs fixtures/jobs.csv git status --short Do not re-record after a cleanup diff. Re-recording after a cleanup hides real regressions. Update goldens only after a product decision. The test reuses the same runner shape. It compares exact bytes, not review vibes. It should fail on a single space. Proposed tests/test score jobs oracle.py : """Characterization tests for score jobs.py. Proposed example only.""" from future import annotations import os import subprocess import sys from pathlib import Path import pytest ROOT = Path file .resolve .parents 1 SCRIPT = ROOT / "score jobs.py" FIXTURE = ROOT / "fixtures" / "jobs.csv" GOLDEN = ROOT / "tests" / "golden" / "score jobs" def run script tmp: Path, extra: dict str, str | None = None - dict: out dir = tmp / "out" out dir.mkdir env = os.environ.copy env "SCORE OUT" = str out dir if extra: env.update extra proc = subprocess.run sys.executable, str SCRIPT , str FIXTURE , cwd=tmp, env=env, text=True, capture output=True, check=False, summary = out dir / "summary.txt" failures = out dir / "failures.json" return { "exit": proc.returncode, "stdout": proc.stdout, "stderr": proc.stderr, "summary": summary.read text if summary.exists else "", "failures": failures.read text if failures.exists else "", } @pytest.mark.parametrize "label,extra", "default", None , "strict", {"SCORE STRICT": "1"} , def test path matches golden tmp path: Path, label: str, extra: dict | None - None: actual = run script tmp path, extra expected = GOLDEN / label assert actual "exit" == int expected / "exit" .read text assert actual "stdout" == expected / "stdout.txt" .read text assert actual "stderr" == expected / "stderr.txt" .read text assert actual "summary" == expected / "summary.txt" .read text assert actual "failures" == expected / "failures.json" .read text Run pytest on the harness before you touch score jobs.py. A red harness means the recorder and test disagree. Fix that mismatch before any refactor work. python3 -m pytest tests/test score jobs oracle.py -q Do not accept a refactor plan as prose. Classify every intended change against the oracle. | Change idea | Touches oracle? | First commit? | Next action | |---|---|---|---| | Rename a local variable | No | Yes | Apply after tests pass | | Extract a pure score helper | No, if prints stay | Yes | Keep I/O in main | | Reorder stdout lines | Yes | No | Reject or retarget product | | Change JSON indent or key order | Yes | No | Freeze json.dumps as-is | | Move file writes into a helper | Maybe | No | Second commit, same goldens | | Drop the unknown-status branch | Yes | No | Needs an explicit spec test | | Add type hints only | No | Yes | Keep runtime identical | The first cleanup commit may only include off-oracle rows. If a row flips on, split the work. Do not bargain with the table in chat. Here is the proposed messy module for this walkthrough. Treat the file as unlabeled sample code. python score jobs.py — proposed messy entrypoint import csv import json import os import sys def main : path = sys.argv 1 out = os.environ.get "SCORE OUT", "out" strict = os.environ.get "SCORE STRICT" == "1" os.makedirs out, exist ok=True rows = list csv.DictReader open path scores = failures = total = 0 for row in rows: status = row "status" weight = int row "weight" if status == "done": s = weight 10 scores.append row "id" , s total += s print "ok", row "id" , s elif status == "failed": failures.append row print "fail", row "id" else: if strict: failures.append row print "fail", row "id" , "unknown" else: print "skip", row "id" , status open os.path.join out, "summary.txt" , "w" .write str total + "\n" open os.path.join out, "failures.json" , "w" .write json.dumps failures sys.exit 0 if scores else 2 if name == " main ": main The smallest safe change extracts the arithmetic only. Leave prints and file writes inside main. php def score done weight: int - int: return weight 10 Replace s = weight 10 with s = score done weight . Run the golden tests after that single replace. Stop if stdout, files, and exit still match. Do not extract main in the same commit. Do not introduce a class for taste. Do not pretty-print the JSON payload yet. Those edits need their own rows in the table. A model can draft the recorder from the inventory file. It can also propose the one-line extract. It cannot own the oracle or the table. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Paste the inventory, the fixture, and the messy file. Ask for a characterization harness, not a rewrite. Keep these rules with any assistant: The free server is optional for this protocol. A local pytest run remains the source of truth. If the model output disagrees with goldens, keep the goldens. Golden masters pin bugs as well as features. That is the method, not a defect. They fail on timestamps, random IDs, and unordered sets. This method does not prove functional correctness by itself. It only proves stability of current edges. Product intent still needs explicit tests later. Large binary outputs do not belong in git goldens. Store a sha256 of those files instead. Live network calls need a recorded fixture. Never hit the live network during these tests. Line endings and locale will break naive string compares. Normalize newlines in the recorder if your team mixes OS images. Do not normalize away spaces that operators already depend on. Skip this protocol for greenfield code with a written spec. Skip it when current behavior is unsafe or destructive. Skip it when snapshots would store secrets. Do not use a model to fix failing goldens. That choice hides the regression you needed to see. Do not batch five extracts into one assistant diff. The table exists to stop that collapse. Tag the commit as a characterization baseline. Keep the inventory file on that commit. The next extract starts from the same goldens. If later work must change stdout, add a spec test first. Then update goldens in a dedicated commit. Never mix format changes with logic changes. Cheap generation does not retire this test sequence. Cheap edits make the sequence more necessary. Messy repos fail at the observable edges. Pin those edges, then cut one line.