{"slug": "golden-master-a-tangled-script-before-the-cleanup-diff", "title": "Golden-Master a Tangled Script Before the Cleanup Diff", "summary": "A developer detailed a protocol for safely refactoring legacy scripts with AI assistance, emphasizing the creation of characterization tests and golden-master oracles before making any code changes. The approach involves recording current outputs, including stdout, stderr, exit codes, and file writes, to serve as a behavior contract. The developer provided a worked example with a Python script and proposed tools for capturing these golden outputs.", "body_md": "Do not clean a tangled script before you freeze outputs. Write characterization tests against today's actual messy behavior. Only then apply the smallest safe change.\n\nAI diffs make local edits cheap and frequent. They do not make observable behavior cheap to verify. A messy repo hides effects in prints, files, and globals.\n\nThis article is a labeled worked example, not production history. The protocol stays useful without any coding assistant. Cheap model output does not replace a checked-in oracle.\n\nA typical messy script mixes calculation, I/O, and formatting. Helpers share one mutable dictionary across branches. Exit codes often depend on print order.\n\nTests are missing, or they mock every call. An assistant then rewrites the whole file. The diff looks small and deceptively tidy.\n\nDownstream jobs then break on whitespace, paths, or codes. The missing artifact is a behavior oracle. The oracle is current output, not intended design.\n\nWork on one entrypoint at a time. Do not start the cleanup inside helpers. Capture four facts before any source edit.\n\nRecord the entrypoint in a checked-in text file. Keep that file next to the characterization tests. Do not trust chat memory for this inventory.\n\nProposed inventory commands for the sample script:\n\n``` python\nmkdir -p tests/golden\ngit rev-parse HEAD\ngit status --short\npython3 -c \"import sys; print(sys.version)\"\n```\n\nProposed `tests/inventory/score_jobs.txt`\n\ncontents:\n\n```\nentrypoint: python3 score_jobs.py jobs.csv\nreads: jobs.csv\nwrites: $SCORE_OUT/summary.txt, $SCORE_OUT/failures.json\nstdout: one status line per job\nstderr: empty on the happy path\nexit: 0 if any job scored, 2 if none scored\nenv: SCORE_STRICT=1 treats unknown status as failure\nenv: SCORE_OUT selects the output directory\n```\n\nCopy a tiny input next to the test. Do not copy production exports into the fixture. Keep the fixture rows ugly on purpose.\n\nProposed `fixtures/jobs.csv`\n\n:\n\n```\nid,status,weight\na1,done,2\nb2,UNKNOWN,1\nc3,done,0\nd4,failed,4\n```\n\nUgly rows are the contract, not noise. Zero weights and unknown statuses encode real branches. Clean sample data hides the mess you must pin.\n\nWrite a recorder before you write assertions. The recorder must run the real entrypoint. It dumps stdout, stderr, exit code, and files.\n\nProposed `tools/record_score_jobs_oracle.py`\n\n:\n\n```\n\"\"\"Record golden outputs for score_jobs.py. Proposed example only.\"\"\"\nfrom __future__ import annotations\n\nimport os\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nROOT = Path(__file__).resolve().parents[1]\nSCRIPT = ROOT / \"score_jobs.py\"\nFIXTURE = ROOT / \"fixtures\" / \"jobs.csv\"\nGOLDEN = ROOT / \"tests\" / \"golden\" / \"score_jobs\"\n\ndef record(label: str, env_extra: dict[str, str]) -> None:\n    tmp = ROOT / \".oracle-tmp\" / label\n    if tmp.exists():\n        for child in tmp.rglob(\"*\"):\n            if child.is_file():\n                child.unlink()\n    out_dir = tmp / \"out\"\n    out_dir.mkdir(parents=True, exist_ok=True)\n    env = os.environ.copy()\n    env[\"SCORE_OUT\"] = str(out_dir)\n    env.update(env_extra)\n    proc = subprocess.run(\n        [sys.executable, str(SCRIPT), str(FIXTURE)],\n        cwd=tmp,\n        env=env,\n        text=True,\n        capture_output=True,\n        check=False,\n    )\n    dest = GOLDEN / label\n    dest.mkdir(parents=True, exist_ok=True)\n    (dest / \"exit\").write_text(str(proc.returncode))\n    (dest / \"stdout.txt\").write_text(proc.stdout)\n    (dest / \"stderr.txt\").write_text(proc.stderr)\n    summary = out_dir / \"summary.txt\"\n    failures = out_dir / \"failures.json\"\n    (dest / \"summary.txt\").write_text(\n        summary.read_text() if summary.exists() else \"\"\n    )\n    (dest / \"failures.json\").write_text(\n        failures.read_text() if failures.exists() else \"\"\n    )\n\nif __name__ == \"__main__\":\n    record(\"default\", {})\n    record(\"strict\", {\"SCORE_STRICT\": \"1\"})\n    print(\"wrote\", GOLDEN)\n```\n\nRun the recorder on an unchanged tree. Read every golden file by hand before commit. Commit those golden files as the behavior contract.\n\n```\npython3 tools/record_score_jobs_oracle.py\ngit add tests/golden/score_jobs fixtures/jobs.csv\ngit status --short\n```\n\nDo not re-record after a cleanup diff. Re-recording after a cleanup hides real regressions. Update goldens only after a product decision.\n\nThe test reuses the same runner shape. It compares exact bytes, not review vibes. It should fail on a single space.\n\nProposed `tests/test_score_jobs_oracle.py`\n\n:\n\n```\n\"\"\"Characterization tests for score_jobs.py. Proposed example only.\"\"\"\nfrom __future__ import annotations\n\nimport os\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nimport pytest\n\nROOT = Path(__file__).resolve().parents[1]\nSCRIPT = ROOT / \"score_jobs.py\"\nFIXTURE = ROOT / \"fixtures\" / \"jobs.csv\"\nGOLDEN = ROOT / \"tests\" / \"golden\" / \"score_jobs\"\n\ndef run_script(tmp: Path, extra: dict[str, str] | None = None) -> dict:\n    out_dir = tmp / \"out\"\n    out_dir.mkdir()\n    env = os.environ.copy()\n    env[\"SCORE_OUT\"] = str(out_dir)\n    if extra:\n        env.update(extra)\n    proc = subprocess.run(\n        [sys.executable, str(SCRIPT), str(FIXTURE)],\n        cwd=tmp,\n        env=env,\n        text=True,\n        capture_output=True,\n        check=False,\n    )\n    summary = out_dir / \"summary.txt\"\n    failures = out_dir / \"failures.json\"\n    return {\n        \"exit\": proc.returncode,\n        \"stdout\": proc.stdout,\n        \"stderr\": proc.stderr,\n        \"summary\": summary.read_text() if summary.exists() else \"\",\n        \"failures\": failures.read_text() if failures.exists() else \"\",\n    }\n\n@pytest.mark.parametrize(\n    \"label,extra\",\n    [(\"default\", None), (\"strict\", {\"SCORE_STRICT\": \"1\"})],\n)\ndef test_path_matches_golden(tmp_path: Path, label: str, extra: dict | None) -> None:\n    actual = run_script(tmp_path, extra)\n    expected = GOLDEN / label\n    assert actual[\"exit\"] == int((expected / \"exit\").read_text())\n    assert actual[\"stdout\"] == (expected / \"stdout.txt\").read_text()\n    assert actual[\"stderr\"] == (expected / \"stderr.txt\").read_text()\n    assert actual[\"summary\"] == (expected / \"summary.txt\").read_text()\n    assert actual[\"failures\"] == (expected / \"failures.json\").read_text()\n```\n\nRun pytest on the harness before you touch score_jobs.py. A red harness means the recorder and test disagree. Fix that mismatch before any refactor work.\n\n```\npython3 -m pytest tests/test_score_jobs_oracle.py -q\n```\n\nDo not accept a refactor plan as prose. Classify every intended change against the oracle.\n\n| Change idea | Touches oracle? | First commit? | Next action |\n|---|---|---|---|\n| Rename a local variable | No | Yes | Apply after tests pass |\n| Extract a pure score helper | No, if prints stay | Yes | Keep I/O in `main`\n|\n| Reorder stdout lines | Yes | No | Reject or retarget product |\n| Change JSON indent or key order | Yes | No | Freeze `json.dumps` as-is |\n| Move file writes into a helper | Maybe | No | Second commit, same goldens |\n| Drop the unknown-status branch | Yes | No | Needs an explicit spec test |\n| Add type hints only | No | Yes | Keep runtime identical |\n\nThe first cleanup commit may only include off-oracle rows. If a row flips on, split the work. Do not bargain with the table in chat.\n\nHere is the proposed messy module for this walkthrough. Treat the file as unlabeled sample code.\n\n``` python\n# score_jobs.py — proposed messy entrypoint\nimport csv\nimport json\nimport os\nimport sys\n\ndef main():\n    path = sys.argv[1]\n    out = os.environ.get(\"SCORE_OUT\", \"out\")\n    strict = os.environ.get(\"SCORE_STRICT\") == \"1\"\n    os.makedirs(out, exist_ok=True)\n    rows = list(csv.DictReader(open(path)))\n    scores = []\n    failures = []\n    total = 0\n    for row in rows:\n        status = row[\"status\"]\n        weight = int(row[\"weight\"])\n        if status == \"done\":\n            s = weight * 10\n            scores.append((row[\"id\"], s))\n            total += s\n            print(\"ok\", row[\"id\"], s)\n        elif status == \"failed\":\n            failures.append(row)\n            print(\"fail\", row[\"id\"])\n        else:\n            if strict:\n                failures.append(row)\n                print(\"fail\", row[\"id\"], \"unknown\")\n            else:\n                print(\"skip\", row[\"id\"], status)\n    open(os.path.join(out, \"summary.txt\"), \"w\").write(str(total) + \"\\n\")\n    open(os.path.join(out, \"failures.json\"), \"w\").write(json.dumps(failures))\n    sys.exit(0 if scores else 2)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThe smallest safe change extracts the arithmetic only. Leave prints and file writes inside main.\n\n``` php\ndef score_done(weight: int) -> int:\n    return weight * 10\n```\n\nReplace `s = weight * 10`\n\nwith `s = score_done(weight)`\n\n. Run the golden tests after that single replace. Stop if stdout, files, and exit still match.\n\nDo not extract `main`\n\nin the same commit. Do not introduce a class for taste. Do not pretty-print the JSON payload yet. Those edits need their own rows in the table.\n\nA model can draft the recorder from the inventory file. It can also propose the one-line extract. It cannot own the oracle or the table.\n\nMonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Paste the inventory, the fixture, and the messy file. Ask for a characterization harness, not a rewrite.\n\nKeep these rules with any assistant:\n\nThe free server is optional for this protocol. A local pytest run remains the source of truth. If the model output disagrees with goldens, keep the goldens.\n\nGolden masters pin bugs as well as features. That is the method, not a defect. They fail on timestamps, random IDs, and unordered sets.\n\nThis method does not prove functional correctness by itself. It only proves stability of current edges. Product intent still needs explicit tests later.\n\nLarge binary outputs do not belong in git goldens. Store a sha256 of those files instead. Live network calls need a recorded fixture. Never hit the live network during these tests.\n\nLine endings and locale will break naive string compares. Normalize newlines in the recorder if your team mixes OS images. Do not normalize away spaces that operators already depend on.\n\nSkip this protocol for greenfield code with a written spec. Skip it when current behavior is unsafe or destructive. Skip it when snapshots would store secrets.\n\nDo not use a model to fix failing goldens. That choice hides the regression you needed to see. Do not batch five extracts into one assistant diff. The table exists to stop that collapse.\n\nTag the commit as a characterization baseline. Keep the inventory file on that commit. The next extract starts from the same goldens.\n\nIf later work must change stdout, add a spec test first. Then update goldens in a dedicated commit. Never mix format changes with logic changes.\n\nCheap generation does not retire this test sequence. Cheap edits make the sequence more necessary. Messy repos fail at the observable edges. Pin those edges, then cut one line.", "url": "https://wpnews.pro/news/golden-master-a-tangled-script-before-the-cleanup-diff", "canonical_source": "https://dev.to/hackrs_6393/golden-master-a-tangled-script-before-the-cleanup-diff-45lo", "published_at": "2026-09-03 17:38:13+00:00", "updated_at": "2026-09-03 17:56:11.210244+00:00", "lang": "en", "topics": ["developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/golden-master-a-tangled-script-before-the-cleanup-diff", "markdown": "https://wpnews.pro/news/golden-master-a-tangled-script-before-the-cleanup-diff.md", "text": "https://wpnews.pro/news/golden-master-a-tangled-script-before-the-cleanup-diff.txt", "jsonld": "https://wpnews.pro/news/golden-master-a-tangled-script-before-the-cleanup-diff.jsonld"}}