Snapshot File Deltas Before You Extract Glue

A developer proposes a characterization-testing approach for refactoring untested 'glue' code that writes files, reads environment variables, or changes directories. The method records a frozen ledger of process behavior—exit code, stdout, stderr, file deltas, and environment keys—before any extraction, to prevent hidden side effects from breaking behavior. The post includes example code for a report script and a harness to capture such ledgers.

Do not extract glue until side effects are pinned. Messy modules leak behavior through files, env, and cwd. A function signature is not a characterization test. AI diffs look clean when tests are missing. Glue code still writes caches, logs, and temp files. Reviewers then ship a rename that changes process behavior. Cheap code generation does not reduce the characterization cost. Technical debt remains in implicit state, not syntax. You need a ledger before the first extract. Untested glue often reads hidden config from the process. It may chdir, mkdir, or rewrite a lockfile. Those actions never appear in the public return value. Return values also miss mkdir side effects on empty input. Missing directories can change later commands in the same job. That is still user-visible behavior, not an internal detail. A ledger is a frozen record of process behavior. It is not a unit test of internal helpers. It pins observable results for a fixed fixture pack. Record five observable channels on every fixture run. Capture exit code, stdout, stderr, file deltas, and env keys. Ignore wall-clock time unless the contract requires it. Hash file contents, not only the file names. Store relative paths instead of absolute sandbox paths. Drop volatile fields like pid, dates, and hostnames. Normalize stdout by stripping any sandbox path prefix. Keep the newline style identical across all runs. Sort JSON object keys before hashing cache files. Consider a report glue script with no tests. It reads CSV from stdin and writes a cache file. It also prints a summary line and an exit code. The listing below is a labeled example, not production code. It mixes parsing, caching, and printing in one function. That mix is the core extract hazard. python proposal: messy glue, not a live codebase import csv, hashlib, json, os, sys CACHE = os.environ.get "REPORT CACHE", ".cache/report.json" def run : os.makedirs os.path.dirname CACHE or ".", exist ok=True rows = list csv.DictReader sys.stdin total = sum int r.get "amount" or 0 for r in rows payload = {"count": len rows , "total": total} digest = hashlib.sha256 json.dumps payload, sort keys=True .encode .hexdigest :12 with open CACHE, "w", encoding="utf-8" as f: json.dump { payload, "digest": digest}, f print f"rows={payload 'count' } total={payload 'total' } digest={digest}" return 0 if name == " main ": raise SystemExit run Do not split the run function just yet. First pin what the whole process does. Then change one seam only after that pin. The harness below is a proposal you can copy. It runs one command inside a temp workspace. It writes a stable JSON ledger for later diffing. python proposal: characterization harness, unexecuted here import hashlib, json, os, shutil, subprocess, tempfile from pathlib import Path VOLATILE ENV = {"PWD", "OLDPWD", "SHLVL", "SSH AUTH SOCK", "TERM"} def hash tree root: Path - dict: out = {} for path in sorted root.rglob " " : if path.is file : rel = str path.relative to root .replace "\\", "/" out rel = hashlib.sha256 path.read bytes .hexdigest return out def capture cmd, stdin bytes, extra env : sandbox = Path tempfile.mkdtemp prefix="ledger-" try: env = { key: value for key, value in os.environ.items if key not in VOLATILE ENV } env.update extra env proc = subprocess.run cmd, input=stdin bytes, cwd=sandbox, env=env, capture output=True, return { "exit": proc.returncode, "stdout": proc.stdout.decode "utf-8", "replace" , "stderr": proc.stderr.decode "utf-8", "replace" , "files": hash tree sandbox , "env subset": {key: env.get key for key in sorted extra env }, } finally: shutil.rmtree sandbox, ignore errors=True def write ledger path, cases : Path path .write text json.dumps cases, indent=2, sort keys=True + "\n" Add a tiny fixture runner next to the harness. Keep the fixtures boring, small, and fully deterministic. Three fixture cases beat one lucky happy path. python proposal: fixture pack for report.py import sys from pathlib import Path REPORT = str Path "report.py" .resolve CASES = { "name": "empty stdin", "stdin": b"", "env": {"REPORT CACHE": "out/report.json"}, }, { "name": "two rows", "stdin": b"amount\n10\n5\n", "env": {"REPORT CACHE": "out/report.json"}, }, { "name": "missing amount", "stdin": b"name\nalice\n", "env": {"REPORT CACHE": "out/report.json"}, }, def main out path : cmd = "python", REPORT ledgers = {} for case in CASES: ledgers case "name" = capture cmd, case "stdin" , case "env" write ledger out path, ledgers if name == " main ": main sys.argv 1 if len sys.argv 1 else "ledger.before.json" Store the before ledger inside the git history. Treat that committed file as a characterization oracle. Diff the ledger file after every candidate extract. Follow these seven steps in strict order. Skip none of these steps on messy glue. Inventory the process contract in one short note. List stdin, env keys, output files, and exit codes. Do not list internal helper names just yet. Write the note as a bullet list in the PR. Name each env key and its default value. Name each file path the process may create. Build a fixture pack from real sample inputs. Prefer anonymized production-like rows over synthetic ones. Keep each fixture file under a few kilobytes. Include empty stdin, two valid rows, and a missing column. Do not include live hostnames or access tokens. Redact any customer field before the file lands. Run the harness and commit the before ledger. Confirm the empty, happy, and malformed cases exist. Confirm that file hashes are relative and stable. Commit harness code with the ledger in one snapshot. That pair is the oracle for later diffs. Do not fold extra formatting into this snapshot. Freeze the git repo at that ledger commit. Do not format, rename, or extract in the same commit. The ledger commit must contain zero behavior edits. Tag the commit if your team likes movable pointers. A local branch name is enough for a solo run. The point is a restore target after a bad extract. Propose the smallest extract that preserves the ledger. One function move is enough for this step. Do not relocate caches or change env names. Move only the cache write into a helper. Leave CSV parsing inside the original run function. Leave stdout formatting inside the original run function. python proposal: smallest extract after the ledger pin import csv, hashlib, json, os, sys CACHE = os.environ.get "REPORT CACHE", ".cache/report.json" def write cache path, record : os.makedirs os.path.dirname path or ".", exist ok=True with open path, "w", encoding="utf-8" as handle: json.dump record, handle def run : rows = list csv.DictReader sys.stdin payload = { "count": len rows , "total": sum int r.get "amount" or 0 for r in rows , } digest = hashlib.sha256 json.dumps payload, sort keys=True .encode .hexdigest :12 write cache CACHE, { payload, "digest": digest} print f"rows={payload 'count' } total={payload 'total' } digest={digest}" return 0 Re-run the same harness into the after ledger. Diff the two JSON files with a structured tool. Any extra JSON key is a failed characterization. python harness.py ledger.before.json git add ledger.before.json report.py harness.py git commit -m "test: pin report glue side-effect ledger" smallest extract only, after the pin commit python harness.py ledger.after.json python - <<'PY' import json from pathlib import Path before = json.loads Path "ledger.before.json" .read text after = json.loads Path "ledger.after.json" .read text print "match" if before == after else "mismatch" if before = after: for name in sorted set before | set after : if before.get name = after.get name : print "case", name PY Keep the extract only when the ledgers match. Revert the whole extract on the first mismatch. Do not "fix forward" inside the same change. A later commit may fix a real product bug. That fix needs new fixtures and a new pin. Do not hide a bugfix inside a rename. Use this table before you accept a diff. | Signal | Keep the extract | Revert the extract | |---|---|---| | Exit codes | Identical per fixture | Any fixture changed | | stdout / stderr | Byte-stable after path rewrite | New warnings or missing lines | | File set | Same relative paths | Extra cache or dropped file | | File hashes | Same digest per path | Any digest changed | | Env contract | Same keys read and written | New required key | | Diff size | One symbol moved | Helpers, names, and cache all move | If three or more rows say revert, stop. The change is not the smallest safe extract. Split the work and then pin again. Byte-stable stdout still fails when a new file appears. New files are process behavior, not formatting noise. Treat those new files as a hard mismatch. A coding model can suggest the extract later. The model still cannot invent the characterization oracle. Generate patch candidates only against a committed ledger. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use the server to run the harness on a clean workspace. Use a free model to propose one extract after the pin. Do not ask the model to invent fixtures from memory. Paste the inventory note and the ledger schema instead. Reject any patch that touches cache paths and helpers together. This ledger ignores in-process mocks and thread timing. It will miss behavior that never touches stdout or files. Shared network calls need a different freeze, such as recorded HTTP. Hashing whole trees can hide permission-only file changes. Empty directories may disappear from the file map. Binary files need an explicit allow list on large repos. Fixture packs go stale when product rules change. A matching ledger can still encode a bug. Characterization preserves current behavior, including known bad behavior. The harness also assumes a single process per fixture. Child daemons will outlive the sandbox cleanup step. Stop those child jobs before hashing the tree. Do not use this on greenfield modules with real tests. Do not use this when the contract is a public API suite. Do not use this for security-sensitive parsers without a fuzzer. Skip it when the glue is a thin wrapper over a stable library. Skip it when you cannot sandbox file writes. Skip it when fixtures would include live secrets. Skip it when the script must talk to a live billing API. Recorded side effects would then include third-party state. That class of glue needs contract tests, not file hashes. Pin the side-effect ledger before any extract. Extract one seam only after the ledgers exist. Then diff the two JSON ledger files. That strict order is the entire safety method. The coding model remains optional in this workflow. The characterization oracle is not optional here. If you run this on a free workspace, keep the ledger in git. Leave the model out until the oracle exists.