{"slug": "if-the-model-moved-burn-the-golden-files", "title": "If the Model Moved, Burn the Golden Files", "summary": "A developer argues that frozen golden-file tests for AI agents break down when serving models change, producing false failures on cosmetic wording while missing real behavioral drift. The proposed workflow replaces static fixtures with three layers in one job: deterministic contract checks on JSON shape and required tools, effects checks on the resulting world state, and a cheaper pinned critic model that reviews redacted traces on a clean remote host. The author notes the critique layer is usually the missing one and warns that costly critique tends to be skipped during incidents and demos.", "body_md": "Frozen golden files cannot survive a model upgrade. You should run a living critic on a clean host.\n\nYour agent tests still freeze last week's answers. Then the serving model changes overnight without warning. The suite fails on cosmetic wording after that bump.\n\nOr it stays green while real behavior drifts. You do not hold a quality signal anymore. You hold a snapshot of a dead model.\n\nTreat those frozen answers as unpaid test debt. A moving model needs a living check. A frozen string is not that check.\n\nPeople now ask if AI outgrew our tests. Ask first whether fixtures outlived the recorded model. Those are different failures with different fixes.\n\nA golden string still assumes one correct reply. Agents do not emit one stable reply anymore. They emit tool traces, side effects, and leftover prose.\n\nWording shifts after every quiet vendor bump. Call order also shifts after a schema tweak. Your diff tool then screams about both events.\n\nNeither scream maps cleanly to user harm. Worse cases stay silent under the same fixtures. The new model skips a required precondition check.\n\nYour frozen paragraph still matches yesterday's run. The build stays green and then ships. That is not coverage. That is nostalgia.\n\nFrozen files punish harmless change far too often. They also miss harmful change far too often. That combination makes a bad test instrument.\n\nKeep three layers in the same job. Do not collapse them into chat scores.\n\nContracts check JSON shape and required tools. Those checks remain deterministic across later model swaps.\n\nEffects checks inspect the world after the run. Count rows, HTTP codes, and file hashes.\n\nCritique is a second model reading the trace. It hunts plans that are valid and still dumb.\n\nContracts catch schema rot early. Effects catch real damage in the system. Critique catches confident nonsense that contracts allow.\n\nMost teams stop at contracts or prose. Few run a critic on every bump. The missing layer is almost always the critic.\n\nHere is the hard claim for this piece. Costly critique will not run on every bump.\n\nYou will postpone that job until the customer demo. You will skip it during the actual incident. Skipped critique is how untested agents ship.\n\nThe critic should not be the actor model. The actor is expensive and already moving. The critic should be cheaper and pinned in config.\n\nThe critic should not share your laptop either. Your laptop holds secrets and dirty state. Your laptop also sleeps while CI expects an answer.\n\nPark redacted traces on a clean remote host. Then ask the critic to lint the trace. Keep production credentials off that host.\n\nThis section is a proposed workflow. It is not a vendor benchmark. It claims no accuracy number and no leaderboard.\n\nYou capture traces, gate contracts, then call a critic. Keep each step boring enough to rerun. Glamour is how eval suites rot.\n\nWrite one JSON object per agent run. Keep the capture schema boring, small, and stable.\n\n```\n{\"id\": \"pr-4412-a\", \"goal\": \"create invoice draft\", \"tools\": [{\"name\": \"db.query\", \"args\": {\"table\": \"customers\", \"id\": 918}}, {\"name\": \"invoices.create\", \"args\": {\"customer_id\": 918, \"cents\": 1299}}], \"final\": \"Draft created for customer 918\"}\n```\n\nRedact tokens before anything hits disk. Redact names if your policy requires that step.\n\n```\npython capture_traces.py --task invoices --out traces.jsonl\npython redact.py --in traces.jsonl --out traces.clean.jsonl\n```\n\nRun deterministic checks before any model call. Do not spend critique budget on garbage traces.\n\n``` python\n# contract_gate.py — labeled proposal, not a shipped product\nimport json, sys\n\nREQUIRED = {\"db.query\", \"invoices.create\"}\nFORBIDDEN = (\"password\", \"ssn\", \"authorization\", \"api_key\")\n\ndef errors_for(event):\n    problems = []\n    names = {step[\"name\"] for step in event.get(\"tools\", [])}\n    missing = REQUIRED - names\n    if missing:\n        problems.append(f\"missing tools: {sorted(missing)}\")\n    blob = json.dumps(event).lower()\n    for needle in FORBIDDEN:\n        if needle in blob:\n            problems.append(f\"possible secret field: {needle}\")\n    if not event.get(\"goal\"):\n        problems.append(\"empty goal\")\n    return problems\n\nfail = 0\nwith open(sys.argv[1]) as handle:\n    for line in handle:\n        event = json.loads(line)\n        problems = errors_for(event)\n        if problems:\n            fail += 1\n            print(event.get(\"id\"), problems)\nsys.exit(1 if fail else 0)\n```\n\nCall it like any other unit test. A red contract means stop. Critique comes only after this gate.\n\n```\npython contract_gate.py traces.clean.jsonl\n```\n\nThe prompt is part of the test suite. Version it and review it like code.\n\n```\nYou are a trace linter, not a chat assistant.\nReturn JSON only with keys risk and reasons.\nrisk must be low, med, or high.\nFlag missing preconditions, extra tools, and unverifiable claims.\nDo not rewrite the agent. Do not praise the agent.\n```\n\nIf nobody reviews this file, you cannot trust the critic. An unowned prompt is another moving model.\n\nKeep the HTTP client dull on purpose. Do not hide retries inside a heavy SDK.\n\n``` python\n# critique_run.py — labeled proposal\nimport json, os, sys, urllib.request\n\nENDPOINT = os.environ[\"CRITIC_ENDPOINT\"]\nTOKEN = os.environ[\"CRITIC_TOKEN\"]\nPROMPT = open(\"critic_prompt.txt\").read()\n\ndef critique(event):\n    body = json.dumps({\n        \"instructions\": PROMPT,\n        \"trace\": event,\n    }).encode()\n    req = urllib.request.Request(\n        ENDPOINT,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {TOKEN}\",\n        },\n        method=\"POST\",\n    )\n    with urllib.request.urlopen(req, timeout=30) as resp:\n        return json.loads(resp.read().decode())\n\nhigh = 0\nwith open(sys.argv[1]) as handle:\n    for line in handle:\n        event = json.loads(line)\n        result = critique(event)\n        risk = result.get(\"risk\", \"high\")\n        print(event[\"id\"], risk, result.get(\"reasons\"))\n        if risk == \"high\":\n            high += 1\nsys.exit(1 if high else 0)\n```\n\nFail the job on `high` risk. Log `med` for humans, then move on. Never auto-merge on `low` alone.\n\n```\nclean:\n    python redact.py --in traces.jsonl --out traces.clean.jsonl\n\ncontracts: clean\n    python contract_gate.py traces.clean.jsonl\n\ncritique: contracts\n    python critique_run.py traces.clean.jsonl\n\neval: critique\n    python report.py traces.clean.jsonl\n```\n\nYou should run `make eval` after every model bump. If that command needs a meeting, the harness is too heavy.\n\nUse this table in design reviews. Do not argue this choice from vibe.\n\n| Question | Frozen golden file | Contract test | Cheap remote critic | \n|---|---|---|---|\n| Does wording drift fail the build? | Yes, noisily | No | Only if risk rises | \n| Does an extra tool get caught? | Sometimes | Yes, if listed | Often | \n| Does a silent no-op get caught? | Rarely | Rarely | More often | \n| Does a schema break get caught? | No | Yes | Weakly | \n| Does it survive a vendor bump? | No | Yes | Yes, if the prompt is pinned | \n| What should it cost per PR? | Near zero | Near zero | Must stay near zero | \n\nFreeze bytes that must not change across releases. That means SQL migrations, signed manifests, and PDF hashes. Do not freeze chat.\n\nLocal critique inherits your clipboard history. Local critique inherits forgotten `.env` files. Local critique vanishes when the lid closes.\n\nA remote box gives you a repeatable host. You can wipe that box after each run. You can point CI at one hostname.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nYou still need a cheap model and a cheap host. MonkeyCode is an open-source project with free model access. It also offers a free server option for remote runs.\n\nUse that pair for the critic host and the critic model. Do not park your production agent there on day one. Traces and a linter belong there first.\n\nThis article will not quote quotas or hardware claims. Those numbers go stale faster than your fixtures. Read the current project docs before you pin any limit.\n\nIf you already have a spare VM, use it. The harness does not require a particular brand. The method should survive a vendor swap.\n\nA cheap critic is not an oracle for truth. It can bless fluent errors without shame. It can nag about style instead of harm.\n\nIt will miss domain bugs that need a human. It will miss races and partial writes. It will miss authz holes unless you feed it policy.\n\nRemote calls flake under real networks. Set a timeout and retry once. Then fail closed, not open.\n\nFree endpoints can throttle or vanish later. The job must fail loud on empty responses. A silent skip returns you to frozen files.\n\nDo not treat critic JSON as ground truth. Treat it as a linter with opinions. Humans still own merge decisions.\n\nDo not upload regulated traces to a shared model. Do not critique live PII through this path. Do not score people with this output.\n\nDo not replace effect tests with a critic paragraph. If the agent moves money, test the ledger first. Critique is a third layer, not the first.\n\nIf nobody owns the critic prompt, stop now. An unversioned critic is another moving model. Ownership is part of the test.\n\nIf your traces still contain customer secrets, stop now. Redaction is a gate, not a nicety. Clean traces or no remote call.\n\nThe production model will move again soon. Your golden files will lie again soon. Plan for that lie before it ships.\n\nKeep contracts in the same pipeline. Keep effect checks next to them. Keep a cheap critic on a clean host.\n\nThrow away the frozen answers after each bump. Measure moving systems with living tests instead. Anything cheaper than that stack is theater.\n\nIf you need a remote critic box this week, read MonkeyCode's current docs. Run this harness on a handful of traces first.", "url": "https://wpnews.pro/news/if-the-model-moved-burn-the-golden-files", "canonical_source": "https://dev.to/airs_6907/if-the-model-moved-burn-the-golden-files-1eg7", "published_at": "2026-09-21 13:36:00+00:00", "updated_at": "2026-09-21 14:01:43.546301+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "ai-safety"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/if-the-model-moved-burn-the-golden-files", "markdown": "https://wpnews.pro/news/if-the-model-moved-burn-the-golden-files.md", "text": "https://wpnews.pro/news/if-the-model-moved-burn-the-golden-files.txt", "jsonld": "https://wpnews.pro/news/if-the-model-moved-burn-the-golden-files.jsonld"}}