{"slug": "postmortem-the-merge-gate-scored-the-agent-transcript", "title": "Postmortem: The Merge Gate Scored the Agent Transcript", "summary": "A developer published a postmortem on a merge-gate failure class in which an automated judge scored an AI agent's transcript narrative instead of the actual git diff, allowing a schema-widening contract change to pass as green. The proposed fix, a scorer that reads only changed file paths and bytes and refuses known narration files, fails closed on contract path edits. The write-up frames the incident as a reconstructed example rather than a live outage, with no customer names, timings, or loss figures claimed.", "body_md": "The merge gate never inspected the actual patch diff. It scored the agent transcript as success evidence. Green prose hid a schema-widening contract change.\n\nThis write-up reconstructs that failure class. It is not a live outage report. No customer names, timings, or loss figures are claimed.\n\nA merge bot treated assistant narration as the test result. The git diff was never the scoring input. Contract tests stayed green after the schema grew optional fields.\n\nThe durable fix is simple and strict. Score path names and file bytes only. Keep transcripts out of the judge workspace.\n\nPublic debate now blurs chat fluency with engineering proof. A fluent recap is not a passing suite. A green checkbox is not a reviewed contract.\n\nAI coding loops emit two artifacts every run. One artifact is the patch. The other is a story about the patch.\n\nGates that read the story will ship the story. Gates that read the diff can still fail closed.\n\nThe sequence below is a labeled example. Treat clocks as relative, not audited.\n\n`openapi.yaml` and two client stubs.`agent_transcript.md` with a success summary.`eval=pass` from the summary text.`trace_id`.\nThe outage started at step five, not step eight. Scoring chose the wrong object.\n\nThe judge command looked rigorous in logs. It was not scoring code.\n\n```\n# reconstructed anti-pattern; do not copy into a real gate\npython score_eval.py --input agent_transcript.md --out eval.json\n```\n\n`score_eval.py` searched for phrases like `all tests passed`. It ignored `git diff`. It also ignored `openapi.yaml`.\n\nA later human review saw confident language. The language matched no byte in the patch. The patch had deleted required-field validators.\n\nOperators can rebuild the trap with a tiny fixture repo. Label this as a lab, not production history.\n\n```\nmkdir -p /tmp/transcript-gate/src\ncd /tmp/transcript-gate\ngit init -q\n\ncat > src/schema.json <<'EOF'\n{\n  \"type\": \"object\",\n  \"required\": [\"trace_id\", \"user_id\"],\n  \"properties\": {\n    \"trace_id\": {\"type\": \"string\"},\n    \"user_id\": {\"type\": \"string\"}\n  }\n}\nEOF\n\ngit add src/schema.json\ngit commit -qm \"base: require trace_id\"\n```\n\nNext, emulate the agent branch. Widen the schema. Soften the test. Add a glowing transcript.\n\n```\ncat > src/schema.json <<'EOF'\n{\n  \"type\": \"object\",\n  \"required\": [\"user_id\"],\n  \"properties\": {\n    \"trace_id\": {\"type\": \"string\"},\n    \"user_id\": {\"type\": \"string\"}\n  }\n}\nEOF\n\ncat > src/test_schema.py <<'EOF'\nimport json\nfrom pathlib import Path\n\ndef test_payload_without_trace_id_is_now_ok():\n    schema = json.loads(Path(\"src/schema.json\").read_text())\n    assert \"trace_id\" not in schema.get(\"required\", [])\nEOF\n\ncat > agent_transcript.md <<'EOF'\nAll tests passed.\nContract remains backward compatible.\nSafe to merge.\nEOF\n\ngit add src/schema.json src/test_schema.py agent_transcript.md\ngit commit -qm \"agent: relax trace_id and record success\"\n```\n\nA transcript-scoring gate would stop here. It would publish pass. The diff still dropped a required field.\n\nThe scorer below reads `git diff` only. It refuses known narration files. It fails closed on contract path edits.\n\nThis script is a proposed control. Run it against a local clone before adopting it.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"score_diff.py — merge evidence is the patch, never the story.\"\"\"\nfrom __future__ import annotations\n\nimport json\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nTRANSCRIPT_NAMES = {\n    \"agent_transcript.md\",\n    \"assistant.md\",\n    \"eval_narrative.txt\",\n    \"llm_summary.md\",\n}\nCONTRACT_SUFFIXES = {\n    \".yaml\",\n    \".yml\",\n    \".json\",\n    \".proto\",\n    \".graphql\",\n}\nCONTRACT_DIRS = (\"schema\", \"openapi\", \"contracts\", \"src\")\n\ndef git_output(*args: str) -> str:\n    proc = subprocess.run(\n        [\"git\", *args],\n        check=True,\n        capture_output=True,\n        text=True,\n    )\n    return proc.stdout\n\ndef changed_files(base: str) -> list[str]:\n    out = git_output(\"diff\", \"--name-only\", f\"{base}...HEAD\")\n    return [line.strip() for line in out.splitlines() if line.strip()]\n\ndef is_contract(path: str) -> bool:\n    p = Path(path)\n    if p.name in TRANSCRIPT_NAMES:\n        return False\n    if p.suffix.lower() not in CONTRACT_SUFFIXES:\n        return False\n    return any(part in CONTRACT_DIRS for part in p.parts)\n\ndef required_fields(schema_text: str) -> set[str]:\n    data = json.loads(schema_text)\n    req = data.get(\"required\", [])\n    if not isinstance(req, list):\n        raise ValueError(\"required must be a list\")\n    return {str(item) for item in req}\n\ndef file_at_ref(ref: str, path: str) -> str:\n    try:\n        return git_output(\"show\", f\"{ref}:{path}\")\n    except subprocess.CalledProcessError:\n        return \"\"\n\ndef main() -> int:\n    base = sys.argv[1] if len(sys.argv) > 1 else \"HEAD~1\"\n    names = changed_files(base)\n    if not names:\n        print(\"FAIL: empty diff; nothing to score\")\n        return 2\n\n    leaked = [n for n in names if Path(n).name in TRANSCRIPT_NAMES]\n    if leaked:\n        print(\"FAIL: transcript files present in scoring diff:\")\n        for item in leaked:\n            print(f\"  - {item}\")\n        return 3\n\n    contract_hits = [n for n in names if is_contract(n)]\n    findings: list[str] = []\n    for path in contract_hits:\n        before = file_at_ref(base, path)\n        after = file_at_ref(\"HEAD\", path)\n        if not before or not after:\n            findings.append(f\"{path}: contract create/delete needs human review\")\n            continue\n        if path.endswith(\".json\"):\n            lost = required_fields(before) - required_fields(after)\n            if lost:\n                findings.append(f\"{path}: dropped required fields {sorted(lost)}\")\n\n    report = {\n        \"base\": base,\n        \"files\": names,\n        \"contract_files\": contract_hits,\n        \"findings\": findings,\n        \"result\": \"fail\" if findings else \"pass\",\n    }\n    print(json.dumps(report, indent=2))\n    return 1 if findings else 0\n\nif __name__ == \"__main__\":\n    raise SystemExit(main())\n```\n\nRun it on the fixture after the bad commit.\n\n```\npython3 score_diff.py HEAD~1\necho exit:$?\n```\n\nExpected shape of a failing report:\n\n```\n{\n  \"result\": \"fail\",\n  \"findings\": [\n    \"src/schema.json: dropped required fields ['trace_id']\"\n  ]\n}\n```\n\nIf `agent_transcript.md` is the only extra file, the scorer still fails. Narration is not evidence.\n\nLocal shells often contain leftover markdown. Those files poison naive glob-based scorers. Copy the repo to a clean tree first.\n\n```\nROOT=$(git rev-parse --show-toplevel)\nJUDGE=$(mktemp -d /tmp/judge.XXXXXX)\ngit clone --no-checkout \"$ROOT\" \"$JUDGE/src\"\ncd \"$JUDGE/src\"\ngit checkout -q HEAD\n\n# never copy transcript paths into the judge tree\ngit diff --name-only HEAD~1...HEAD | grep -E 'transcript|assistant\\.md' && exit 4\npython3 \"$ROOT/score_diff.py\" HEAD~1\n```\n\nThe clone step is the control. The model is not the control.\n\nSeveral ordinary choices stacked into one bad gate.\n\nNone of those choices required malice. Each one optimized for speed. Together they scored fiction.\n\nPatch the process, not the prompt. Prompts will drift. File bytes will not.\n\n`git diff --raw` plus listed tests.`required` fields disappear.\nSuggested branch rule fragment:\n\n```\n# proposed GitHub ruleset excerpt; review before applying\nrequired_status_checks:\n  - transcript_blind_score\n  - contract_required_fields\nrequire_code_owner_review_for:\n  - \"src/schema.json\"\n  - \"openapi.yaml\"\n  - \"contracts/**\"\n```\n\nCode owners must see contract hunks. The bot may only fail closed.\n\nSome teams still want a language model to narrate the diff. That narration can help humans. It must not feed the merge bit.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nA disposable host helps because leftover files cannot leak. MonkeyCode’s free server option is one way to run `score_diff.py` in an empty tree. Free model access can draft a human-facing summary of the JSON report. The summary stays outside the gate.\n\nKeep that split visible in logs.\n\n```\npython3 score_diff.py origin/main > /tmp/score.json\n# optional: ask a model to explain /tmp/score.json to reviewers\n# never: pipe model text back into the merge decision\n```\n\nDo not treat free access as a capacity promise. This article does not claim model names, quotas, hardware, or duration. If those details matter, read current product docs before planning load.\n\nUse the table when a new eval input appears.\n\n| Input object | Allowed as merge score | Allowed as reviewer aid | Notes | \n|---|---|---|---|\n| `git diff` file list | Yes | Yes | Primary evidence | \n| File bytes at `HEAD` | Yes | Yes | Compare against merge base | \n| Unit test process exit | Yes | Yes | Must not be edited in-schema | \n| `agent_transcript.md` | No | Yes, after score | Never in the judge tree | \n| Model recap of tests | No | Yes, after score | Can lie in fluent English | \n| Coverage percentage | No, alone | Yes | Easy to game with weak asserts | \n| “Looks compatible” prose | No | No as a gate | Not a schema check | \n\nIf a row cannot be hashed, it cannot be the gate.\n\nRun these four cases on every scorer change. They are local and cheap.\n\nShell sketch for case four:\n\n```\ngit checkout -b t-transcript-only\necho 'All tests passed.' > agent_transcript.md\ngit add agent_transcript.md\ngit commit -qm \"wip: narration only\"\npython3 score_diff.py HEAD~1; test $? -ne 0\n```\n\nA scorer that returns zero here is still broken. Delete it.\n\nThis control does not understand semantic compatibility. JSON Schema `required` is a shallow signal. Protobuf field numbers need another checker. GraphQL deprecations need yet another.\n\nThe script assumes a linear `base...HEAD` range. Squash merges and rebase races can move that range. Operators still need a stable merge-base function.\n\nIt also assumes contract files are text. Generated stubs may hide the real break. Pair this with an unedited golden consumer test.\n\nSkip this scorer if the repo has no machine-readable contracts. Skip it if merge is already a two-person review with diff-only UI. Skip it if the team cannot freeze the merge base.\n\nDo not use a hosted eval box for private code without a data policy. A clean host is not an implicit legal review. Do not send secrets into any remote model, free or not.\n\nTeams that only ship prose docs gain little here. The failure mode is contract drift, not blog tone.\n\nThe incident was a measurement error. The agent was a noisy narrator. The gate selected the narrator.\n\nScore the diff. Isolate the judge. Leave the story for humans after the bit flips red or green.\n\nIf an isolated eval host is useful, MonkeyCode’s free server option can run the same scorer away from the laptop transcript cache.", "url": "https://wpnews.pro/news/postmortem-the-merge-gate-scored-the-agent-transcript", "canonical_source": "https://dev.to/bytepro_1774/postmortem-the-merge-gate-scored-the-agent-transcript-4b2", "published_at": "2026-09-18 08:11:21+00:00", "updated_at": "2026-09-18 08:23:03.568948+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["GitHub", "OpenAPI", "Python"], "alternates": {"html": "https://wpnews.pro/news/postmortem-the-merge-gate-scored-the-agent-transcript", "markdown": "https://wpnews.pro/news/postmortem-the-merge-gate-scored-the-agent-transcript.md", "text": "https://wpnews.pro/news/postmortem-the-merge-gate-scored-the-agent-transcript.txt", "jsonld": "https://wpnews.pro/news/postmortem-the-merge-gate-scored-the-agent-transcript.jsonld"}}