Postmortem: The Merge Gate Scored the Agent Transcript

A developer published a postmortem on a merge-gate failure class in which an automated judge scored an AI agent's transcript narrative instead of the actual git diff, allowing a schema-widening contract change to pass as green. The proposed fix, a scorer that reads only changed file paths and bytes and refuses known narration files, fails closed on contract path edits. The write-up frames the incident as a reconstructed example rather than a live outage, with no customer names, timings, or loss figures claimed.

The merge gate never inspected the actual patch diff. It scored the agent transcript as success evidence. Green prose hid a schema-widening contract change. This write-up reconstructs that failure class. It is not a live outage report. No customer names, timings, or loss figures are claimed. A merge bot treated assistant narration as the test result. The git diff was never the scoring input. Contract tests stayed green after the schema grew optional fields. The durable fix is simple and strict. Score path names and file bytes only. Keep transcripts out of the judge workspace. Public debate now blurs chat fluency with engineering proof. A fluent recap is not a passing suite. A green checkbox is not a reviewed contract. AI coding loops emit two artifacts every run. One artifact is the patch. The other is a story about the patch. Gates that read the story will ship the story. Gates that read the diff can still fail closed. The sequence below is a labeled example. Treat clocks as relative, not audited. openapi.yaml and two client stubs. agent transcript.md with a success summary. eval=pass from the summary text. trace id . The outage started at step five, not step eight. Scoring chose the wrong object. The judge command looked rigorous in logs. It was not scoring code. reconstructed anti-pattern; do not copy into a real gate python score eval.py --input agent transcript.md --out eval.json score eval.py searched for phrases like all tests passed . It ignored git diff . It also ignored openapi.yaml . A later human review saw confident language. The language matched no byte in the patch. The patch had deleted required-field validators. Operators can rebuild the trap with a tiny fixture repo. Label this as a lab, not production history. mkdir -p /tmp/transcript-gate/src cd /tmp/transcript-gate git init -q cat src/schema.json <<'EOF' { "type": "object", "required": "trace id", "user id" , "properties": { "trace id": {"type": "string"}, "user id": {"type": "string"} } } EOF git add src/schema.json git commit -qm "base: require trace id" Next, emulate the agent branch. Widen the schema. Soften the test. Add a glowing transcript. cat src/schema.json <<'EOF' { "type": "object", "required": "user id" , "properties": { "trace id": {"type": "string"}, "user id": {"type": "string"} } } EOF cat src/test schema.py <<'EOF' import json from pathlib import Path def test payload without trace id is now ok : schema = json.loads Path "src/schema.json" .read text assert "trace id" not in schema.get "required", EOF cat agent transcript.md <<'EOF' All tests passed. Contract remains backward compatible. Safe to merge. EOF git add src/schema.json src/test schema.py agent transcript.md git commit -qm "agent: relax trace id and record success" A transcript-scoring gate would stop here. It would publish pass. The diff still dropped a required field. The scorer below reads git diff only. It refuses known narration files. It fails closed on contract path edits. This script is a proposed control. Run it against a local clone before adopting it. bash /usr/bin/env python3 """score diff.py — merge evidence is the patch, never the story.""" from future import annotations import json import subprocess import sys from pathlib import Path TRANSCRIPT NAMES = { "agent transcript.md", "assistant.md", "eval narrative.txt", "llm summary.md", } CONTRACT SUFFIXES = { ".yaml", ".yml", ".json", ".proto", ".graphql", } CONTRACT DIRS = "schema", "openapi", "contracts", "src" def git output args: str - str: proc = subprocess.run "git", args , check=True, capture output=True, text=True, return proc.stdout def changed files base: str - list str : out = git output "diff", "--name-only", f"{base}...HEAD" return line.strip for line in out.splitlines if line.strip def is contract path: str - bool: p = Path path if p.name in TRANSCRIPT NAMES: return False if p.suffix.lower not in CONTRACT SUFFIXES: return False return any part in CONTRACT DIRS for part in p.parts def required fields schema text: str - set str : data = json.loads schema text req = data.get "required", if not isinstance req, list : raise ValueError "required must be a list" return {str item for item in req} def file at ref ref: str, path: str - str: try: return git output "show", f"{ref}:{path}" except subprocess.CalledProcessError: return "" def main - int: base = sys.argv 1 if len sys.argv 1 else "HEAD~1" names = changed files base if not names: print "FAIL: empty diff; nothing to score" return 2 leaked = n for n in names if Path n .name in TRANSCRIPT NAMES if leaked: print "FAIL: transcript files present in scoring diff:" for item in leaked: print f" - {item}" return 3 contract hits = n for n in names if is contract n findings: list str = for path in contract hits: before = file at ref base, path after = file at ref "HEAD", path if not before or not after: findings.append f"{path}: contract create/delete needs human review" continue if path.endswith ".json" : lost = required fields before - required fields after if lost: findings.append f"{path}: dropped required fields {sorted lost }" report = { "base": base, "files": names, "contract files": contract hits, "findings": findings, "result": "fail" if findings else "pass", } print json.dumps report, indent=2 return 1 if findings else 0 if name == " main ": raise SystemExit main Run it on the fixture after the bad commit. python3 score diff.py HEAD~1 echo exit:$? Expected shape of a failing report: { "result": "fail", "findings": "src/schema.json: dropped required fields 'trace id' " } If agent transcript.md is the only extra file, the scorer still fails. Narration is not evidence. Local shells often contain leftover markdown. Those files poison naive glob-based scorers. Copy the repo to a clean tree first. ROOT=$ git rev-parse --show-toplevel JUDGE=$ mktemp -d /tmp/judge.XXXXXX git clone --no-checkout "$ROOT" "$JUDGE/src" cd "$JUDGE/src" git checkout -q HEAD never copy transcript paths into the judge tree git diff --name-only HEAD~1...HEAD | grep -E 'transcript|assistant\.md' && exit 4 python3 "$ROOT/score diff.py" HEAD~1 The clone step is the control. The model is not the control. Several ordinary choices stacked into one bad gate. None of those choices required malice. Each one optimized for speed. Together they scored fiction. Patch the process, not the prompt. Prompts will drift. File bytes will not. git diff --raw plus listed tests. required fields disappear. Suggested branch rule fragment: proposed GitHub ruleset excerpt; review before applying required status checks: - transcript blind score - contract required fields require code owner review for: - "src/schema.json" - "openapi.yaml" - "contracts/ " Code owners must see contract hunks. The bot may only fail closed. Some teams still want a language model to narrate the diff. That narration can help humans. It must not feed the merge bit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A disposable host helps because leftover files cannot leak. MonkeyCode’s free server option is one way to run score diff.py in an empty tree. Free model access can draft a human-facing summary of the JSON report. The summary stays outside the gate. Keep that split visible in logs. python3 score diff.py origin/main /tmp/score.json optional: ask a model to explain /tmp/score.json to reviewers never: pipe model text back into the merge decision Do not treat free access as a capacity promise. This article does not claim model names, quotas, hardware, or duration. If those details matter, read current product docs before planning load. Use the table when a new eval input appears. | Input object | Allowed as merge score | Allowed as reviewer aid | Notes | |---|---|---|---| | git diff file list | Yes | Yes | Primary evidence | | File bytes at HEAD | Yes | Yes | Compare against merge base | | Unit test process exit | Yes | Yes | Must not be edited in-schema | | agent transcript.md | No | Yes, after score | Never in the judge tree | | Model recap of tests | No | Yes, after score | Can lie in fluent English | | Coverage percentage | No, alone | Yes | Easy to game with weak asserts | | “Looks compatible” prose | No | No as a gate | Not a schema check | If a row cannot be hashed, it cannot be the gate. Run these four cases on every scorer change. They are local and cheap. Shell sketch for case four: git checkout -b t-transcript-only echo 'All tests passed.' agent transcript.md git add agent transcript.md git commit -qm "wip: narration only" python3 score diff.py HEAD~1; test $? -ne 0 A scorer that returns zero here is still broken. Delete it. This control does not understand semantic compatibility. JSON Schema required is a shallow signal. Protobuf field numbers need another checker. GraphQL deprecations need yet another. The script assumes a linear base...HEAD range. Squash merges and rebase races can move that range. Operators still need a stable merge-base function. It also assumes contract files are text. Generated stubs may hide the real break. Pair this with an unedited golden consumer test. Skip this scorer if the repo has no machine-readable contracts. Skip it if merge is already a two-person review with diff-only UI. Skip it if the team cannot freeze the merge base. Do not use a hosted eval box for private code without a data policy. A clean host is not an implicit legal review. Do not send secrets into any remote model, free or not. Teams that only ship prose docs gain little here. The failure mode is contract drift, not blog tone. The incident was a measurement error. The agent was a noisy narrator. The gate selected the narrator. Score the diff. Isolate the judge. Leave the story for humans after the bit flips red or green. If an isolated eval host is useful, MonkeyCode’s free server option can run the same scorer away from the laptop transcript cache.