If You Cannot Replay the Agent, Do Not Merge

A developer argues that coding-agent patches should be rejected unless they come with a replay file that reconstructs the same side effects from stored inputs, rather than relying on a scrolling demo or a green terminal pasted into Slack. The post proposes a JSON replay contract with fields such as task_hash, model_id, base_git_sha, allowed_tools, per-call argument and result digests, patch_digest, and human_decision, plus a tool allowlist and an example CI checker. The piece was prepared as part of MonkeyCode's product outreach, which offers free model access and a free server option.

You should refuse any agent patch you cannot replay. A scrolling demo is not a build artifact. Merge the replay file first, then consider the diff. Unreproducible agents are not production-grade coding assistants. They behave like unrecorded pair programmers with total amnesia. You would not accept that from a human contractor. You would demand logs from a flaky integration test. Demand the same evidence from every coding-agent session. Otherwise you are shipping a one-time stage performance. Opinion, not a hedged framework dump: replay or reject. Clever plans cannot replace a missing tool transcript. Your future self cannot debug a vanished chain of tool calls. Replay is not running the same prompt and hoping. Replay means stored inputs reconstruct the same side effects. You keep enough evidence to rebuild the job later. A useful replay file must answer six concrete questions. If any row is missing, you do not have a replay. You have a diary entry with extra punctuation. Diary entries do not belong on the main branch. Free or cheap tokens hide the cost of wasted loops. You retry until something compiles, then forget the path. Cheap retries without transcripts create irreproducible success stories. Teams then paste a green terminal into Slack threads. Nobody can name the tool order that produced files. The next intern cannot reconstruct the change two weeks later. That is how "the agent just knows the repo" myths start. The agent does not actually know your repository. It sampled a path you failed to record. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that sandbox to practice the replay contract, not to skip it. Keep the same checker when you later pay for inference. Free sandboxes do not lower the merge bar. They only keep experiments off your invoice. Watch for patches that exist only inside a chat pane. Watch for tool traces truncated to look "readable" in demos. Watch for model labels copied from a pricing page. If the transcript was edited by hand, discard the run. If file hashes drifted after cleanup, discard the run. If the PR description is only vibes, demand the file. Pin a JSON contract before you write more agent glue. The schema below is an example, not a shipped standard. Adapt field names to your orchestrator without adding poetry. { "schema version": "1", "task id": "ISSUE-1842", "task hash": "sha256:REPLACE WITH PROMPT FILE HASH", "model id": "pinned-opaque-string", "base git sha": "abc1234", "allowed tools": "read file", "write file", "run tests" , "tool calls": { "seq": 1, "name": "read file", "args digest": "sha256:ARGS", "result digest": "sha256:BYTES" } , "patch digest": "sha256:GIT DIFF", "human decision": "review", "secrets redacted": true } Every tool calls item needs a digest, not a novel. Store stdout hashes when the bytes are large. Store raw text only when the payload is tiny and non-secret. Keep a tight allowlist beside that JSON file. Unknown names fail the run. Broad shell access does not belong in the first draft. tools.allow example read file write file run tests Validate the file in CI before you discuss the patch. The checker below is an example, not a product. Do not treat a passing checker as proof of model quality. bash /usr/bin/env python3 """Example only: unexecuted replay-file checker. Not a benchmark.""" from future import annotations import hashlib import json import sys from pathlib import Path REQUIRED = { "schema version", "task hash", "model id", "base git sha", "allowed tools", "tool calls", "patch digest", "human decision", "secrets redacted", } def sha256 bytes data: bytes - str: return "sha256:" + hashlib.sha256 data .hexdigest def fail message: str - None: print "REPLAY FAIL:", message raise SystemExit 2 def load allowlist path: Path - set str : names = set for line in path.read text encoding="utf-8" .splitlines : text = line.strip if text and not text.startswith " " : names.add text if not names: fail "allowlist is empty" return names def main argv: list str - None: if len argv = 3: fail "usage: check replay.py replay.json tools.allow" replay path = Path argv 1 allow path = Path argv 2 payload = json.loads replay path.read text encoding="utf-8" missing = REQUIRED - set payload if missing: fail f"missing keys: {sorted missing }" if payload.get "secrets redacted" is not True: fail "secrets redacted must be true" if payload.get "human decision" not in {"review", "reject", "isolate"}: fail "human decision is not an allowed value" if not str payload.get "patch digest", "" .startswith "sha256:" : fail "patch digest is not a sha256 digest" allow = load allowlist allow path declared = set payload.get "allowed tools" or if declared - allow: fail f"tools not in allowlist file: {sorted declared - allow }" calls = payload.get "tool calls" or if not isinstance calls, list or not calls: fail "tool calls must be a non-empty list" for item in calls: name = item.get "name" if name not in allow: fail f"tool call not allowed: {name}" if not str item.get "args digest", "" .startswith "sha256:" : fail f"args digest missing for {name}" if not str item.get "result digest", "" .startswith "sha256:" : fail f"result digest missing for {name}" print "REPLAY OK", sha256 bytes replay path.read bytes if name == " main ": main sys.argv Run it on fixtures that should fail. A validator that never fails is only decoration. You want red on missing hashes, unknown tools, and empty diffs. Use a boring table instead of another agent self-eval. A human still owns merge after the table returns. The table exists to block unreproducible theater. | Signal | Action | Rule you enforce | |---|---|---| | Replay file missing from the PR | Block | No transcript means no merge | | Tool name outside tools.allow | Block | The run is already dirty | | Digest does not match saved bytes | Block | You cannot replay this job | | human decision is empty or merge | Block | The agent does not own merge | | Tests arrived in the same agent burst | Hold | A human rewrites those tests | | Replay valid and tests are human-owned | Review | You still read the diff | Notice merge is absent from allowed decisions. Review, reject, or isolate are the only honest exits. Auto-merge is how unreproducible patches escape. Put the replay file next to the patch in the PR. Reject the PR when the checker exits non-zero. Reject the PR when hashes and diffs disagree. Example only: local gate before you open the PR. python3 check replay.py replay.json tools.allow git rev-parse HEAD git diff --binary "$BASE SHA" /tmp/agent.patch python3 - <<'PY' from pathlib import Path import hashlib, json, sys payload = json.loads Path "replay.json" .read text raw = Path "/tmp/agent.patch" .read bytes digest = "sha256:" + hashlib.sha256 raw .hexdigest if digest = payload "patch digest" : print "PATCH DIGEST MISMATCH", digest sys.exit 2 print "PATCH DIGEST OK" PY Do not let the agent write this checker. You write the checker and you own it. The agent may propose patches, never the gate. When check replay.py fails, stop generating more code. You are now in forensics, not in authoring. Follow this order and do not skip steps. replay.json parses as UTF-8 JSON without comments. task hash matches a sha256 of the prompt file. result digest matches the saved artifact bytes. If step six still drifts, you do not have replay yet. You have a non-deterministic tool or an unpinned runtime. Freeze the runtime before you blame the prompt. Drift usually means an unpinned tool, not a "creative" model. Network time, random files, and live clocks leak entropy. You stub those before you tune prompts again. Record the stub, not the live socket. Live search and live HTTP are not replay sources. If a tool must touch the network, isolate that step from merge. This contract does not make a weak model strong. It does not prove the patch is correct, only reconstructable. Correctness still needs tests you wrote without the agent. It will not capture GPU nondeterminism you refused to pin. It will not redact secrets you stuffed into prompts. It will not help if you never review the diff. Floating-point tools, network calls, and clocks break naive replay. You must stub those tools or record their exact bytes. If you cannot stub them, isolate that step from merge. Do not use this as a permit for unattended production writes. Do not use this in regulated systems without a real audit log. Do not use this to rubber-stamp generated tests. Skip the JSON file if you never invoke tools. A single-shot edit in your own editor needs a review, not theater. This gate is for looped agents with side effects. If you cannot store transcripts safely, do not start. Prompts include secrets, customer names, and private URLs. You need a redaction step before any shared sandbox. You do not need a smarter agent to ship safer diffs. You need a replay file that survives a cold restart. If you cannot replay the agent, you do not merge.