{"slug": "if-you-cannot-replay-the-agent-do-not-merge", "title": "If You Cannot Replay the Agent, Do Not Merge", "summary": "A developer argues that coding-agent patches should be rejected unless they come with a replay file that reconstructs the same side effects from stored inputs, rather than relying on a scrolling demo or a green terminal pasted into Slack. The post proposes a JSON replay contract with fields such as task_hash, model_id, base_git_sha, allowed_tools, per-call argument and result digests, patch_digest, and human_decision, plus a tool allowlist and an example CI checker. The piece was prepared as part of MonkeyCode's product outreach, which offers free model access and a free server option.", "body_md": "You should refuse any agent patch you cannot replay. A scrolling demo is not a build artifact. Merge the replay file first, then consider the diff.\n\nUnreproducible agents are not production-grade coding assistants. They behave like unrecorded pair programmers with total amnesia. You would not accept that from a human contractor.\n\nYou would demand logs from a flaky integration test. Demand the same evidence from every coding-agent session. Otherwise you are shipping a one-time stage performance.\n\nOpinion, not a hedged framework dump: replay or reject. Clever plans cannot replace a missing tool transcript. Your future self cannot debug a vanished chain of tool calls.\n\nReplay is not running the same prompt and hoping. Replay means stored inputs reconstruct the same side effects. You keep enough evidence to rebuild the job later.\n\nA useful replay file must answer six concrete questions.\n\nIf any row is missing, you do not have a replay. You have a diary entry with extra punctuation. Diary entries do not belong on the main branch.\n\nFree or cheap tokens hide the cost of wasted loops. You retry until something compiles, then forget the path. Cheap retries without transcripts create irreproducible success stories.\n\nTeams then paste a green terminal into Slack threads. Nobody can name the tool order that produced files. The next intern cannot reconstruct the change two weeks later.\n\nThat is how \"the agent just knows the repo\" myths start. The agent does not actually know your repository. It sampled a path you failed to record.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that sandbox to practice the replay contract, not to skip it.\n\nKeep the same checker when you later pay for inference. Free sandboxes do not lower the merge bar. They only keep experiments off your invoice.\n\nWatch for patches that exist only inside a chat pane. Watch for tool traces truncated to look \"readable\" in demos. Watch for model labels copied from a pricing page.\n\nIf the transcript was edited by hand, discard the run. If file hashes drifted after cleanup, discard the run. If the PR description is only vibes, demand the file.\n\nPin a JSON contract before you write more agent glue. The schema below is an example, not a shipped standard. Adapt field names to your orchestrator without adding poetry.\n\n```\n{\n  \"schema_version\": \"1\",\n  \"task_id\": \"ISSUE-1842\",\n  \"task_hash\": \"sha256:REPLACE_WITH_PROMPT_FILE_HASH\",\n  \"model_id\": \"pinned-opaque-string\",\n  \"base_git_sha\": \"abc1234\",\n  \"allowed_tools\": [\"read_file\", \"write_file\", \"run_tests\"],\n  \"tool_calls\": [\n    {\n      \"seq\": 1,\n      \"name\": \"read_file\",\n      \"args_digest\": \"sha256:ARGS\",\n      \"result_digest\": \"sha256:BYTES\"\n    }\n  ],\n  \"patch_digest\": \"sha256:GIT_DIFF\",\n  \"human_decision\": \"review\",\n  \"secrets_redacted\": true\n}\n```\n\nEvery `tool_calls` item needs a digest, not a novel. Store stdout hashes when the bytes are large. Store raw text only when the payload is tiny and non-secret.\n\nKeep a tight allowlist beside that JSON file. Unknown names fail the run. Broad `shell` access does not belong in the first draft.\n\n```\n# tools.allow  (example)\nread_file\nwrite_file\nrun_tests\n```\n\nValidate the file in CI before you discuss the patch. The checker below is an example, not a product. Do not treat a passing checker as proof of model quality.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Example only: unexecuted replay-file checker. Not a benchmark.\"\"\"\nfrom __future__ import annotations\n\nimport hashlib\nimport json\nimport sys\nfrom pathlib import Path\n\nREQUIRED = {\n    \"schema_version\",\n    \"task_hash\",\n    \"model_id\",\n    \"base_git_sha\",\n    \"allowed_tools\",\n    \"tool_calls\",\n    \"patch_digest\",\n    \"human_decision\",\n    \"secrets_redacted\",\n}\n\ndef sha256_bytes(data: bytes) -> str:\n    return \"sha256:\" + hashlib.sha256(data).hexdigest()\n\ndef fail(message: str) -> None:\n    print(\"REPLAY_FAIL:\", message)\n    raise SystemExit(2)\n\ndef load_allowlist(path: Path) -> set[str]:\n    names = set()\n    for line in path.read_text(encoding=\"utf-8\").splitlines():\n        text = line.strip()\n        if text and not text.startswith(\"#\"):\n            names.add(text)\n    if not names:\n        fail(\"allowlist is empty\")\n    return names\n\ndef main(argv: list[str]) -> None:\n    if len(argv) != 3:\n        fail(\"usage: check_replay.py replay.json tools.allow\")\n    replay_path = Path(argv[1])\n    allow_path = Path(argv[2])\n    payload = json.loads(replay_path.read_text(encoding=\"utf-8\"))\n    missing = REQUIRED - set(payload)\n    if missing:\n        fail(f\"missing keys: {sorted(missing)}\")\n    if payload.get(\"secrets_redacted\") is not True:\n        fail(\"secrets_redacted must be true\")\n    if payload.get(\"human_decision\") not in {\"review\", \"reject\", \"isolate\"}:\n        fail(\"human_decision is not an allowed value\")\n    if not str(payload.get(\"patch_digest\", \"\")).startswith(\"sha256:\"):\n        fail(\"patch_digest is not a sha256 digest\")\n    allow = load_allowlist(allow_path)\n    declared = set(payload.get(\"allowed_tools\") or [])\n    if declared - allow:\n        fail(f\"tools not in allowlist file: {sorted(declared - allow)}\")\n    calls = payload.get(\"tool_calls\") or []\n    if not isinstance(calls, list) or not calls:\n        fail(\"tool_calls must be a non-empty list\")\n    for item in calls:\n        name = item.get(\"name\")\n        if name not in allow:\n            fail(f\"tool call not allowed: {name}\")\n        if not str(item.get(\"args_digest\", \"\")).startswith(\"sha256:\"):\n            fail(f\"args_digest missing for {name}\")\n        if not str(item.get(\"result_digest\", \"\")).startswith(\"sha256:\"):\n            fail(f\"result_digest missing for {name}\")\n    print(\"REPLAY_OK\", sha256_bytes(replay_path.read_bytes()))\n\nif __name__ == \"__main__\":\n    main(sys.argv)\n```\n\nRun it on fixtures that should fail. A validator that never fails is only decoration. You want red on missing hashes, unknown tools, and empty diffs.\n\nUse a boring table instead of another agent self-eval. A human still owns merge after the table returns. The table exists to block unreproducible theater.\n\n| Signal | Action | Rule you enforce | \n|---|---|---|\n| Replay file missing from the PR | Block | No transcript means no merge | \n| Tool name outside `tools.allow` | Block | The run is already dirty | \n| Digest does not match saved bytes | Block | You cannot replay this job | \n| `human_decision` is empty or`merge` | Block | The agent does not own merge | \n| Tests arrived in the same agent burst | Hold | A human rewrites those tests | \n| Replay valid and tests are human-owned | Review | You still read the diff | \n\nNotice `merge` is absent from allowed decisions. Review, reject, or isolate are the only honest exits. Auto-merge is how unreproducible patches escape.\n\nPut the replay file next to the patch in the PR. Reject the PR when the checker exits non-zero. Reject the PR when hashes and diffs disagree.\n\n```\n# Example only: local gate before you open the PR.\npython3 check_replay.py replay.json tools.allow\ngit rev-parse HEAD\ngit diff --binary \"$BASE_SHA\" > /tmp/agent.patch\npython3 - <<'PY'\nfrom pathlib import Path\nimport hashlib, json, sys\npayload = json.loads(Path(\"replay.json\").read_text())\nraw = Path(\"/tmp/agent.patch\").read_bytes()\ndigest = \"sha256:\" + hashlib.sha256(raw).hexdigest()\nif digest != payload[\"patch_digest\"]:\n    print(\"PATCH_DIGEST_MISMATCH\", digest)\n    sys.exit(2)\nprint(\"PATCH_DIGEST_OK\")\nPY\n```\n\nDo not let the agent write this checker. You write the checker and you own it. The agent may propose patches, never the gate.\n\nWhen `check_replay.py` fails, stop generating more code. You are now in forensics, not in authoring. Follow this order and do not skip steps.\n\n`replay.json` parses as UTF-8 JSON without comments.`task_hash` matches a sha256 of the prompt file.`result_digest` matches the saved artifact bytes.\nIf step six still drifts, you do not have replay yet. You have a non-deterministic tool or an unpinned runtime. Freeze the runtime before you blame the prompt.\n\nDrift usually means an unpinned tool, not a \"creative\" model. Network time, random files, and live clocks leak entropy. You stub those before you tune prompts again.\n\nRecord the stub, not the live socket. Live search and live HTTP are not replay sources. If a tool must touch the network, isolate that step from merge.\n\nThis contract does not make a weak model strong. It does not prove the patch is correct, only reconstructable. Correctness still needs tests you wrote without the agent.\n\nIt will not capture GPU nondeterminism you refused to pin. It will not redact secrets you stuffed into prompts. It will not help if you never review the diff.\n\nFloating-point tools, network calls, and clocks break naive replay. You must stub those tools or record their exact bytes. If you cannot stub them, isolate that step from merge.\n\nDo not use this as a permit for unattended production writes. Do not use this in regulated systems without a real audit log. Do not use this to rubber-stamp generated tests.\n\nSkip the JSON file if you never invoke tools. A single-shot edit in your own editor needs a review, not theater. This gate is for looped agents with side effects.\n\nIf you cannot store transcripts safely, do not start. Prompts include secrets, customer names, and private URLs. You need a redaction step before any shared sandbox.\n\nYou do not need a smarter agent to ship safer diffs. You need a replay file that survives a cold restart. If you cannot replay the agent, you do not merge.", "url": "https://wpnews.pro/news/if-you-cannot-replay-the-agent-do-not-merge", "canonical_source": "https://dev.to/airs_6907/if-you-cannot-replay-the-agent-do-not-merge-ogh", "published_at": "2026-09-11 16:01:52+00:00", "updated_at": "2026-09-11 16:11:24.717926+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/if-you-cannot-replay-the-agent-do-not-merge", "markdown": "https://wpnews.pro/news/if-you-cannot-replay-the-agent-do-not-merge.md", "text": "https://wpnews.pro/news/if-you-cannot-replay-the-agent-do-not-merge.txt", "jsonld": "https://wpnews.pro/news/if-you-cannot-replay-the-agent-do-not-merge.jsonld"}}