If You Cannot Replay the Agent, Do Not Merge

You should refuse any agent patch you cannot replay. A scrolling demo is not a build artifact. Merge the replay file first, then consider the diff.

Unreproducible agents are not production-grade coding assistants. They behave like unrecorded pair programmers with total amnesia. You would not accept that from a human contractor.

You would demand logs from a flaky integration test. Demand the same evidence from every coding-agent session. Otherwise you are shipping a one-time stage performance.

Opinion, not a hedged framework dump: replay or reject. Clever plans cannot replace a missing tool transcript. Your future self cannot debug a vanished chain of tool calls.

Replay is not running the same prompt and hoping. Replay means stored inputs reconstruct the same side effects. You keep enough evidence to rebuild the job later.

A useful replay file must answer six concrete questions.

If any row is missing, you do not have a replay. You have a diary entry with extra punctuation. Diary entries do not belong on the main branch.

Free or cheap tokens hide the cost of wasted loops. You retry until something compiles, then forget the path. Cheap retries without transcripts create irreproducible success stories.

Teams then paste a green terminal into Slack threads. Nobody can name the tool order that produced files. The next intern cannot reconstruct the change two weeks later.

That is how "the agent just knows the repo" myths start. The agent does not actually know your repository. It sampled a path you failed to record.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that sandbox to practice the replay contract, not to skip it.

Keep the same checker when you later pay for inference. Free sandboxes do not lower the merge bar. They only keep experiments off your invoice.

Watch for patches that exist only inside a chat pane. Watch for tool traces truncated to look "readable" in demos. Watch for model labels copied from a pricing page.

If the transcript was edited by hand, discard the run. If file hashes drifted after cleanup, discard the run. If the PR description is only vibes, demand the file.

Pin a JSON contract before you write more agent glue. The schema below is an example, not a shipped standard. Adapt field names to your orchestrator without adding poetry.

{
  "schema_version": "1",
  "task_id": "ISSUE-1842",
  "task_hash": "sha256:REPLACE_WITH_PROMPT_FILE_HASH",
  "model_id": "pinned-opaque-string",
  "base_git_sha": "abc1234",
  "allowed_tools": ["read_file", "write_file", "run_tests"],
  "tool_calls": [
    {
      "seq": 1,
      "name": "read_file",
      "args_digest": "sha256:ARGS",
      "result_digest": "sha256:BYTES"
    }
  ],
  "patch_digest": "sha256:GIT_DIFF",
  "human_decision": "review",
  "secrets_redacted": true
}

Every tool_calls item needs a digest, not a novel. Store stdout hashes when the bytes are large. Store raw text only when the payload is tiny and non-secret.

Keep a tight allowlist beside that JSON file. Unknown names fail the run. Broad shell access does not belong in the first draft.

read_file
write_file
run_tests

Validate the file in CI before you discuss the patch. The checker below is an example, not a product. Do not treat a passing checker as proof of model quality.

#!/usr/bin/env python3
"""Example only: unexecuted replay-file checker. Not a benchmark."""
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

REQUIRED = {
    "schema_version",
    "task_hash",
    "model_id",
    "base_git_sha",
    "allowed_tools",
    "tool_calls",
    "patch_digest",
    "human_decision",
    "secrets_redacted",
}

def sha256_bytes(data: bytes) -> str:
    return "sha256:" + hashlib.sha256(data).hexdigest()

def fail(message: str) -> None:
    print("REPLAY_FAIL:", message)
    raise SystemExit(2)

def load_allowlist(path: Path) -> set[str]:
    names = set()
    for line in path.read_text(encoding="utf-8").splitlines():
        text = line.strip()
        if text and not text.startswith("#"):
            names.add(text)
    if not names:
        fail("allowlist is empty")
    return names

def main(argv: list[str]) -> None:
    if len(argv) != 3:
        fail("usage: check_replay.py replay.json tools.allow")
    replay_path = Path(argv[1])
    allow_path = Path(argv[2])
    payload = json.loads(replay_path.read_text(encoding="utf-8"))
    missing = REQUIRED - set(payload)
    if missing:
        fail(f"missing keys: {sorted(missing)}")
    if payload.get("secrets_redacted") is not True:
        fail("secrets_redacted must be true")
    if payload.get("human_decision") not in {"review", "reject", "isolate"}:
        fail("human_decision is not an allowed value")
    if not str(payload.get("patch_digest", "")).startswith("sha256:"):
        fail("patch_digest is not a sha256 digest")
    allow = load_allowlist(allow_path)
    declared = set(payload.get("allowed_tools") or [])
    if declared - allow:
        fail(f"tools not in allowlist file: {sorted(declared - allow)}")
    calls = payload.get("tool_calls") or []
    if not isinstance(calls, list) or not calls:
        fail("tool_calls must be a non-empty list")
    for item in calls:
        name = item.get("name")
        if name not in allow:
            fail(f"tool call not allowed: {name}")
        if not str(item.get("args_digest", "")).startswith("sha256:"):
            fail(f"args_digest missing for {name}")
        if not str(item.get("result_digest", "")).startswith("sha256:"):
            fail(f"result_digest missing for {name}")
    print("REPLAY_OK", sha256_bytes(replay_path.read_bytes()))

if __name__ == "__main__":
    main(sys.argv)

Run it on fixtures that should fail. A validator that never fails is only decoration. You want red on missing hashes, unknown tools, and empty diffs.

Use a boring table instead of another agent self-eval. A human still owns merge after the table returns. The table exists to block unreproducible theater.

Notice merge is absent from allowed decisions. Review, reject, or isolate are the only honest exits. Auto-merge is how unreproducible patches escape.

Put the replay file next to the patch in the PR. Reject the PR when the checker exits non-zero. Reject the PR when hashes and diffs disagree.

python3 check_replay.py replay.json tools.allow
git rev-parse HEAD
git diff --binary "$BASE_SHA" > /tmp/agent.patch
python3 - <<'PY'
from pathlib import Path
import hashlib, json, sys
payload = json.loads(Path("replay.json").read_text())
raw = Path("/tmp/agent.patch").read_bytes()
digest = "sha256:" + hashlib.sha256(raw).hexdigest()
if digest != payload["patch_digest"]:
    print("PATCH_DIGEST_MISMATCH", digest)
    sys.exit(2)
print("PATCH_DIGEST_OK")
PY

Do not let the agent write this checker. You write the checker and you own it. The agent may propose patches, never the gate.

When check_replay.py fails, stop generating more code. You are now in forensics, not in authoring. Follow this order and do not skip steps.

replay.json parses as UTF-8 JSON without comments.task_hash matches a sha256 of the prompt file.result_digest matches the saved artifact bytes. If step six still drifts, you do not have replay yet. You have a non-deterministic tool or an unpinned runtime. Freeze the runtime before you blame the prompt.

Drift usually means an unpinned tool, not a "creative" model. Network time, random files, and live clocks leak entropy. You stub those before you tune prompts again.

Record the stub, not the live socket. Live search and live HTTP are not replay sources. If a tool must touch the network, isolate that step from merge.

This contract does not make a weak model strong. It does not prove the patch is correct, only reconstructable. Correctness still needs tests you wrote without the agent.

It will not capture GPU nondeterminism you refused to pin. It will not redact secrets you stuffed into prompts. It will not help if you never review the diff.

Floating-point tools, network calls, and clocks break naive replay. You must stub those tools or record their exact bytes. If you cannot stub them, isolate that step from merge.

Do not use this as a permit for unattended production writes. Do not use this in regulated systems without a real audit log. Do not use this to rubber-stamp generated tests.

Skip the JSON file if you never invoke tools. A single-shot edit in your own editor needs a review, not theater. This gate is for looped agents with side effects.

If you cannot store transcripts safely, do not start. Prompts include secrets, customer names, and private URLs. You need a redaction step before any shared sandbox.

You do not need a smarter agent to ship safer diffs. You need a replay file that survives a cold restart. If you cannot replay the agent, you do not merge.