{"slug": "letter-to-14-07-me-replay-the-tool-trace-not-the-chat", "title": "Letter to 14:07-Me: Replay the Tool Trace, Not the Chat", "summary": "A developer published a postmortem-style guide arguing that AI coding agents should be debugged by replaying a machine-readable tool trace rather than re-reading the chat log, since chat summaries are a lossy record of tool I/O. The method pins a hashed tool schema, logs one JSON object per call to trace.jsonl with an idempotency key, and uses a model-free replay script to catch schema drift and duplicate writes. The writeup was prepared as part of MonkeyCode's product outreach, with the project described as open source.", "body_md": "Dear 14:07-Me,\n\nYou will waste a day on one agent loop.\n\nThe chat log will look complete and polite.\n\nThe remote tools will not match that story.\n\nThis letter is a postmortem template.\n\nIt is not a victory lap.\n\nTreat every command below as a labeled example.\n\nA ticket asks for a small API helper.\n\nYou paste the spec into a coding agent.\n\nYou let it call tools against a shared box.\n\nBy 18:00 the helper still flakes.\n\nYou reread the chat instead of the wire.\n\nThat is the first expensive habit.\n\nTool calling is not a conversation.\n\nIt is HTTP with a model in the middle.\n\nIf the envelope is missing, the day is gone.\n\nYou edited the function description mid-loop.\n\nThe model then called a field you had renamed.\n\nRetries looked like model noise. They were schema drift.\n\nA renamed property is not a smarter prompt.\n\nIt is a broken contract with yesterday's calls.\n\nHash the schema, or you will debug ghosts.\n\nThe agent posted the same resource twice.\n\nYour debug run became production-shaped side effects.\n\nYou spent hours cleaning duplicate rows, not prompts.\n\nA second POST is not extra evidence.\n\nIt is a second write with a new identity.\n\nWithout a key, replay is vandalism.\n\nThe model summarized a 200 as success.\n\nThe body failed your contract on `id`.\n\nChat text cannot replay. A JSONL file can.\n\nEnglish is a lossy codec for tool I/O.\n\nStatus, hash, and body survive. Summaries do not.\n\nClose the thread until the file checks out.\n\nA pinned schema file.\n\nA one-line tool envelope.\n\nA replay command that needs no model.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nI mention MonkeyCode only as the remote runner.\n\nThe project is open source.\n\nOperator notes list free model access and a free server option.\n\nThose notes do not define quotas, hardware, or uptime.\n\nRemove the product name. The method still holds.\n\nDo this on your laptop.\n\nDo not start the agent yet.\n\n`tools.schema.json`.\nExample schema, labeled as a sample, not a live API:\n\n```\n{\n  \"name\": \"create_report\",\n  \"method\": \"POST\",\n  \"path\": \"/v1/reports\",\n  \"required\": [\"title\", \"idempotency_key\"],\n  \"properties\": {\n    \"title\": { \"type\": \"string\", \"minLength\": 1, \"maxLength\": 120 },\n    \"idempotency_key\": { \"type\": \"string\", \"pattern\": \"^[a-f0-9-]{36}$\" }\n  }\n}\n```\n\nCheck the hash with a boring command.\n\n```\nsha256sum tools.schema.json > tools.schema.sha256\ncat tools.schema.sha256\n```\n\nIf the agent rewrites the schema, the hash breaks.\n\nYou stop. You do not \"just retry\".\n\nMid-loop schema edits are how 14:07 becomes 18:00.\n\nKeep a second copy outside the agent workspace.\n\nAgents rewrite nearby files when stuck.\n\nYour source of truth should not sit in that blast radius.\n\nChat is not a protocol.\n\nYour envelope is.\n\nExample envelope for one mutating call:\n\n```\n{\n  \"ts\": \"2026-09-23T14:07:00Z\",\n  \"schema_sha256\": \"REPLACE_WITH_HASH\",\n  \"tool\": \"create_report\",\n  \"idempotency_key\": \"11111111-1111-4111-8111-111111111111\",\n  \"request\": { \"title\": \"daily-trace\" },\n  \"response\": {\n    \"status\": 201,\n    \"body\": { \"id\": \"rpt_01\" }\n  }\n}\n```\n\nRules you will keep:\n\n`idempotency_key`.\nName the file `trace.jsonl`.\n\nOne object per line. No pretty-print across lines.\n\nPretty JSON is for humans. JSONL is for replay.\n\nRedact secrets before the line is written.\n\nAuthorization headers do not belong in traces.\n\nIf a token appears, delete the file and rotate it.\n\nFollow these steps in order.\n\nSkip none of them.\n\n`trace.jsonl` on the remote box.\nProposed checker (unexecuted sample):\n\n``` python\n# replay_trace.py — sample harness, not production code\nimport json, sys, hashlib, pathlib\n\nREQUIRED = (\"ts\", \"schema_sha256\", \"tool\", \"idempotency_key\", \"request\", \"response\")\n\ndef load_schema_hash(path):\n    data = pathlib.Path(path).read_bytes()\n    return hashlib.sha256(data).hexdigest()\n\ndef main(trace_path, schema_path):\n    expected = load_schema_hash(schema_path)\n    seen_keys = set()\n    errors = []\n    with open(trace_path) as fh:\n        for i, line in enumerate(fh, 1):\n            line = line.strip()\n            if not line:\n                continue\n            row = json.loads(line)\n            missing = [k for k in REQUIRED if k not in row]\n            if missing:\n                errors.append(f\"line {i}: missing {missing}\")\n                continue\n            if row[\"schema_sha256\"] != expected:\n                errors.append(f\"line {i}: schema hash drift\")\n            key = (row[\"tool\"], row[\"idempotency_key\"])\n            if key in seen_keys:\n                errors.append(f\"line {i}: duplicate idempotency key\")\n            seen_keys.add(key)\n            status = row[\"response\"].get(\"status\")\n            if not isinstance(status, int):\n                errors.append(f\"line {i}: status is not an int\")\n            body = row[\"response\"].get(\"body\") or {}\n            if row[\"tool\"].startswith(\"create\") and status not in (200, 201):\n                errors.append(f\"line {i}: unexpected status {status}\")\n            if row[\"tool\"].startswith(\"create\") and status in (200, 201):\n                if not isinstance(body.get(\"id\"), str) or not body[\"id\"]:\n                    errors.append(f\"line {i}: create returned no id\")\n    if errors:\n        print(\"\\n\".join(errors))\n        sys.exit(1)\n    print(f\"ok {len(seen_keys)} unique calls\")\n\nif __name__ == \"__main__\":\n    main(sys.argv[1], sys.argv[2])\n```\n\nRun it like this:\n\n```\npython replay_trace.py trace.jsonl tools.schema.json\n```\n\nIf this exits non-zero, do not prompt again.\n\nFix the envelope. Then rerun the checker.\n\nThe model cannot patch a missing `id` field with nicer prose.\n\nAdd a hard stop around the loop itself.\n\nA shell wrapper is enough for a lab box.\n\n```\n# labeled example: stop after 20 envelopes\nif [ \"$(wc -l < trace.jsonl)\" -ge 20 ]; then\n  echo \"trace cap hit\" >&2\n  exit 2\nfi\n```\n\nTwenty lines is arbitrary on purpose.\n\nPick a cap before the agent starts.\n\nDo not negotiate the cap with the model.\n\n| Signal | You assumed | Check instead | Next action | \n|---|---|---|---|\n| Chat says \"created\" | Resource exists | `status` plus body`id` | Replay JSONL | \n| Second retry \"fails\" | Model is flaky | Duplicate `idempotency_key` | Inspect store, not prompt | \n| Field missing in body | Prompt too weak | Schema hash changed | Restore `tools.schema.json` | \n| 429 from the API | Need a bigger model | Loop has no backoff cap | Stop the loop | \n| Free server feels slow | Hardware is the bug | Trace has N identical POSTs | Deduplicate keys | \n| 201 with empty `id` | Serializer bug later | Envelope body is already wrong | Fail replay, do not continue | \n\nRead the table before you change the prompt.\n\nMost of those rows are I/O bugs.\n\nPrompt edits do not restore a hash or a key.\n\nHere is the same afternoon as a timeline.\n\nUse it when you start to reread the chat.\n\n`title` to `name` in the tool text.`id` does not.\nEach hour had a cheaper check.\n\nNone of those checks required a larger model.\n\nThey required a file the agent could not narrate away.\n\nA coding agent on your laptop mixes two risks.\n\nTool side effects. Untrusted generated commands.\n\nA separate free server keeps the laptop quieter.\n\nIt does not make the trace optional.\n\nIt does not make the schema frozen.\n\nCopy only the schema, the checker, and the empty trace.\n\nDo not copy your laptop credentials.\n\nDo not mount your home directory into that box.\n\n```\n# labeled example: ship the protocol, not your machine\nscp tools.schema.json replay_trace.py box:~/run/\nssh box 'touch ~/run/trace.jsonl && wc -l ~/run/trace.jsonl'\n```\n\nIf you try MonkeyCode's free models on that server, keep the same envelope.\n\nSame hash. Same JSONL. Same replay.\n\nThe vendor is not the protocol.\n\nDo not paste secrets into the prompt or the trace.\n\nRedact tokens before you copy files off the box.\n\nA free server is still a shared disk with logs.\n\nThis method does not prove business correctness.\n\nA 201 can still store a wrong title.\n\nReplay only proves the envelope was consistent.\n\nJSONL is not an audit system.\n\nAnyone with disk access can rewrite it.\n\nSign the file if you need a stronger claim.\n\n```\n# labeled example: detach a copy you can compare later\ncp trace.jsonl \"trace-$(date -u +%Y%m%dT%H%M%SZ).jsonl\"\nsha256sum trace-*.jsonl\n```\n\nFree model access can change without notice.\n\nA free server is not a compliance boundary.\n\nDo not put regulated data on it.\n\nIdempotency keys need server support.\n\nIf the API ignores the key, duplicates remain.\n\nTest that path with two identical envelopes.\n\nThe checker above does not call the live API.\n\nIt only reads what you recorded.\n\nA silent tool that never writes JSONL will look like success.\n\nClock stamps in the envelope are metadata.\n\nThey do not freeze remote state.\n\nDo not treat `ts` as proof the resource still exists.\n\nSkip this if you cannot write a schema file.\n\nSkip this if the API has no replayable HTTP surface.\n\nSkip this for production incident response under a clock.\n\nDo not use a shared free server for customer PII.\n\nDo not use it as your only backup.\n\nDo not treat chat summaries as proof.\n\nSkip this if your tools are purely local side-effect storms.\n\nFile deletes and package publishes need stronger isolation.\n\nAn envelope does not replace a sandbox.\n\nSkip this if nobody will read `trace.jsonl` on failure.\n\nUnused protocol files become another prompt toy.\n\nThe checker only works if you stop when it fails.\n\n14:07-Me, stop rereading the dialogue.\n\nHash the schema. Append the envelope. Replay the file.\n\nThat sequence is the whole day, recovered.\n\nIf you later try the free server path, take the checker with you.\n\nLeave the chat closed until `replay_trace.py` prints `ok`.", "url": "https://wpnews.pro/news/letter-to-14-07-me-replay-the-tool-trace-not-the-chat", "canonical_source": "https://dev.to/codejs_8314/letter-to-1407-me-replay-the-tool-trace-not-the-chat-28j3", "published_at": "2026-09-23 16:46:15+00:00", "updated_at": "2026-09-23 16:59:00.095780+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "ai-products"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/letter-to-14-07-me-replay-the-tool-trace-not-the-chat", "markdown": "https://wpnews.pro/news/letter-to-14-07-me-replay-the-tool-trace-not-the-chat.md", "text": "https://wpnews.pro/news/letter-to-14-07-me-replay-the-tool-trace-not-the-chat.txt", "jsonld": "https://wpnews.pro/news/letter-to-14-07-me-replay-the-tool-trace-not-the-chat.jsonld"}}