Letter to 14:07-Me: Replay the Tool Trace, Not the Chat

A developer published a postmortem-style guide arguing that AI coding agents should be debugged by replaying a machine-readable tool trace rather than re-reading the chat log, since chat summaries are a lossy record of tool I/O. The method pins a hashed tool schema, logs one JSON object per call to trace.jsonl with an idempotency key, and uses a model-free replay script to catch schema drift and duplicate writes. The writeup was prepared as part of MonkeyCode's product outreach, with the project described as open source.

Dear 14:07-Me, You will waste a day on one agent loop. The chat log will look complete and polite. The remote tools will not match that story. This letter is a postmortem template. It is not a victory lap. Treat every command below as a labeled example. A ticket asks for a small API helper. You paste the spec into a coding agent. You let it call tools against a shared box. By 18:00 the helper still flakes. You reread the chat instead of the wire. That is the first expensive habit. Tool calling is not a conversation. It is HTTP with a model in the middle. If the envelope is missing, the day is gone. You edited the function description mid-loop. The model then called a field you had renamed. Retries looked like model noise. They were schema drift. A renamed property is not a smarter prompt. It is a broken contract with yesterday's calls. Hash the schema, or you will debug ghosts. The agent posted the same resource twice. Your debug run became production-shaped side effects. You spent hours cleaning duplicate rows, not prompts. A second POST is not extra evidence. It is a second write with a new identity. Without a key, replay is vandalism. The model summarized a 200 as success. The body failed your contract on id . Chat text cannot replay. A JSONL file can. English is a lossy codec for tool I/O. Status, hash, and body survive. Summaries do not. Close the thread until the file checks out. A pinned schema file. A one-line tool envelope. A replay command that needs no model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode only as the remote runner. The project is open source. Operator notes list free model access and a free server option. Those notes do not define quotas, hardware, or uptime. Remove the product name. The method still holds. Do this on your laptop. Do not start the agent yet. tools.schema.json . Example schema, labeled as a sample, not a live API: { "name": "create report", "method": "POST", "path": "/v1/reports", "required": "title", "idempotency key" , "properties": { "title": { "type": "string", "minLength": 1, "maxLength": 120 }, "idempotency key": { "type": "string", "pattern": "^ a-f0-9- {36}$" } } } Check the hash with a boring command. sha256sum tools.schema.json tools.schema.sha256 cat tools.schema.sha256 If the agent rewrites the schema, the hash breaks. You stop. You do not "just retry". Mid-loop schema edits are how 14:07 becomes 18:00. Keep a second copy outside the agent workspace. Agents rewrite nearby files when stuck. Your source of truth should not sit in that blast radius. Chat is not a protocol. Your envelope is. Example envelope for one mutating call: { "ts": "2026-09-23T14:07:00Z", "schema sha256": "REPLACE WITH HASH", "tool": "create report", "idempotency key": "11111111-1111-4111-8111-111111111111", "request": { "title": "daily-trace" }, "response": { "status": 201, "body": { "id": "rpt 01" } } } Rules you will keep: idempotency key . Name the file trace.jsonl . One object per line. No pretty-print across lines. Pretty JSON is for humans. JSONL is for replay. Redact secrets before the line is written. Authorization headers do not belong in traces. If a token appears, delete the file and rotate it. Follow these steps in order. Skip none of them. trace.jsonl on the remote box. Proposed checker unexecuted sample : python replay trace.py — sample harness, not production code import json, sys, hashlib, pathlib REQUIRED = "ts", "schema sha256", "tool", "idempotency key", "request", "response" def load schema hash path : data = pathlib.Path path .read bytes return hashlib.sha256 data .hexdigest def main trace path, schema path : expected = load schema hash schema path seen keys = set errors = with open trace path as fh: for i, line in enumerate fh, 1 : line = line.strip if not line: continue row = json.loads line missing = k for k in REQUIRED if k not in row if missing: errors.append f"line {i}: missing {missing}" continue if row "schema sha256" = expected: errors.append f"line {i}: schema hash drift" key = row "tool" , row "idempotency key" if key in seen keys: errors.append f"line {i}: duplicate idempotency key" seen keys.add key status = row "response" .get "status" if not isinstance status, int : errors.append f"line {i}: status is not an int" body = row "response" .get "body" or {} if row "tool" .startswith "create" and status not in 200, 201 : errors.append f"line {i}: unexpected status {status}" if row "tool" .startswith "create" and status in 200, 201 : if not isinstance body.get "id" , str or not body "id" : errors.append f"line {i}: create returned no id" if errors: print "\n".join errors sys.exit 1 print f"ok {len seen keys } unique calls" if name == " main ": main sys.argv 1 , sys.argv 2 Run it like this: python replay trace.py trace.jsonl tools.schema.json If this exits non-zero, do not prompt again. Fix the envelope. Then rerun the checker. The model cannot patch a missing id field with nicer prose. Add a hard stop around the loop itself. A shell wrapper is enough for a lab box. labeled example: stop after 20 envelopes if "$ wc -l < trace.jsonl " -ge 20 ; then echo "trace cap hit" &2 exit 2 fi Twenty lines is arbitrary on purpose. Pick a cap before the agent starts. Do not negotiate the cap with the model. | Signal | You assumed | Check instead | Next action | |---|---|---|---| | Chat says "created" | Resource exists | status plus body id | Replay JSONL | | Second retry "fails" | Model is flaky | Duplicate idempotency key | Inspect store, not prompt | | Field missing in body | Prompt too weak | Schema hash changed | Restore tools.schema.json | | 429 from the API | Need a bigger model | Loop has no backoff cap | Stop the loop | | Free server feels slow | Hardware is the bug | Trace has N identical POSTs | Deduplicate keys | | 201 with empty id | Serializer bug later | Envelope body is already wrong | Fail replay, do not continue | Read the table before you change the prompt. Most of those rows are I/O bugs. Prompt edits do not restore a hash or a key. Here is the same afternoon as a timeline. Use it when you start to reread the chat. title to name in the tool text. id does not. Each hour had a cheaper check. None of those checks required a larger model. They required a file the agent could not narrate away. A coding agent on your laptop mixes two risks. Tool side effects. Untrusted generated commands. A separate free server keeps the laptop quieter. It does not make the trace optional. It does not make the schema frozen. Copy only the schema, the checker, and the empty trace. Do not copy your laptop credentials. Do not mount your home directory into that box. labeled example: ship the protocol, not your machine scp tools.schema.json replay trace.py box:~/run/ ssh box 'touch ~/run/trace.jsonl && wc -l ~/run/trace.jsonl' If you try MonkeyCode's free models on that server, keep the same envelope. Same hash. Same JSONL. Same replay. The vendor is not the protocol. Do not paste secrets into the prompt or the trace. Redact tokens before you copy files off the box. A free server is still a shared disk with logs. This method does not prove business correctness. A 201 can still store a wrong title. Replay only proves the envelope was consistent. JSONL is not an audit system. Anyone with disk access can rewrite it. Sign the file if you need a stronger claim. labeled example: detach a copy you can compare later cp trace.jsonl "trace-$ date -u +%Y%m%dT%H%M%SZ .jsonl" sha256sum trace- .jsonl Free model access can change without notice. A free server is not a compliance boundary. Do not put regulated data on it. Idempotency keys need server support. If the API ignores the key, duplicates remain. Test that path with two identical envelopes. The checker above does not call the live API. It only reads what you recorded. A silent tool that never writes JSONL will look like success. Clock stamps in the envelope are metadata. They do not freeze remote state. Do not treat ts as proof the resource still exists. Skip this if you cannot write a schema file. Skip this if the API has no replayable HTTP surface. Skip this for production incident response under a clock. Do not use a shared free server for customer PII. Do not use it as your only backup. Do not treat chat summaries as proof. Skip this if your tools are purely local side-effect storms. File deletes and package publishes need stronger isolation. An envelope does not replace a sandbox. Skip this if nobody will read trace.jsonl on failure. Unused protocol files become another prompt toy. The checker only works if you stop when it fails. 14:07-Me, stop rereading the dialogue. Hash the schema. Append the envelope. Replay the file. That sequence is the whole day, recovered. If you later try the free server path, take the checker with you. Leave the chat closed until replay trace.py prints ok .