cd /news/ai-agents/letter-to-14-07-me-replay-the-tool-t… · home topics ai-agents article
[ARTICLE · art-138356] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Letter to 14:07-Me: Replay the Tool Trace, Not the Chat

A developer published a postmortem-style guide arguing that AI coding agents should be debugged by replaying a machine-readable tool trace rather than re-reading the chat log, since chat summaries are a lossy record of tool I/O. The method pins a hashed tool schema, logs one JSON object per call to trace.jsonl with an idempotency key, and uses a model-free replay script to catch schema drift and duplicate writes. The writeup was prepared as part of MonkeyCode's product outreach, with the project described as open source.

by read7 min views3 publishedSep 23, 2026

Dear 14:07-Me,

You will waste a day on one agent loop.

The chat log will look complete and polite.

The remote tools will not match that story.

This letter is a postmortem template.

It is not a victory lap.

Treat every command below as a labeled example.

A ticket asks for a small API helper.

You paste the spec into a coding agent.

You let it call tools against a shared box.

By 18:00 the helper still flakes.

You reread the chat instead of the wire.

That is the first expensive habit.

Tool calling is not a conversation.

It is HTTP with a model in the middle.

If the envelope is missing, the day is gone.

You edited the function description mid-loop.

The model then called a field you had renamed.

Retries looked like model noise. They were schema drift.

A renamed property is not a smarter prompt.

It is a broken contract with yesterday's calls.

Hash the schema, or you will debug ghosts.

The agent posted the same resource twice.

Your debug run became production-shaped side effects.

You spent hours cleaning duplicate rows, not prompts.

A second POST is not extra evidence.

It is a second write with a new identity.

Without a key, replay is vandalism.

The model summarized a 200 as success.

The body failed your contract on id.

Chat text cannot replay. A JSONL file can.

English is a lossy codec for tool I/O.

Status, hash, and body survive. Summaries do not.

Close the thread until the file checks out.

A pinned schema file.

A one-line tool envelope.

A replay command that needs no model.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I mention MonkeyCode only as the remote runner.

The project is open source.

Operator notes list free model access and a free server option.

Those notes do not define quotas, hardware, or uptime.

Remove the product name. The method still holds.

Do this on your laptop.

Do not start the agent yet.

tools.schema.json. Example schema, labeled as a sample, not a live API:

{
  "name": "create_report",
  "method": "POST",
  "path": "/v1/reports",
  "required": ["title", "idempotency_key"],
  "properties": {
    "title": { "type": "string", "minLength": 1, "maxLength": 120 },
    "idempotency_key": { "type": "string", "pattern": "^[a-f0-9-]{36}$" }
  }
}

Check the hash with a boring command.

sha256sum tools.schema.json > tools.schema.sha256
cat tools.schema.sha256

If the agent rewrites the schema, the hash breaks.

You stop. You do not "just retry".

Mid-loop schema edits are how 14:07 becomes 18:00.

Keep a second copy outside the agent workspace.

Agents rewrite nearby files when stuck.

Your source of truth should not sit in that blast radius.

Chat is not a protocol.

Your envelope is.

Example envelope for one mutating call:

{
  "ts": "2026-09-23T14:07:00Z",
  "schema_sha256": "REPLACE_WITH_HASH",
  "tool": "create_report",
  "idempotency_key": "11111111-1111-4111-8111-111111111111",
  "request": { "title": "daily-trace" },
  "response": {
    "status": 201,
    "body": { "id": "rpt_01" }
  }
}

Rules you will keep:

idempotency_key. Name the file trace.jsonl.

One object per line. No pretty-print across lines.

Pretty JSON is for humans. JSONL is for replay.

Redact secrets before the line is written.

Authorization headers do not belong in traces.

If a token appears, delete the file and rotate it.

Follow these steps in order.

Skip none of them.

trace.jsonl on the remote box. Proposed checker (unexecuted sample):

import json, sys, hashlib, pathlib

REQUIRED = ("ts", "schema_sha256", "tool", "idempotency_key", "request", "response")

def load_schema_hash(path):
    data = pathlib.Path(path).read_bytes()
    return hashlib.sha256(data).hexdigest()

def main(trace_path, schema_path):
    expected = load_schema_hash(schema_path)
    seen_keys = set()
    errors = []
    with open(trace_path) as fh:
        for i, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            missing = [k for k in REQUIRED if k not in row]
            if missing:
                errors.append(f"line {i}: missing {missing}")
                continue
            if row["schema_sha256"] != expected:
                errors.append(f"line {i}: schema hash drift")
            key = (row["tool"], row["idempotency_key"])
            if key in seen_keys:
                errors.append(f"line {i}: duplicate idempotency key")
            seen_keys.add(key)
            status = row["response"].get("status")
            if not isinstance(status, int):
                errors.append(f"line {i}: status is not an int")
            body = row["response"].get("body") or {}
            if row["tool"].startswith("create") and status not in (200, 201):
                errors.append(f"line {i}: unexpected status {status}")
            if row["tool"].startswith("create") and status in (200, 201):
                if not isinstance(body.get("id"), str) or not body["id"]:
                    errors.append(f"line {i}: create returned no id")
    if errors:
        print("\n".join(errors))
        sys.exit(1)
    print(f"ok {len(seen_keys)} unique calls")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Run it like this:

python replay_trace.py trace.jsonl tools.schema.json

If this exits non-zero, do not prompt again.

Fix the envelope. Then rerun the checker.

The model cannot patch a missing id field with nicer prose.

Add a hard stop around the loop itself.

A shell wrapper is enough for a lab box.

if [ "$(wc -l < trace.jsonl)" -ge 20 ]; then
  echo "trace cap hit" >&2
  exit 2
fi

Twenty lines is arbitrary on purpose.

Pick a cap before the agent starts.

Do not negotiate the cap with the model.

Signal You assumed Check instead Next action
Chat says "created" Resource exists status plus bodyid Replay JSONL
Second retry "fails" Model is flaky Duplicate idempotency_key Inspect store, not prompt
Field missing in body Prompt too weak Schema hash changed Restore tools.schema.json
429 from the API Need a bigger model Loop has no backoff cap Stop the loop
Free server feels slow Hardware is the bug Trace has N identical POSTs Deduplicate keys
201 with empty id Serializer bug later Envelope body is already wrong Fail replay, do not continue

Read the table before you change the prompt.

Most of those rows are I/O bugs.

Prompt edits do not restore a hash or a key.

Here is the same afternoon as a timeline.

Use it when you start to reread the chat.

title to name in the tool text.id does not. Each hour had a cheaper check.

None of those checks required a larger model.

They required a file the agent could not narrate away.

A coding agent on your laptop mixes two risks.

Tool side effects. Untrusted generated commands.

A separate free server keeps the laptop quieter.

It does not make the trace optional.

It does not make the schema frozen.

Copy only the schema, the checker, and the empty trace.

Do not copy your laptop credentials.

Do not mount your home directory into that box.

scp tools.schema.json replay_trace.py box:~/run/
ssh box 'touch ~/run/trace.jsonl && wc -l ~/run/trace.jsonl'

If you try MonkeyCode's free models on that server, keep the same envelope.

Same hash. Same JSONL. Same replay.

The vendor is not the protocol.

Do not paste secrets into the prompt or the trace.

Redact tokens before you copy files off the box.

A free server is still a shared disk with logs.

This method does not prove business correctness.

A 201 can still store a wrong title.

Replay only proves the envelope was consistent.

JSONL is not an audit system.

Anyone with disk access can rewrite it.

Sign the file if you need a stronger claim.

cp trace.jsonl "trace-$(date -u +%Y%m%dT%H%M%SZ).jsonl"
sha256sum trace-*.jsonl

Free model access can change without notice.

A free server is not a compliance boundary.

Do not put regulated data on it.

Idempotency keys need server support.

If the API ignores the key, duplicates remain.

Test that path with two identical envelopes.

The checker above does not call the live API.

It only reads what you recorded.

A silent tool that never writes JSONL will look like success.

Clock stamps in the envelope are metadata.

They do not freeze remote state.

Do not treat ts as proof the resource still exists.

Skip this if you cannot write a schema file.

Skip this if the API has no replayable HTTP surface.

Skip this for production incident response under a clock.

Do not use a shared free server for customer PII.

Do not use it as your only backup.

Do not treat chat summaries as proof.

Skip this if your tools are purely local side-effect storms.

File deletes and package publishes need stronger isolation.

An envelope does not replace a sandbox.

Skip this if nobody will read trace.jsonl on failure.

Unused protocol files become another prompt toy.

The checker only works if you stop when it fails.

14:07-Me, stop rereading the dialogue.

Hash the schema. Append the envelope. Replay the file.

That sequence is the whole day, recovered.

If you later try the free server path, take the checker with you.

Leave the chat closed until replay_trace.py prints ok.

── more in #ai-agents 4 stories · sorted by recency
── more on @monkeycode 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/letter-to-14-07-me-r…] indexed:0 read:7min 2026-09-23 ·