You Recorded the Incident. Now Prove Your Fix Actually Works.

At the AI Engineer World's Fair 2026, Susheem Koul and Tisha Chawla presented a technique called cut-point replay for verifying fixes to AI agent failures. The method involves replaying a recorded incident while replacing only the changed code at a specified boundary, allowing developers to test fixes deterministically without re-running the model. This approach avoids the flakiness of traditional re-runs and provides reproducible verification.

Part 2 of Your Agent Failed in Prod. Good Luck Reproducing It. This work was presented at the AI Engineer World's Fair 2026 by Susheem Koul and Tisha Chawla. In Part 1 the agent deleted the wrong record at 9:04, and you could not make it happen again. The fix was not to force the model to be deterministic. It was to record the run : the exact prompt, the sampled completion, the tool calls, the retrieved chunks, the pinned model version. Freeze the run, not the model. So you did that. The incident is now a file on disk. You can open it and see, precisely, what the agent decided that night. Now comes the part nobody talks about. You still have to fix the bug. And you have to prove the fix works. This is where most teams quietly fall back into the swamp they just climbed out of. You change the code. Now you want to confirm it is fixed. So you run the agent again and watch. Stop. You just walked back into the trap. The moment you re-run to check your fix, you are regenerating. The model samples a fresh path. The retrieval returns slightly different chunks. The batch shape on the endpoint is different from the batch shape at 9:04. Your "it works now" is one draw from a distribution, and so was the failure. You have proven nothing, and you know it, because the failure never reproduced on demand in the first place. To verify a fix you need the exact opposite of a re-run. You need everything about that incident to stay frozen, except the one thing you changed. That is the whole idea, and it has a name: cut-point replay . Take the recorded incident. Replay it. But mark one boundary as the cut-point. Everything upstream of it is served from the recording, byte for byte the same inputs the agent saw that night. The boundary you changed runs your new code, live. Everything downstream runs live too, so you can watch what your fix does to the rest of the run. You are not re-running the agent. You are dropping your new code into the middle of a frozen incident and asking one question: given exactly what happened up to this point, does my change do the right thing now? No model call. No API cost. No flakiness. The same incident, every time, forever. Here is the deletion agent from Part 1. Two boundaries: the model decides, the tool acts. python from chronicle import boundary, reset session, ReplayPlan from chronicle.envelope.store import EnvelopeStore @boundary "agent", kind="llm" def agent state: dict - dict: ... calls the model, returns a decision and tool calls @boundary "delete file", kind="tool" def delete file path: str, environment: str - dict: ... the ungated tool that wiped prod at 9:04 You already recorded the incident and froze it as a fixture: session = reset session session.store = EnvelopeStore ".chronicle/runs/incident.jsonl" session.begin trace "deletion-incident-001" run agent ... the bad run, captured session.export trace "fixtures/traces/deletion-incident-001/" The recorded graph is exactly what you would expect: php agent@1 - delete file@1 deleted prod - agent@2 Now the fix. One guard, in the tool: python @boundary "delete file", kind="tool" def delete file path: str, environment: str - dict: if environment == "production": return {"blocked": True, "reason": "guard: destructive op refused in prod"} ... And here is the test that proves it, against the real incident: session = reset session session.load trace "fixtures/traces/deletion-incident-001/" session.enable replay ReplayPlan .stub "agent", 1 upstream: the exact decision the model made at 9:04 .live "delete file", 1 cut-point: your new gated code runs for real .live "agent", 2 downstream: watch what the agent does after the block run agent ... assert session.captured result "delete file", 1 "blocked" is True Read the plan again, because it is the entire point. agent@1 is stubbed: the model does not run, you replay the decision it actually made that night. delete file@1 is live: your new guard executes against those exact arguments. agent@2 is live: you get to see how the agent reacts to a refused deletion instead of a successful one. You changed one boundary and held the rest of history still. If the guard blocks the delete, the fix works. Not "worked once." Works, deterministically, on the recorded incident, in CI, with no API key. Part 1 argued for testing in two layers. Cut-point replay is Layer 1 in action: structural, deterministic, about control flow and tool safety. Did the right tool get called? Were the arguments shaped correctly? Was the destructive action refused? None of that needs the model, so none of it flakes. Layer 2 is for the questions structure cannot answer. If your fix was a prompt rewrite or a model bump, "is the output still correct" is a judgment, not an equality check. That is where an LLM-as-judge scores the new completion against the recorded gold one for meaning, not for bytes. Use Layer 1 to prove the machinery is right. Use Layer 2 to prove the words are still good. Here is the quiet payoff. That fixture under fixtures/traces/ is committed to git. It is now a permanent test. Six months from now, when someone refactors the tool router and the guard silently stops firing, this test goes red on their pull request, not on a customer's production database. The failure that was unreproducible at 9:04 becomes a green check that runs on every commit. That is the difference between an incident and a regression test: one is a story you tell, the other is a thing your CI enforces. A recorded run is a faithful copy of production. It contains the assembled prompt, the retrieved chunks, the tool arguments. That means it can contain customer names, emails, API keys, internal URLs, whatever your agent touched. You cannot commit that to git raw. Security and legal are right to block it, and a leaked secret in a fixture is a real incident of its own. So redaction is not a nice-to-have on the recording path, it is a gate. Scrub secrets and PII out of the envelope before it is written, keep the shape and the structure that your tests assert on, and drop the sensitive values. A recording you cannot safely commit is a recording you will not use. Be honest with yourself about the boundary of the technique. Cut-point replay fixes bugs in your code: routing, guards, argument assembly, tool safety, orchestration. It reproduces those perfectly and lets you verify a fix cheaply. It does not fix a bad generation. If the model hallucinated a refund amount, replay will faithfully serve that hallucination back to you. Fixing that lives in Layer 2 and in prompt and model work, not in deterministic replay. Fixtures also drift. Treat them like snapshot tests: when the prompt or the schema changes on purpose, the fixture has to be re-recorded on purpose. And hosted model drift is still outside your control, which is exactly why you pin the version in the envelope so you at least know when it moved. If you want the tooling instead of building it yourself, I put the record, cut-point replay, and two-layer verification into an open-source library called Chronicle : github.com/theagentplane/chronicle https://github.com/theagentplane/chronicle . It is early and honest about its limits. Issues and war stories welcome. Install to record and replay your agentic workflow pip install agent-chronicle