Debugging a failed agent run costs more than the run itself

An engineer from Loop & Retry argues that debugging a failed LLM agent run is far more expensive than the run itself, because agent failures are stochastic and non-reproducible, forcing engineers to pay a 'reproduction tax' of repeated runs and waiting time before they can even begin fixing the issue. The post quantifies this with an example where reproducing a 2% failure rate costs about $69 in tokens and engineer time before any fix is attempted.

Originally published on Loop & Retry — field notes on building LLM agents that survive production. Here is the intuition to kill: a failed run is cheap because you can just run it again. A deterministic system earns you that assumption — the bug is sitting there, reproducible on demand, and debugging is a bounded search through code you control. An agent is not that system. When a run fails, the run that failed is gone . What you re-run is a different run that happens to share a prompt, and it may well succeed. The token cost of the failure was never the expensive part. The expensive part is that you have to pay, over and over, to make the failure happen again in front of you. This post is about that second bill — the cost of reproduction — and why it dominates the cost of the actual fix. It's the cost twin of the measuring-failure-in-production post https://loopandretry.github.io/posts/measuring-agent-failure-in-production/?ref=devto : that one was about noticing silent failures; this one is about what it costs to understand one once you've noticed it. Together with the $200 postmortem https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto , this is why the cost of a failure often has nothing to do with the failure itself and everything to do with what you do or don't to recover from it. A traditional bug report comes with a stack trace, an input, and a promise: feed the input back in and you'll see the trace again. That promise is what makes debugging tractable. You bisect, you add a log line, you re-run, you narrow. Every re-run is free information because every re-run is the same run. An agent breaks that chain in two places. First, the model call is stochastic — same prompt, different sampled tokens, different tool calls, different path. Second, the world moved: the API the agent called returns different data now, the row it read got updated, the rate limiter is in a different state. So the failed trajectory isn't a function of inputs you still have. It was a function of inputs plus two sources of entropy you didn't record. Re-running is not re-observing. It's rolling the dice again and hoping for the same bad number. That changes the unit economics of debugging completely. In a deterministic system, reproduction cost is ~zero and all your money goes to the fix. In an agentic system, you pay a reproduction tax before you can even begin the fix — and if the failure is rare, that tax is enormous. Say a failure mode shows up with probability p per run. To debug it the classic way — re-run until it happens, then inspect — you need, in expectation, 1/p runs to see it once. Each run costs tokens and wall-clock. And an engineer is sitting there through all of it, which is the most expensive axis in the whole system https://loopandretry.github.io/posts/cost-beyond-tokens/?ref=devto . What it costs to reproduce one instance of a p-probability failure by re-running until it recurs. All rates illustrative — swap in yours. p = 0.02 failure probability per run a 2%-of-runs bug RUN COST = 0.05 $ in tokens + tool calls per full run RUN SECONDS = 40 wall-clock per run ENG RATE = 120 / 3600 $/second for the engineer waiting on it expected runs = 1 / p ~50 runs to see it once token cost = expected runs RUN COST $2.50 in runs wall seconds = expected runs RUN SECONDS ~33 minutes of re-running eng cost = wall seconds ENG RATE ~$67 of engineer time waiting print f"runs to reproduce : {expected runs:.0f}" print f"token cost : ${token cost:.2f}" print f"engineer time : ${eng cost:.2f}" print f"total to reproduce: ${token cost + eng cost:.2f} before any fix " runs to reproduce : 50 token cost : $2.50 engineer time : $66.67 total to reproduce: $69.17 before any fix Look at the ratio. The tokens burned re-running — the line that shows up on an invoice — are $2.50. The reproduction tax , mostly a human waiting for a dice roll to come up bad again, is nearly $70. And every bit of that is spent before anyone has written a single character of the fix. This is the same shape as the $200 postmortem https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto , except the money isn't being burned by the agent — it's being burned by you, trying to make the agent misbehave on command. And 1/p is the optimistic case, because it assumes each re-run is an independent draw from the same distribution. It isn't. The world moved, so some of your re-runs can't reproduce the bug at any p — the state that triggered it is gone. For those, expected reproduction cost isn't high, it's infinite. You will never see it again by re-running, and you'll spend the afternoon proving that. Notice what the model is actually charging you for: the cost per reproduction times the number of reproductions. Repair cost — the engineer's time once they can see the failed trajectory — barely moved between the deterministic and agentic worlds. The entire blowup is in the reproduction term. So that's the term to attack. This is part of the wider cost picture: as cost-beyond-tokens https://loopandretry.github.io/posts/cost-beyond-tokens/?ref=devto shows, engineer time spent debugging is actually one of the expensive axes in a system. Failing to instrument for replay doesn't just make debugging harder; it's a direct cost multiplier. You attack it by recording the run instead of re-rolling it. If you capture, for every step, the exact prompt sent, the exact sampled response, and the exact tool inputs and outputs, then a failed trajectory becomes a replay rather than a re-run . Reproduction cost drops from 1/p × RUN COST to approximately zero — you open the trace of the run that already failed. The two sources of entropy that made re-running useless model sampling, world state are now bytes on disk. python The always-on trace turns reproduction into a file read. def step agent, state, trace : prompt = render prompt state resp = model.call prompt the stochastic part... tool out = run tool resp.tool, resp.args ...and the moved-world part trace.append { ...both pinned to disk, per step "prompt": prompt, "response": resp.raw, "tool": resp.tool, "args": resp.args, "tool out": tool out, } return apply resp, state Debugging a failure is now: load the trace, look. No re-running, no dice. def reproduce run id : return load trace run id cost: one file read, p irrelevant The economic decision is now a one-liner. Tracing has a standing cost — storage, a little latency, some plumbing — that you pay on every run, the vast majority of which succeed. Call it TRACE COST per run. It's worth it exactly when TRACE COST < p × reproduction tax you avoid Plug the numbers in: even if tracing costs a full cent per run, and the failure is rare at 2%, the avoided reproduction tax is ~$70. The break-even failure rate is absurdly low — you'd keep tracing on for a bug that shows up in one run in seven thousand. This is why "just turn on tracing" is nearly always right for agents and merely nice-to-have for deterministic services: the value of a trace scales with how expensive reproduction is without one, and for a non-deterministic system that cost is unbounded. For a deterministic system, reproduction is free and debugging cost is repair cost. For an agent, the run that failed is a non-deterministic event you did not record, so reproduction costs 1/p re-runs of tokens and — mostly — engineer time waiting, and for world-dependent failures it's not reproducible at all. That reproduction tax, not the tokens and not the fix, is where your debugging bill goes. Record every run's prompts, samples, and tool I/O so a failure becomes a replay instead of a re-roll, and the whole tax collapses to a file read. Tracing looks like overhead until you price the afternoon you'll otherwise spend trying to make a 2% bug happen on demand.