Your agent's failures are silent: measuring failure modes in production

An engineer from Loop & Retry details how LLM agents fail silently in production, unlike traditional services that throw exceptions. The post introduces a taxonomy of failure outcomes—success, hard_error, budget_exhausted, gave_up, looped, and wrong—and emphasizes labeling every run with an explicit outcome to surface hidden failures. The engineer provides code for terminal-state logging and argues that distinguishing failure types is critical for effective debugging and improvement.

Originally published on Loop & Retry — field notes on building LLM agents that survive production. The failure that hurts is the one that doesn't throw. A traditional service fails loudly: an exception, a 500, a stack trace, a red line on a dashboard. An agent fails quietly. It runs to completion, returns a confident answer, exits zero — and the answer is wrong, or it spent forty steps and \$3 to conclude it couldn't do the thing, or it looped politely until it hit a cap nobody's watching. The $200 postmortem https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto was a loud failure I happened to catch because costs spiked. The expensive ones are the quiet failures you never labeled, because you can't alert on a category you don't record. And for subjective tasks, your LLM judge might be hiding failures that look like success https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/?ref=devto in its biased measurements. What to measure when your agent works https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/?ref=devto covered the happy path. This is the inverse: what to measure when it doesn't , and how to know that it didn't. Here's the trap. Your agent has a try/except at the top of the loop. Exceptions get logged, counted, alerted. Your error rate looks like 0.5% and everyone's happy. Meanwhile: {"results": } and the agent treated empty as "done."None of those increment your exception counter. All of them are failures. Your real failure rate isn't 0.5%; it's 0.5% that you can see plus an unknown, larger number you can't. Step one is to make every run end in a labeled outcome, not just "exception or not." Every agent run should terminate with an explicit, recorded outcome. Not a boolean — a category. The minimum useful set: | Outcome | What happened | How you detect it | |---|---|---| success | Task done, verified | A post-hoc check passed see below | hard error | Exception, crash, unrecoverable tool failure | The one you already catch | budget exhausted | Hit a step / token / time cap mid-task | The cap fired before a terminal state | gave up | Agent declared it couldn't finish | Model emitted a "cannot complete" terminal action | looped | Repeated states without progress | Progress detector tripped | wrong The point of the taxonomy is that these have different fixes . budget exhausted means your caps are too tight or your task is too big — raise the cap or decompose. gave up means a capability or tool gap — the agent knew it was stuck, which is the good failure. looped means your loop lacks a progress check. wrong is the dangerous one, because it's indistinguishable from success at runtime. Collapsing all of these into "error rate" throws away exactly the information that tells you what to do. Terminal-state logging. The single highest-value change: make the loop's exit path assign an outcome. If you fall out of the loop because a cap fired, that's budget exhausted — don't let it masquerade as success. python def run agent task, step cap=40, token cap=200 000 : state = init task for step in range step cap : action = model step state if action.is terminal: outcome = "gave up" if action.type == "cannot complete" else "success" return finish state, outcome, step, tokens state if tokens state token cap: return finish state, "budget exhausted", step, tokens state , cap="token" state = apply action, state return finish state, "budget exhausted", step cap, tokens state , cap="step" def finish state, outcome, steps, toks, cap=None : log.info "agent run end", outcome=outcome, steps=steps, tokens=toks, cap=cap return state.result, outcome Now outcome is a dimension you can group by. "What fraction of runs hit the step cap this week?" becomes a query instead of a mystery. A progress detector for looped. A cheap one: hash the salient state open goals, last tool called + args each step and count repeats. Three visits to the same hash without a new goal closing means no progress — break with looped . This turns an invisible, expensive non-termination into a labeled, bounded event you can alert on. Post-hoc verification for wrong. This is the hard one, because wrong looks identical to success while the run is happening. You cannot catch it at runtime; you catch it Once outcomes are labeled, two derived metrics tell you almost everything: Silent-failure ratio — budget exhausted + gave up + looped + wrong / total , i.e. failures that didn't throw, over all runs. This is the number your exception counter was hiding. Track it as your true failure rate. If it's an order of magnitude above your exception rate — and it usually is at first — that gap is your observability debt. Cost of failure — tokens and dollars spent on runs that ended in anything but success . A wrong run that took forty steps cost you a full run's tokens and whatever the bad output does downstream. Attribute spend to outcome and you'll often find a large slice of your bill is being burned by a small slice of runs failing expensively — the same shape as the $200 incident https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto , just spread thin enough that no single night sets off an alarm. If your agent monitoring only counts exceptions, you're measuring the failures that were kind enough to crash. The ones that cost you are silent: they exhaust a budget, give up, loop, or return a confident wrong answer with exit code zero. Make every run end in a labeled outcome , add a progress detector and post-hoc sampling, and track the silent-failure ratio as your real failure rate. You can't fix a failure mode you've never named — and the whole reason agents feel unreliable in production is that most teams are naming exactly one of them. And if you're running multi-agent crews https://loopandretry.github.io/posts/multi-agent-failure-modes/?ref=devto , instrument the failure modes that are unique to crew coordination: agent disagreement, circular delegation, cascading errors across the team.