{"slug": "your-agent-s-failures-are-silent-measuring-failure-modes-in-production", "title": "Your agent's failures are silent: measuring failure modes in production", "summary": "An engineer from Loop & Retry details how LLM agents fail silently in production, unlike traditional services that throw exceptions. The post introduces a taxonomy of failure outcomes—success, hard_error, budget_exhausted, gave_up, looped, and wrong—and emphasizes labeling every run with an explicit outcome to surface hidden failures. The engineer provides code for terminal-state logging and argues that distinguishing failure types is critical for effective debugging and improvement.", "body_md": "*Originally published on Loop & Retry — field notes on building LLM agents that survive production.*\n\nThe failure that hurts is the one that doesn't throw. A traditional service fails loudly: an exception, a 500, a stack trace, a red line on a dashboard. An agent fails quietly. It runs to completion, returns a confident answer, exits zero — and the answer is wrong, or it spent forty steps and \\$3 to conclude it couldn't do the thing, or it looped politely until it hit a cap nobody's watching. [The $200 postmortem](https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto) was a *loud* failure I happened to catch because costs spiked. The expensive ones are the quiet failures you never labeled, because you can't alert on a category you don't record. And for subjective tasks, [your LLM judge might be hiding failures that look like success](https://loopandretry.github.io/posts/llm-as-judge-is-lying-to-you/?ref=devto) in its biased measurements.\n\n[What to measure when your agent works](https://loopandretry.github.io/posts/what-to-measure-when-your-agent-works/?ref=devto) covered the happy path. This is the inverse: what to measure when it *doesn't*, and how to know that it didn't.\n\nHere's the trap. Your agent has a `try/except`\n\nat the top of the loop. Exceptions get logged, counted, alerted. Your error rate looks like 0.5% and everyone's happy. Meanwhile:\n\n`{\"results\": []}`\n\nand the agent treated empty as \"done.\"None of those increment your exception counter. All of them are failures. Your real failure rate isn't 0.5%; it's 0.5% *that you can see* plus an unknown, larger number you can't. Step one is to make every run end in a *labeled* outcome, not just \"exception or not.\"\n\nEvery agent run should terminate with an explicit, recorded outcome. Not a boolean — a category. The minimum useful set:\n\n| Outcome | What happened | How you detect it |\n|---|---|---|\n`success` |\nTask done, verified | A post-hoc check passed (see below) |\n`hard_error` |\nException, crash, unrecoverable tool failure | The one you already catch |\n`budget_exhausted` |\nHit a step / token / time cap mid-task | The cap fired before a terminal state |\n`gave_up` |\nAgent declared it couldn't finish | Model emitted a \"cannot complete\" terminal action |\n`looped` |\nRepeated states without progress | Progress detector tripped (\n|\n\n`wrong`\n\nThe point of the taxonomy is that these have *different fixes*. `budget_exhausted`\n\nmeans your caps are too tight or your task is too big — raise the cap or decompose. `gave_up`\n\nmeans a capability or tool gap — the agent knew it was stuck, which is the *good* failure. `looped`\n\nmeans your loop lacks a progress check. `wrong`\n\nis the dangerous one, because it's indistinguishable from `success`\n\nat runtime. Collapsing all of these into \"error rate\" throws away exactly the information that tells you what to do.\n\n**Terminal-state logging.** The single highest-value change: make the loop's exit path assign an outcome. If you fall out of the loop because a cap fired, that's `budget_exhausted`\n\n— don't let it masquerade as success.\n\n``` python\ndef run_agent(task, step_cap=40, token_cap=200_000):\n    state = init(task)\n    for step in range(step_cap):\n        action = model_step(state)\n        if action.is_terminal:\n            outcome = \"gave_up\" if action.type == \"cannot_complete\" else \"success\"\n            return finish(state, outcome, step, tokens(state))\n        if tokens(state) > token_cap:\n            return finish(state, \"budget_exhausted\", step, tokens(state), cap=\"token\")\n        state = apply(action, state)\n    return finish(state, \"budget_exhausted\", step_cap, tokens(state), cap=\"step\")\n\ndef finish(state, outcome, steps, toks, cap=None):\n    log.info(\"agent_run_end\", outcome=outcome, steps=steps, tokens=toks, cap=cap)\n    return state.result, outcome\n```\n\nNow `outcome`\n\nis a dimension you can group by. \"What fraction of runs hit the step cap this week?\" becomes a query instead of a mystery.\n\n**A progress detector for looped.** A cheap one: hash the salient state (open goals, last tool called + args) each step and count repeats. Three visits to the same hash without a new goal closing means no progress — break with\n\n`looped`\n\n. This turns an invisible, expensive non-termination into a labeled, bounded event you can alert on.**Post-hoc verification for wrong.** This is the hard one, because\n\n`wrong`\n\nlooks identical to `success`\n\nwhile the run is happening. You cannot catch it at runtime; you catch it Once outcomes are labeled, two derived metrics tell you almost everything:\n\n**Silent-failure ratio** — `(budget_exhausted + gave_up + looped + wrong) / total`\n\n, i.e. failures that *didn't* throw, over all runs. This is the number your exception counter was hiding. Track it as your true failure rate. If it's an order of magnitude above your exception rate — and it usually is at first — that gap is your observability debt.\n\n**Cost of failure** — tokens (and dollars) spent on runs that ended in anything but `success`\n\n. A `wrong`\n\nrun that took forty steps cost you a full run's tokens *and* whatever the bad output does downstream. Attribute spend to outcome and you'll often find a large slice of your bill is being burned by a small slice of runs failing expensively — the same shape as the [$200 incident](https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/?ref=devto), just spread thin enough that no single night sets off an alarm.\n\nIf your agent monitoring only counts exceptions, you're measuring the failures that were kind enough to crash. The ones that cost you are silent: they exhaust a budget, give up, loop, or return a confident wrong answer with exit code zero. Make every run end in a *labeled outcome*, add a progress detector and post-hoc sampling, and track the silent-failure ratio as your real failure rate. You can't fix a failure mode you've never named — and the whole reason agents feel unreliable in production is that most teams are naming exactly one of them. And if you're running [multi-agent crews](https://loopandretry.github.io/posts/multi-agent-failure-modes/?ref=devto), instrument the failure modes that are unique to crew coordination: agent disagreement, circular delegation, cascading errors across the team.", "url": "https://wpnews.pro/news/your-agent-s-failures-are-silent-measuring-failure-modes-in-production", "canonical_source": "https://dev.to/loopandretry/your-agents-failures-are-silent-measuring-failure-modes-in-production-3lbg", "published_at": "2026-08-13 20:36:36+00:00", "updated_at": "2026-08-13 21:18:17.874808+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Loop & Retry"], "alternates": {"html": "https://wpnews.pro/news/your-agent-s-failures-are-silent-measuring-failure-modes-in-production", "markdown": "https://wpnews.pro/news/your-agent-s-failures-are-silent-measuring-failure-modes-in-production.md", "text": "https://wpnews.pro/news/your-agent-s-failures-are-silent-measuring-failure-modes-in-production.txt", "jsonld": "https://wpnews.pro/news/your-agent-s-failures-are-silent-measuring-failure-modes-in-production.jsonld"}}