Your agent's failures are silent: measuring failure modes in production

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The failure that hurts is the one that doesn't throw. A traditional service fails loudly: an exception, a 500, a stack trace, a red line on a dashboard. An agent fails quietly. It runs to completion, returns a confident answer, exits zero — and the answer is wrong, or it spent forty steps and $3 to conclude it couldn't do the thing, or it looped politely until it hit a cap nobody's watching. The $200 postmortem was a loud failure I happened to catch because costs spiked. The expensive ones are the quiet failures you never labeled, because you can't alert on a category you don't record. And for subjective tasks, your LLM judge might be hiding failures that look like success in its biased measurements.

What to measure when your agent works covered the happy path. This is the inverse: what to measure when it doesn't, and how to know that it didn't.

Here's the trap. Your agent has a try/except

at the top of the loop. Exceptions get logged, counted, alerted. Your error rate looks like 0.5% and everyone's happy. Meanwhile:

{"results": []}

and the agent treated empty as "done."None of those increment your exception counter. All of them are failures. Your real failure rate isn't 0.5%; it's 0.5% that you can see plus an unknown, larger number you can't. Step one is to make every run end in a labeled outcome, not just "exception or not."

Every agent run should terminate with an explicit, recorded outcome. Not a boolean — a category. The minimum useful set:

wrong

The point of the taxonomy is that these have different fixes. budget_exhausted

means your caps are too tight or your task is too big — raise the cap or decompose. gave_up

means a capability or tool gap — the agent knew it was stuck, which is the good failure. looped

means your loop lacks a progress check. wrong

is the dangerous one, because it's indistinguishable from success

at runtime. Collapsing all of these into "error rate" throws away exactly the information that tells you what to do.

Terminal-state logging. The single highest-value change: make the loop's exit path assign an outcome. If you fall out of the loop because a cap fired, that's budget_exhausted

— don't let it masquerade as success.

def run_agent(task, step_cap=40, token_cap=200_000):
    state = init(task)
    for step in range(step_cap):
        action = model_step(state)
        if action.is_terminal:
            outcome = "gave_up" if action.type == "cannot_complete" else "success"
            return finish(state, outcome, step, tokens(state))
        if tokens(state) > token_cap:
            return finish(state, "budget_exhausted", step, tokens(state), cap="token")
        state = apply(action, state)
    return finish(state, "budget_exhausted", step_cap, tokens(state), cap="step")

def finish(state, outcome, steps, toks, cap=None):
    log.info("agent_run_end", outcome=outcome, steps=steps, tokens=toks, cap=cap)
    return state.result, outcome

Now outcome

is a dimension you can group by. "What fraction of runs hit the step cap this week?" becomes a query instead of a mystery.

A progress detector for looped. A cheap one: hash the salient state (open goals, last tool called + args) each step and count repeats. Three visits to the same hash without a new goal closing means no progress — break with

looped

. This turns an invisible, expensive non-termination into a labeled, bounded event you can alert on.Post-hoc verification for wrong. This is the hard one, because

wrong

looks identical to success

while the run is happening. You cannot catch it at runtime; you catch it Once outcomes are labeled, two derived metrics tell you almost everything:

Silent-failure ratio — (budget_exhausted + gave_up + looped + wrong) / total

, i.e. failures that didn't throw, over all runs. This is the number your exception counter was hiding. Track it as your true failure rate. If it's an order of magnitude above your exception rate — and it usually is at first — that gap is your observability debt.

Cost of failure — tokens (and dollars) spent on runs that ended in anything but success

. A wrong

run that took forty steps cost you a full run's tokens and whatever the bad output does downstream. Attribute spend to outcome and you'll often find a large slice of your bill is being burned by a small slice of runs failing expensively — the same shape as the $200 incident, just spread thin enough that no single night sets off an alarm.

If your agent monitoring only counts exceptions, you're measuring the failures that were kind enough to crash. The ones that cost you are silent: they exhaust a budget, give up, loop, or return a confident wrong answer with exit code zero. Make every run end in a labeled outcome, add a progress detector and post-hoc sampling, and track the silent-failure ratio as your real failure rate. You can't fix a failure mode you've never named — and the whole reason agents feel unreliable in production is that most teams are naming exactly one of them. And if you're running multi-agent crews, instrument the failure modes that are unique to crew coordination: agent disagreement, circular delegation, cascading errors across the team.