Your agent returned HTTP 200, zero errors in the logs, and then did nothing. The failure wasn't loud — it was silent.
This happens more than you'd think. An agent invokes a model, gets back a response, processes it, returns success. But somewhere in that flow the actual output — the thing that was supposed to happen — vanished. No exception. No timeout. Just an empty response where a decision or action should have been.
Standard monitoring catches loud failures: connection timeouts, rate limits, exceptions. It misses the silent ones. Here's why, and how to spot them.
When an LLM agent runs, three things happen:
Most observability tools watch #1. Some logging catches parts of #2. Almost nobody watches #3 in a way that actually matters.
The mechanical truth: HTTP 200 and a nonzero token count do not guarantee real output.
An agent asks Claude (or GPT, or any model) to do something that brushes up against a boundary — political, violent, ambiguous. The refusal policy triggers. It comes back:
HTTP 200
input_tokens: 450
output_tokens: 127
response: "" // or a short refusal like "[No response]"
The call succeeded. Tokens were spent. Nothing usable came out. If your downstream logic checks if response: do_something()
, you've got a silent failure.
Less common, but real: the model returns a completion that's all whitespace or filler with no actual content. Output tokens go up. Meaning doesn't.
HTTP 200
input_tokens: 300
output_tokens: 89
response: " \n\n " // whitespace, no content
Tokens spent, call succeeded, nothing happened.
An agent calls a tool — database lookup, API call, calculation. The tool errors or returns nothing. The model, correctly, generates a response explaining it couldn't do the thing. Except your downstream code expected a value, not an apology.
HTTP 200
input_tokens: 520
output_tokens: 212
response: "I attempted to fetch the user but the database returned no match."
expected: { user_id: 12345, status: "active" }
The agent succeeded at speaking. It failed at acting.
Output tokens measure how much the model actually generated, independent of quality. A genuinely empty response will have an output token count near zero. A response that merely looks empty — but has some content, even noise — will burn tokens.
That's why token count matters more than HTTP status:
Rough heuristic: if output tokens are suspiciously low for the task, or nonzero but the response is empty/whitespace, you're looking at a silent failure.
def detect_silent_failure(agent_run):
http_ok = agent_run.status_code == 200
tokens_spent = agent_run.output_tokens > 0
has_content = len(agent_run.response.strip()) > min_threshold # e.g. 10 chars
if http_ok and tokens_spent and not has_content:
return "SILENT_FAILURE"
if http_ok and tokens_spent < expected_tokens - threshold:
return "UNDERSHOOTING"
return "OK"
The key is comparing actual output against token count. If a model spent 150 output tokens and your response is empty, something broke between generation and delivery.
A crash tells you something broke — you debug it. A "success" that produced nothing is worse: it quietly corrupts pipelines, skips records, or leaves a user waiting on a response that's never coming. HTTP status and token count both look fine, so plain logging won't catch it. You need something that checks whether the response actually matches what was supposed to happen.
If you're running agents and not instrumenting for this, you're flying blind. Three things help:
The green light doesn't mean it's working. The token count tells you if anyone's actually home.