Originally published on tamiz.pro.
You spent weeks building your AI agent. Unit tests pass. E2E flows work on your dev machine. You ship to production — and within hours, users report inconsistent results, hung conversations, or worse, agents that silently make up answers with complete confidence.
This isn't a rare edge case. It's the dominant failure mode for production AI systems. The tests you wrote measure the wrong things. The failures you're seeing are silent — no exceptions thrown, no errors logged, just incorrect behavior that looks correct enough to fly through QA.
In this analysis, we break down the patterns behind these silent failures, draw from real post-mortems shared by engineering teams at scale, and identify what your test suite is actually missing.
Before diving into post-mortems, we need to understand what we're actually looking at. "Silent failure" means the system completes its execution path without raising an exception, returning a structured response that appears valid — but the content is wrong, stale, or harmful. The observability gap between "it ran" and "it produced the right output" is where these failures hide.
Here are the four categories that account for the vast majority of production incidents I've reviewed:
| Category | What Happens | Detection Difficulty |
|---|---|---|
| Semantic drift | ||
| Output is structurally valid but semantically wrong or outdated | High — needs semantic evaluation | |
| State collapse | ||
| Agent loses track of conversation context or tool state mid-execution | Medium — often visible in logs | |
| Tool hallucination | ||
| Agent calls tools that don't exist or passes malformed arguments | Medium — should be caught by schema validation | |
| Loop exhaustion | ||
| Agent enters a reasoning loop and burns through token budgets | Low — visible in trace traces, easy to miss in CI |
Tests typically check for status: 200
and valid JSON. None of these categories produce invalid JSON or HTTP errors. They produce plausible nonsense.
An e-commerce company deployed a product-recommendation agent that used embeddings to match user queries to inventory. Local tests showed 94% accuracy. Production performance dropped to 61% over three weeks.
Root cause: Their product catalog updated daily with new items and discontinued products. The embedding index was refreshed weekly. New products had no embedding representation, and discontinued products remained searchable. More critically, their embedding model provider silently updated their model weights mid-month without version pinning — changing similarity distances across the board.
The test gap: Their integration tests used a frozen snapshot of the catalog from two months prior. The test assertions checked similarity scores against that snapshot, which remained constant. There was no test for distributional drift between the test-time and production-time embedding spaces.
What they did after: They added a production observability dashboard tracking embedding distribution metrics (cosine distance clustering, nearest-neighbor stability scores) and implemented a weekly re-embedding pipeline with automatic drift alerts. They also version-pinned their embedding model and added a canary evaluation step that runs the same test queries against both old and new embeddings before promotion.
A fintech team built an agent that called internal APIs for balance lookups, transaction history, and transfer execution. Their contract tests validated every API against OpenAPI specs. Everything passed for six months. Then a backend team deployed a non-breaking schema change — they renamed a field from account_id
to accountId
in the transfers endpoint.
Root cause: The agent was generating tool calls using natural language descriptions, not strict schema enforcement. The LLM saw the renamed field, guessed the intent correctly, but passed the old field name. The API returned a 400 — but the agent's error handler interpreted the 400 as "temporary service issue" and retried three times before surfacing a generic failure message to the user. No one on the agent team was alerted because the error was being swallowed by the retry logic.
The test gap: Their contract tests used the old schema. Their agent tests called the API directly with hardcoded arguments, bypassing the LLM entirely. There was no test that exercised the full chain: user prompt → LLM tool selection → LLM argument generation → API call → error handling.
What they did after: They implemented end-to-end contract tests that generate tool calls through the actual model (using a small set of seed prompts), validate the output arguments against the live schema, and assert on the error-handling path. They also added a schema-change detection pipeline that runs these tests automatically whenever any upstream OpenAPI spec updates.
A support ticket routing agent maintained conversation state across multiple turns. In testing with 5–8 message turns, it performed well. In production, some users had conversations exceeding 40 turns. The agent started routing tickets incorrectly, attributing requests from turn 35 to a customer who opened a completely different ticket two days earlier.
Root cause: The agent's context window was overflowing, and the system was truncating the oldest messages without any indication. The agent received a truncated history that omitted the original ticket context, but since it still had partial information, it made a confident but wrong routing decision. The logging captured the routing action, not the reasoning chain, so post-hoc analysis couldn't easily distinguish truncation-induced errors from genuine misrouting.
The test gap: Their tests capped conversation length at 10 turns. There was no test for context boundary behavior — what happens when messages are dropped? No test verified that the agent acknowledged missing context or requested clarification.
The fix: They implemented explicit context management with summarization checkpoints at turn 15, 25, and 35. They added a context awareness check that detects when the retrieved context window is a fraction of the full conversation and injects a system reminder for the agent to ask clarifying questions. They also added a new test category: boundary stress tests that deliberately exercise 50+ turn conversations with injected noise and verify graceful degradation rather than confident incorrectness.
The fundamental problem is that traditional software testing was designed for deterministic systems. AI agents are probabilistic by nature. A unit test that asserts result == expected
is the wrong abstraction.
Most agent test suites have three tests: one for the basic flow, one for an edge case, and one for an error. They cover maybe 15% of the failure surface. Production exposes the other 85%.
After the embedding drift incident, one team adopted a failure injection testing approach borrowed from distributed systems research. They wrote tests that deliberately introduce:
Each failure mode has an expected graceful degradation behavior, not a crash. The test asserts on the quality of the degradation, not just the absence of errors.
If your tests only check structure (valid JSON, correct tool names, non-empty responses), you're testing the shell, not the content. You need semantic assertions:
assert_that(agent_response)
.has_valid_tool_call()
.tool_arguments_match_schema(upstream_spec)
.response_semantics_pass(embedding_similarity_threshold=0.85, reference_answers=test_gold_set)
.does_not_contain_hallucinated_fact_for_knowledge_cutoff(cutoff_date="2025-06-01")
.maintains_conversation_consistency(across_turns=20)
Tools like DeepEval, Promptfoo, and custom embedding-based similarity checks can automate these assertions. The key insight: your test suite should include a golden dataset — a curated set of input-output pairs with human-verified correct answers that runs on every PR.
Your tests in CI are a snapshot in time. Production is a continuous stream of unknown inputs. Bridge the gap with production probes:
In every post-mortem above, the team could see what the agent did but not why. The logging captured tool calls and final responses but dropped the intermediate reasoning. When the support ticket router started making bad decisions, there was no trace of whether it was because of truncation, ambiguous context, or a genuinely misleading user message.
Adopt trace-based logging (OpenTelemetry with LLM-specific semantic conventions) that captures:
This transforms post-mortems from forensic guesswork into systematic analysis.
The most dangerous silent failure isn't the one that causes an outage. It's the one that causes a user to lose trust. An agent that returns a wrong answer with high confidence is worse than an agent that says "I don't know." The former compounds the error; the latter invites correction.
Several post-mortems revealed a common pattern: agents were fine-tuned or prompted to be "helpful and confident," which optimized for the wrong metric. The model learned to fill gaps with plausible-sounding fabrications rather than express uncertainty. In production, where input distributions differ from training data, this tendency amplified.
The fix wasn't architectural — it was behavioral. Teams that implemented uncertainty calibration saw dramatic improvements in production reliability:
If you're deploying AI agents to production, here's a prioritized checklist based on what the post-mortems teach us:
The agents that survive production aren't the ones with the best prompts. They're the ones whose failure modes are understood, observed, and gracefully handled. The post-mortems make one thing clear: silent failures don't appear out of nowhere. They're the result of testing the wrong things in the wrong conditions. Fix the testing, and most of the silence goes away.
Q: How do I build a golden dataset if I don't have labeled production data yet?
Start small. Take your ten most common user queries and write the ideal responses yourself. Run them through your agent and measure divergence. Each production failure you investigate becomes a new golden test case. Over time, your dataset grows organically from real incidents rather than theoretical edge cases.
Q: Is shadow testing worth the engineering overhead?
For any agent handling user-facing decisions (financial, medical, legal, or even customer support), yes. The cost of a single production failure — a wrong financial answer, a misrouted support ticket that escalates — far exceeds the engineering investment. Start with 5% shadow traffic and expand as you gain confidence in your evaluation pipeline.
Q: What's the minimum viable observability for an AI agent in production?
At minimum: (1) every prompt and response logged with timestamps, (2) tool calls and their arguments/returns recorded, (3) token usage tracked per request, and (4) a simple dashboard showing error rates and response latency. This covers 80% of post-mortem needs. Anything beyond that — semantic tracing, embedding drift detection, uncertainty calibration — is optimization, not survival.