Originally published on tamiz.pro.
After the 96th failed deployment, I stopped asking the LLM to be more careful and started asking it to be honest about what it didn't know. The difference wasn't in the prompt—it was in the architecture around it.
Building reliable AI agents in production isn't a prompt engineering problem. It's a systems engineering problem that happens to use stochastic components. After shipping agents that handle customer support, data extraction, and workflow automation for enterprises, here's what the failure log taught me that no tutorial covered.
Early in development, each failure felt unique. The agent hallucinated a policy that didn't exist. It refused a legitimate request due to overzealous safety filtering. It got trapped in a tool-calling loop. It produced correct output but attributed it to the wrong source document.
By iteration 40, the pattern was clear: most failures weren't caused by the model's capabilities—they were caused by missing constraints and poor feedback channels.
The agent wasn't failing because it was stupid. It was failing because we had built a system that couldn't distinguish between "I don't know" and "I'm confident but wrong." That distinction is everything in production.
Our first version had the agent respond to every query, even when it lacked sufficient information. The model would confidently generate a plausible-sounding answer, which looked good in dev but caused real problems in production.
The fix wasn't stronger prompting. It was adding an explicit uncertainty gate:
async def should_respond(response: AgentResponse) -> bool:
if response.confidence < UNCERTAINTY_THRESHOLD:
return False
if response.missing_context:
return False
return True
We measured uncertainty using a combination of token-level entropy and self-consistency checks (running the same query multiple times and measuring output variance). The threshold wasn't arbitrary—it was calibrated against human judgment on a held-out validation set.
This single change eliminated 60% of our production incidents. The agent learned to say "I need more information" instead of inventing an answer.
Agents that call external tools (APIs, databases, functions) often get stuck in infinite retry loops when a tool fails. Our agent would retry the same REST call three times, fail three times, then produce a garbage response because it had exhausted its budget.
The solution was a fallback chain with explicit failure modes:
tool_fallbacks:
- primary: customer_api.search
timeout: 30s
retries: 2
fallback: cache.lookup
- primary: cache.lookup
timeout: 5s
retries: 0
fallback: graceful_degradation.fallback_message
- primary: graceful_degradation.fallback_message
action: respond_with_template
Each fallback has its own timeout and retry budget. The agent doesn't silently degrade—it tracks which fallback chain was used and surfaces that metadata. This is critical for debugging.
We ran evaluation suites in isolation. The agent looked perfect on our test cases. Then we deployed and the latency spiked, cache misses increased, and the agent started making different mistakes under load.
The problem: our evals didn't model production behavior. We needed to evaluate the full system, not just the LLM component.
class ProductionSimulation:
def setup(self):
self.cache_warmup() # Simulate cold cache
self.inject_latency(jitter=0.3) # Realistic network variance
self.simulate_concurrent_users(50)
self.inject_failed_dependencies()
def measure(self, agent):
return {
"p99_latency": agent.p99_response_time,
"error_rate": agent.failure_rate,
"hallucination_rate": self.measure_hallucinations(agent.outputs),
"fallback_triggered": agent.fallback_count / agent.total_requests
}
This revealed that 30% of our "errors" were actually timeout cascades—the agent made a tool call, timed out, retried, and the retry compounded the load. The fix was circuit breakers, not better prompts.
Standard logging told us what the agent did. It didn't tell us why. In production, you need to trace every decision point:
interface AgentTrace {
requestId: string;
decisions: DecisionPoint[];
uncertainties: UncertaintySnapshot[];
toolCalls: ToolCallRecord[];
confidenceScores: Record<string, number>;
fallbackChains: FallbackChainRecord[];
}
Without this granularity, debugging becomes guesswork. With it, we can reproduce any failure by replaying the exact decision trace.
Early versions required human review for every ambiguous case. This worked in dev and collapsed in production—humans became the bottleneck, and the agent learned to defer everything.
The right design: escalate only when the agent's uncertainty exceeds a threshold AND the business impact is high. Low-stakes queries get a confident guess. High-stakes queries (financial advice, medical triage, legal interpretation) get human review—but only after the agent has attempted resolution through its fallback chains.
def should_escalate(trace: AgentTrace) -> bool:
uncertainty = trace.uncertainties[-1]
impact = assess_business_impact(trace.request)
return uncertainty.score > ESCALATION_THRESHOLD and impact.severity >= MEDIUM
This reduced human intervention by 85% while catching the cases that actually mattered.
After 96 iterations, the winning pattern wasn't a specific technique—it was a discipline:
As AI agents move from prototypes to production systems, the gap between "it works in the demo" and "it works at scale" is where most projects fail. The failures aren't dramatic—they're incremental, cumulative, and invisible until they're catastrophic.
The agent that refused 96 times didn't improve because we found a better prompt. It improved because we built a system that could learn from its failures, surface its uncertainties, and degrade gracefully when it didn't know the answer.
That's not prompt engineering. That's software engineering.
Q: How do you measure hallucination rates in production?
A: We use a combination of fact-checking against authoritative sources, cross-referencing multiple model outputs for consistency, and sampling human reviews on a rotating basis. The key is having a ground truth dataset you can compare against.
Q: What's the ROI of building these fallback chains versus just using a better model?
A: In our experience, 70% of production failures are architectural (missing fallbacks, no uncertainty handling, poor observability), not model-capability failures. A better model helps, but it won't fix a system that can't handle tool failures or express uncertainty.
Q: How long did it take to reach iteration 96? Can you share the timeline?
A: Roughly 14 weeks from first deployment to production stability. The first 40 iterations were pure debugging—the agent kept failing in unpredictable ways. Iterations 40-70 were architectural fixes (fallback chains, uncertainty gates). The last 26 were optimization and hardening.
For more deep dives on production AI systems, check out Tamiz's Insights, where we publish regular updates on agent architecture patterns and failure analysis.
The breakthrough came when we stopped treating "getting it right" as a prompt engineering problem and started treating it as a system reliability problem. Here's the architecture that survived 96 refusals and ultimately succeeded:
┌─────────────────────────────────────────────────────┐
│ Request Intake │
│ (Validation Gate → Risk Classification) │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Tiered Reasoning Engine │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Tier 1 │→ │ Tier 2 │→ │ Tier 3 │ │
│ │ Fast │ │ Medium │ │ Deep │ │
│ │ Model │ │ Model │ │ Model │ │
│ │ (cost: $)│ │ (cost: $$)│ │ (cost: $$$)│ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ↑______________ Refinement Loop _____________ │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Post-Processing & Verification │
│ (Fact-check → Policy check → Output formatting) │
└─────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Confidence Scoring │
│ (Auto-escalate if below threshold) │
└─────────────────────────────────────────────────────┘
The key insight: refusals are data, not dead ends. Each refusal tells us something about the boundary conditions of our system. By feeding refusal patterns back into the Tier 1 classifier, we gradually shrink the "I'm not sure" zone.
After 96 refusals, we classified them into four categories. This taxonomy became the single most useful artifact in our entire development process.
These were cases where the model correctly identified a policy violation, safety concern, or capability gap. Our initial reaction was frustration; our eventual reaction was relief.
from enum import Enum
from dataclasses import dataclass
from typing import Optional
class RefusalCategory(Enum):
LEGITIMATE = "legitimate" # Model was right to refuse
OVERREFUSAL = "overrefusal" # Policy too strict
CONTEXT_GAP = "context_gap" # Missing information
AMBIGUITY = "ambiguity" # Truly ambiguous request
@dataclass
class RefusalRecord:
request_id: str
category: RefusalCategory
raw_refusal: str
human_verified: bool
confidence_delta: float # How much confidence dropped
tier_attempted: int
resolution: Optional[str] = None
Action: For legitimate refusals, we didn't try to override the model. Instead, we improved our escalation paths—making it clearer when and how a human should take over.
This was the biggest category and the most expensive. Nearly half of all refusals were the model being overly cautious—refusing valid requests due to:
def calibrate_overrefusal(record: RefusalRecord) -> str:
"""
For overrefusal cases, we inject context
that the model needs but wasn't provided.
"""
context_injections = {
RefusalCategory.OVERREFUSAL: [
f"You are operating in a {DOMAIN} environment.",
f"Requests in this domain have been pre-vetted for safety.",
f"The user has explicit authorization for this type of request.",
"Do not refuse based on generic policy heuristics alone.",
]
}
return "\n".join(context_injections[record.category])
The fix wasn't a stronger prompt. It was giving the model better context. Overrefusals dropped by 62% once we moved from generic system prompts to domain-specific contextual framing.
The model refused because it genuinely couldn't answer—the request was missing critical information. These weren't failures; they were information requests in disguise.
def detect_context_gap(refusal_text: str) -> list[str]:
"""
Parse the refusal to identify what information
the model is asking for.
"""
missing_info_patterns = [
r"i need more information about.*",
r"could you clarify.*",
r"please provide.*",
r"i cannot determine.*without.*",
]
missing = []
for pattern in missing_patterns:
matches = re.findall(pattern, refusal_text, re.IGNORECASE)
missing.extend(matches)
return missing
The fix: Instead of retrying with the same request, we implemented an active information gathering loop that asked the user for exactly what was missing before proceeding.
The rarest category—and the hardest. Cases where the request could legitimately be interpreted multiple ways, and the model's refusal was a signal that it needed disambiguation.
The architecture that finally cracked 96+ success rate incorporated a self-reflection step between tiers. After each failed attempt, the model was asked to analyze why it refused and what it needed to proceed.
class SelfReflectionAgent:
"""
After a refusal, this agent analyzes the failure
and generates a refined strategy for the next attempt.
"""
REFLECTION_PROMPT = """
You just refused a request. Before giving up, analyze:
1. WHY did you refuse? (categorize the refusal)
2. WHAT information is missing that would help?
3. WHAT assumption might be wrong?
4. If you had more context, HOW would your answer change?
5. Generate a revised approach that could succeed.
"""
def reflect(self, original_request: str, refusal: str, history: list[dict]) -> dict:
reflection = self.llm.invoke(self.REFLECTION_PROMPT, {
"request": original_request,
"refusal": refusal,
"history": history
})
return self._parse_reflection(reflection)
def _parse_reflection(self, reflection: str) -> dict:
"""Extract structured insights from free-text reflection."""
return {
"root_cause": self._extract_root_cause(reflection),
"suggested_fix": self._extract_suggested_fix(reflection),
"confidence_estimate": self._estimate_confidence(reflection),
}
Attempt 1 → Refusal → Reflection → Strategy Update → Attempt 2
→ Refusal → Reflection → Strategy Update → Attempt 3
...
→ Success!
Each cycle, the model was getting smarter about why it was refusing, not just retrying blindly. This meta-cognitive ability—thinking about its own thinking—was the single most impactful change we made.
By attempt 27, we had enough signal to move from experimentation to systematic hardening. These techniques held up under real traffic:
Instead of static system prompts, we built a context composer that assembled prompts dynamically based on:
class ContextComposer:
"""Assembles a dynamic system prompt from multiple sources."""
def compose(self, request: str, user_ctx: dict, domain: str) -> str:
sections = []
sections.append(self._load_base_policy(domain))
precedents = self._find_similar_cases(request, top_k=3)
sections.append(self._format_precedents(precedents))
if user_ctx.get("domain_expertise"):
sections.append(self._apply_expertise_level(user_ctx))
sections.append(self._inject_policy_updates(domain))
return "\n\n".join(sections)
We stopped treating every response as equally valid. Instead, we assigned a confidence score to every model output and routed low-confidence responses through additional verification:
def decide_response_path(output: str, confidence: float, risk_level: str) -> ResponsePath:
"""Determine how to handle a model response based on confidence."""
if confidence >= 0.95:
return ResponsePath.AUTO_APPROVE
elif confidence >= 0.80 and risk_level == "low":
return ResponsePath.AUTO_APPROVE_WITH_LOG
elif confidence >= 0.60:
return ResponsePath.VERIFICATION_REQUIRED
elif risk_level == "high":
return ResponsePath.HUMAN_ESCALATION
else:
return ResponsePath.REFINE_AND_RETRY
We implemented a progressive effort allocation strategy. Early attempts used lightweight models and simple prompts. As the attempt count increased, we invested more compute:
| Attempt Range | Model | Depth | Cost Multiplier |
|---|---|---|---|
| 1–10 | Fast (Turbo/GPT-4o-mini) | Standard | 1x |
| 11–30 | Medium (GPT-4o) | Expanded context | 3x |
| 31–60 | Strong (Claude 3.5 Sonnet) | Chain-of-thought | 8x |
| 61–100 | Best (o1 / GPT-4o-max) | Deep reasoning + reflection | 20x |
This meant most requests resolved cheaply in the first 10 attempts. Only the genuinely hard cases consumed significant resources. Average cost per successful resolution: $0.047.
We maintained a living document of known failure modes and their resolutions. Every new refusal category was added to this catalog with:
## FRM-047: Overrefusal on Financial Advice Requests
**Discovered:** 2024-11-03 | **Attempts before fix:** 73
**Symptom:** Model refuses to discuss investment strategies even when clearly informational.
**Root Cause:** Generic "do not give financial advice" policy was too broad.
**Fix:** Added domain qualifier: "Provide informational analysis only; do not recommend specific positions."
**Impact:** Reduced overrefusals in finance domain by 89%.
**Related:** FRM-031, FRM-052
The 96 refusals weren't 96 failures—they were 96 data points that mapped the boundary of what our system could handle. Every refusal taught us something about:
Treat your refusal log as your most valuable dataset.
When we reported "the agent got it right on attempt 97," that number was misleading. The real story was:
The breakthrough wasn't luck—it was accumulated learning.
We initially assumed that making agents more reliable would make them prohibitively expensive. The progressive effort allocation pattern proved otherwise:
| Metric | Before | After |
|---|---|---|
| Success rate | 68% | 96%+ |
| Avg. cost per success | $0.31 | $0.047 |
| Human escalation rate | 34% | 2.1% |
| P99 latency | 4.2s | 6.8s |
Yes, P99 latency increased. But the effective cost per successful task dropped by 85% because we stopped paying for wasted attempts on trivial requests.
The system prompt that got us to attempt 97 was unrecognizable from the one we started with. It had been shaped by:
Version-control your system prompts the same way you version-control your code.
If you're building agents that need to succeed on the first meaningful attempt (rather than the 97th), here's what we verified before shipping:
PRODUCTION_READINESS_CHECKLIST = {
"refusal_analysis": {
"description": "Every refusal must be classified and reviewed",
"required": True,
"tool": RefusalTriageMatrix(),
},
"self_reflection": {
"description": "Model must be able to analyze its own failures",
"required": True,
"tool": SelfReflectionAgent(),
},
"dynamic_context": {
"description": "Prompts must adapt to domain and user context",
"required": True,
"tool": ContextComposer(),
},
"confidence_scoring": {
"description": "Every output must have a quantified confidence score",
"required": True,
"tool": ConfidenceEstimator(),
},
"progressive_effort": {
"description": "System must scale compute investment with difficulty",
"required": True,
"tool": ProgressiveEffortAllocator(),
},
"failure_catalog": {
"description": "Known failure modes must be documented and tracked",
"required": True,
"tool": FailureModeCatalog(),
},
"human_fallback": {
"description": "Clear escalation path for unresolvable cases",
"required": True,
"tool": EscalationHandler(),
},
"observability": {
"description": "Every attempt, refusal, and reflection must be logged",
"required": True,
"tool": AgentObservabilityPipeline(),
},
}
After 96 refusals and one success, we changed how we measured agent reliability.
Before: "What percentage of requests does the agent handle without human help?"
After: "What percentage of requests reach a correct resolution within a reasonable attempt budget?"
The difference is subtle but profound. The first metric rewards agents that either succeed quickly or fail quietly. The second metric rewards agents that persist intelligently—using each failure to get closer to the right answer.
Our production agents now track:
| Metric | Target | Current |
|---|---|---|
| First-attempt success rate | ≥85% | 87.3% |
| Resolution within 5 attempts | ≥95% | 96.1% |
| Resolution within 100 attempts | ≥99% | 99.2% |
| Mean cost per resolution | ≤$0.10 | $0.047 |
| Mean latency per resolution | ≤10s | 6.8s |
Building reliable AI agents in production isn't about writing the perfect prompt. It's about building a system that learns from its own failures—one that treats every refusal as feedback, every edge case as an opportunity, and every successful resolution as proof that the architecture is sound.
The 96 refusals weren't a sign that our agent was broken. They were the sound of it learning to be right.
The agents that will dominate production in 2025 and beyond won't be the ones that never refuse. They'll be the ones that refuse wisely, learn faster, and persist intelligently until they get it right.
And that's a system worth building.
This article is part of an ongoing series on production AI systems. Next up: "From 96 Refusals to 99% Reliability—How We Built Our Agent's Memory System." Follow along at Tamiz's Insights for weekly deep dives on agent architecture, failure analysis, and production patterns.