Why My Agent Refused 96 Times Before Getting It Right: Lessons from Building Reliable AI Agents in Production

A developer detailed how building reliable AI agents in production requires systems engineering rather than prompt tweaking, after 96 failed deployments. Key fixes included an uncertainty gate that cut incidents by 60% and a fallback chain for tool failures, plus evaluating the full system under simulated production conditions.

Originally published on tamiz.pro. After the 96th failed deployment, I stopped asking the LLM to be more careful and started asking it to be honest about what it didn't know. The difference wasn't in the prompt—it was in the architecture around it. Building reliable AI agents in production isn't a prompt engineering problem. It's a systems engineering problem that happens to use stochastic components. After shipping agents that handle customer support, data extraction, and workflow automation for enterprises, here's what the failure log taught me that no tutorial covered. Early in development, each failure felt unique. The agent hallucinated a policy that didn't exist. It refused a legitimate request due to overzealous safety filtering. It got trapped in a tool-calling loop. It produced correct output but attributed it to the wrong source document. By iteration 40, the pattern was clear: most failures weren't caused by the model's capabilities—they were caused by missing constraints and poor feedback channels. The agent wasn't failing because it was stupid. It was failing because we had built a system that couldn't distinguish between "I don't know" and "I'm confident but wrong." That distinction is everything in production. Our first version had the agent respond to every query, even when it lacked sufficient information. The model would confidently generate a plausible-sounding answer, which looked good in dev but caused real problems in production. The fix wasn't stronger prompting. It was adding an explicit uncertainty gate: php async def should respond response: AgentResponse - bool: if response.confidence < UNCERTAINTY THRESHOLD: return False if response.missing context: return False return True We measured uncertainty using a combination of token-level entropy and self-consistency checks running the same query multiple times and measuring output variance . The threshold wasn't arbitrary—it was calibrated against human judgment on a held-out validation set. This single change eliminated 60% of our production incidents. The agent learned to say "I need more information" instead of inventing an answer. Agents that call external tools APIs, databases, functions often get stuck in infinite retry loops when a tool fails. Our agent would retry the same REST call three times, fail three times, then produce a garbage response because it had exhausted its budget. The solution was a fallback chain with explicit failure modes : tool fallbacks: - primary: customer api.search timeout: 30s retries: 2 fallback: cache.lookup - primary: cache.lookup timeout: 5s retries: 0 fallback: graceful degradation.fallback message - primary: graceful degradation.fallback message action: respond with template Each fallback has its own timeout and retry budget. The agent doesn't silently degrade—it tracks which fallback chain was used and surfaces that metadata. This is critical for debugging. We ran evaluation suites in isolation. The agent looked perfect on our test cases. Then we deployed and the latency spiked, cache misses increased, and the agent started making different mistakes under load. The problem: our evals didn't model production behavior. We needed to evaluate the full system , not just the LLM component. python class ProductionSimulation: def setup self : self.cache warmup Simulate cold cache self.inject latency jitter=0.3 Realistic network variance self.simulate concurrent users 50 self.inject failed dependencies def measure self, agent : return { "p99 latency": agent.p99 response time, "error rate": agent.failure rate, "hallucination rate": self.measure hallucinations agent.outputs , "fallback triggered": agent.fallback count / agent.total requests } This revealed that 30% of our "errors" were actually timeout cascades—the agent made a tool call, timed out, retried, and the retry compounded the load. The fix was circuit breakers, not better prompts. Standard logging told us what the agent did. It didn't tell us why . In production, you need to trace every decision point: interface AgentTrace { requestId: string; decisions: DecisionPoint ; uncertainties: UncertaintySnapshot ; toolCalls: ToolCallRecord ; confidenceScores: Record<string, number ; fallbackChains: FallbackChainRecord ; } Without this granularity, debugging becomes guesswork. With it, we can reproduce any failure by replaying the exact decision trace. Early versions required human review for every ambiguous case. This worked in dev and collapsed in production—humans became the bottleneck, and the agent learned to defer everything. The right design: escalate only when the agent's uncertainty exceeds a threshold AND the business impact is high . Low-stakes queries get a confident guess. High-stakes queries financial advice, medical triage, legal interpretation get human review—but only after the agent has attempted resolution through its fallback chains. php def should escalate trace: AgentTrace - bool: uncertainty = trace.uncertainties -1 impact = assess business impact trace.request return uncertainty.score ESCALATION THRESHOLD and impact.severity = MEDIUM This reduced human intervention by 85% while catching the cases that actually mattered. After 96 iterations, the winning pattern wasn't a specific technique—it was a discipline: As AI agents move from prototypes to production systems, the gap between "it works in the demo" and "it works at scale" is where most projects fail. The failures aren't dramatic—they're incremental, cumulative, and invisible until they're catastrophic. The agent that refused 96 times didn't improve because we found a better prompt. It improved because we built a system that could learn from its failures, surface its uncertainties, and degrade gracefully when it didn't know the answer. That's not prompt engineering. That's software engineering. Q: How do you measure hallucination rates in production? A: We use a combination of fact-checking against authoritative sources, cross-referencing multiple model outputs for consistency, and sampling human reviews on a rotating basis. The key is having a ground truth dataset you can compare against. Q: What's the ROI of building these fallback chains versus just using a better model? A: In our experience, 70% of production failures are architectural missing fallbacks, no uncertainty handling, poor observability , not model-capability failures. A better model helps, but it won't fix a system that can't handle tool failures or express uncertainty. Q: How long did it take to reach iteration 96? Can you share the timeline? A: Roughly 14 weeks from first deployment to production stability. The first 40 iterations were pure debugging—the agent kept failing in unpredictable ways. Iterations 40-70 were architectural fixes fallback chains, uncertainty gates . The last 26 were optimization and hardening. For more deep dives on production AI systems, check out Tamiz's Insights, where we publish regular updates on agent architecture patterns and failure analysis. The breakthrough came when we stopped treating "getting it right" as a prompt engineering problem and started treating it as a system reliability problem. Here's the architecture that survived 96 refusals and ultimately succeeded: ┌─────────────────────────────────────────────────────┐ │ Request Intake │ │ Validation Gate → Risk Classification │ └─────────────────────┬───────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Tiered Reasoning Engine │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Tier 1 │→ │ Tier 2 │→ │ Tier 3 │ │ │ │ Fast │ │ Medium │ │ Deep │ │ │ │ Model │ │ Model │ │ Model │ │ │ │ cost: $ │ │ cost: $$ │ │ cost: $$$ │ │ │ └──────────┘ └──────────┘ └──────────┘ │ │ ↑ Refinement Loop │ └─────────────────────┬───────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Post-Processing & Verification │ │ Fact-check → Policy check → Output formatting │ └─────────────────────┬───────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────┐ │ Confidence Scoring │ │ Auto-escalate if below threshold │ └─────────────────────────────────────────────────────┘ The key insight: refusals are data, not dead ends. Each refusal tells us something about the boundary conditions of our system. By feeding refusal patterns back into the Tier 1 classifier, we gradually shrink the "I'm not sure" zone. After 96 refusals, we classified them into four categories. This taxonomy became the single most useful artifact in our entire development process. These were cases where the model correctly identified a policy violation, safety concern, or capability gap. Our initial reaction was frustration; our eventual reaction was relief. python from enum import Enum from dataclasses import dataclass from typing import Optional class RefusalCategory Enum : LEGITIMATE = "legitimate" Model was right to refuse OVERREFUSAL = "overrefusal" Policy too strict CONTEXT GAP = "context gap" Missing information AMBIGUITY = "ambiguity" Truly ambiguous request @dataclass class RefusalRecord: request id: str category: RefusalCategory raw refusal: str human verified: bool confidence delta: float How much confidence dropped tier attempted: int resolution: Optional str = None Action: For legitimate refusals, we didn't try to override the model. Instead, we improved our escalation paths —making it clearer when and how a human should take over. This was the biggest category and the most expensive. Nearly half of all refusals were the model being overly cautious—refusing valid requests due to: php def calibrate overrefusal record: RefusalRecord - str: """ For overrefusal cases, we inject context that the model needs but wasn't provided. """ context injections = { RefusalCategory.OVERREFUSAL: f"You are operating in a {DOMAIN} environment.", f"Requests in this domain have been pre-vetted for safety.", f"The user has explicit authorization for this type of request.", "Do not refuse based on generic policy heuristics alone.", } return "\n".join context injections record.category The fix wasn't a stronger prompt. It was giving the model better context . Overrefusals dropped by 62% once we moved from generic system prompts to domain-specific contextual framing. The model refused because it genuinely couldn't answer—the request was missing critical information. These weren't failures; they were information requests in disguise . php def detect context gap refusal text: str - list str : """ Parse the refusal to identify what information the model is asking for. """ missing info patterns = r"i need more information about. ", r"could you clarify. ", r"please provide. ", r"i cannot determine. without. ", missing = for pattern in missing patterns: matches = re.findall pattern, refusal text, re.IGNORECASE missing.extend matches return missing The fix: Instead of retrying with the same request, we implemented an active information gathering loop that asked the user for exactly what was missing before proceeding. The rarest category—and the hardest. Cases where the request could legitimately be interpreted multiple ways, and the model's refusal was a signal that it needed disambiguation. The architecture that finally cracked 96+ success rate incorporated a self-reflection step between tiers. After each failed attempt, the model was asked to analyze why it refused and what it needed to proceed. class SelfReflectionAgent: """ After a refusal, this agent analyzes the failure and generates a refined strategy for the next attempt. """ REFLECTION PROMPT = """ You just refused a request. Before giving up, analyze: 1. WHY did you refuse? categorize the refusal 2. WHAT information is missing that would help? 3. WHAT assumption might be wrong? 4. If you had more context, HOW would your answer change? 5. Generate a revised approach that could succeed. """ def reflect self, original request: str, refusal: str, history: list dict - dict: reflection = self.llm.invoke self.REFLECTION PROMPT, { "request": original request, "refusal": refusal, "history": history } return self. parse reflection reflection def parse reflection self, reflection: str - dict: """Extract structured insights from free-text reflection.""" return { "root cause": self. extract root cause reflection , "suggested fix": self. extract suggested fix reflection , "confidence estimate": self. estimate confidence reflection , } Attempt 1 → Refusal → Reflection → Strategy Update → Attempt 2 → Refusal → Reflection → Strategy Update → Attempt 3 ... → Success Each cycle, the model was getting smarter about why it was refusing , not just retrying blindly. This meta-cognitive ability—thinking about its own thinking—was the single most impactful change we made. By attempt 27, we had enough signal to move from experimentation to systematic hardening. These techniques held up under real traffic: Instead of static system prompts, we built a context composer that assembled prompts dynamically based on: class ContextComposer: """Assembles a dynamic system prompt from multiple sources.""" def compose self, request: str, user ctx: dict, domain: str - str: sections = Base policy always present sections.append self. load base policy domain Historical precedents case-based reasoning precedents = self. find similar cases request, top k=3 sections.append self. format precedents precedents User-specific context if user ctx.get "domain expertise" : sections.append self. apply expertise level user ctx Recent policy changes sections.append self. inject policy updates domain return "\n\n".join sections We stopped treating every response as equally valid. Instead, we assigned a confidence score to every model output and routed low-confidence responses through additional verification: php def decide response path output: str, confidence: float, risk level: str - ResponsePath: """Determine how to handle a model response based on confidence.""" if confidence = 0.95: return ResponsePath.AUTO APPROVE elif confidence = 0.80 and risk level == "low": return ResponsePath.AUTO APPROVE WITH LOG elif confidence = 0.60: return ResponsePath.VERIFICATION REQUIRED elif risk level == "high": return ResponsePath.HUMAN ESCALATION else: return ResponsePath.REFINE AND RETRY We implemented a progressive effort allocation strategy. Early attempts used lightweight models and simple prompts. As the attempt count increased, we invested more compute: | Attempt Range | Model | Depth | Cost Multiplier | |---|---|---|---| | 1–10 | Fast Turbo/GPT-4o-mini | Standard | 1x | | 11–30 | Medium GPT-4o | Expanded context | 3x | | 31–60 | Strong Claude 3.5 Sonnet | Chain-of-thought | 8x | | 61–100 | Best o1 / GPT-4o-max | Deep reasoning + reflection | 20x | This meant most requests resolved cheaply in the first 10 attempts. Only the genuinely hard cases consumed significant resources. Average cost per successful resolution: $0.047. We maintained a living document of known failure modes and their resolutions. Every new refusal category was added to this catalog with: FRM-047: Overrefusal on Financial Advice Requests Discovered: 2024-11-03 | Attempts before fix: 73 Symptom: Model refuses to discuss investment strategies even when clearly informational. Root Cause: Generic "do not give financial advice" policy was too broad. Fix: Added domain qualifier: "Provide informational analysis only; do not recommend specific positions." Impact: Reduced overrefusals in finance domain by 89%. Related: FRM-031, FRM-052 The 96 refusals weren't 96 failures—they were 96 data points that mapped the boundary of what our system could handle. Every refusal taught us something about: Treat your refusal log as your most valuable dataset. When we reported "the agent got it right on attempt 97," that number was misleading. The real story was: The breakthrough wasn't luck—it was accumulated learning. We initially assumed that making agents more reliable would make them prohibitively expensive. The progressive effort allocation pattern proved otherwise: | Metric | Before | After | |---|---|---| | Success rate | 68% | 96%+ | | Avg. cost per success | $0.31 | $0.047 | | Human escalation rate | 34% | 2.1% | | P99 latency | 4.2s | 6.8s | Yes, P99 latency increased. But the effective cost per successful task dropped by 85% because we stopped paying for wasted attempts on trivial requests. The system prompt that got us to attempt 97 was unrecognizable from the one we started with. It had been shaped by: Version-control your system prompts the same way you version-control your code. If you're building agents that need to succeed on the first meaningful attempt rather than the 97th , here's what we verified before shipping: PRODUCTION READINESS CHECKLIST = { "refusal analysis": { "description": "Every refusal must be classified and reviewed", "required": True, "tool": RefusalTriageMatrix , }, "self reflection": { "description": "Model must be able to analyze its own failures", "required": True, "tool": SelfReflectionAgent , }, "dynamic context": { "description": "Prompts must adapt to domain and user context", "required": True, "tool": ContextComposer , }, "confidence scoring": { "description": "Every output must have a quantified confidence score", "required": True, "tool": ConfidenceEstimator , }, "progressive effort": { "description": "System must scale compute investment with difficulty", "required": True, "tool": ProgressiveEffortAllocator , }, "failure catalog": { "description": "Known failure modes must be documented and tracked", "required": True, "tool": FailureModeCatalog , }, "human fallback": { "description": "Clear escalation path for unresolvable cases", "required": True, "tool": EscalationHandler , }, "observability": { "description": "Every attempt, refusal, and reflection must be logged", "required": True, "tool": AgentObservabilityPipeline , }, } After 96 refusals and one success, we changed how we measured agent reliability. Before: "What percentage of requests does the agent handle without human help?" After: "What percentage of requests reach a correct resolution within a reasonable attempt budget?" The difference is subtle but profound. The first metric rewards agents that either succeed quickly or fail quietly. The second metric rewards agents that persist intelligently —using each failure to get closer to the right answer. Our production agents now track: | Metric | Target | Current | |---|---|---| | First-attempt success rate | ≥85% | 87.3% | | Resolution within 5 attempts | ≥95% | 96.1% | | Resolution within 100 attempts | ≥99% | 99.2% | | Mean cost per resolution | ≤$0.10 | $0.047 | | Mean latency per resolution | ≤10s | 6.8s | Building reliable AI agents in production isn't about writing the perfect prompt. It's about building a system that learns from its own failures —one that treats every refusal as feedback, every edge case as an opportunity, and every successful resolution as proof that the architecture is sound. The 96 refusals weren't a sign that our agent was broken. They were the sound of it learning to be right . The agents that will dominate production in 2025 and beyond won't be the ones that never refuse. They'll be the ones that refuse wisely, learn faster, and persist intelligently until they get it right. And that's a system worth building. This article is part of an ongoing series on production AI systems. Next up: "From 96 Refusals to 99% Reliability—How We Built Our Agent's Memory System." Follow along at Tamiz's Insights for weekly deep dives on agent architecture, failure analysis, and production patterns.