"My Agent Refused 96 Times": Building Self-Editing Agents with Hard Failure Modes

A senior ML engineer's benchmark of a production support agent revealed that the system refused to answer a valid question 96 times in a test set, but engineering review showed the model correctly identified insufficient or contradictory context in those cases, choosing refusal over hallucination. The article advocates for designing agents with hard failure modes, treating 'I don't know' as a first-class return type, and implementing verification gates to prevent confident hallucinations in critical systems.

Originally published on tamiz.pro. In the early days of shipping LLM-based agents, we optimized for output volume. If the model could not find the answer, it often generated a plausible one anyway. This is the "yes-man" problem. In critical systems—financial auditing, code generation, or compliance checks—this creates a dangerous class of errors: confident hallucinations . Recently, a senior ML engineer shared a benchmark result from a production support agent: the system refused to answer a valid question 96 times in a test set. While the product team initially flagged this as a failure rate, the engineering review revealed the opposite. In 96% of those cases, the model correctly identified that the retrieved context was insufficient or contradictory. It chose the hard failure mode: refusal. This is a pivotal shift in how we design autonomous systems. We are moving from probabilistic output to deterministic verification. This article explores how to architect agents with hard failure modes, treating "I don't know" as a first-class return type rather than an exception to be suppressed. Large Language Models are trained on corpora where the goal is often to be helpful and coherent. Consequently, they exhibit a strong bias toward generating a response, even when the semantic signal is absent. This is known as sycophancy in evaluation contexts. When you build a RAG Retrieval-Augmented Generation agent, the default pipeline is: The flaw lies in step 3. Without a gatekeeper, if the retrieved chunks contain irrelevant data due to embedding similarity thresholds being too loose , the model will attempt to bridge the gap with internal parametric knowledge, leading to hallucination. Consider the following naive implementation, which represents the "volume-first" architecture: php DANGEROUS: No verification gate async def get answer question: str - str: context = await retrieve context question prompt = f"Context:\n{context}\n\nQuestion: {question}" return await llm.complete prompt In a high-stakes environment, this function might return "The SQL query should use JOIN on table B" even if table B was never mentioned in the context. The agent has failed, but the API client sees a successful 200 OK with text. The error is silent. A hard failure mode is a deterministic exit path where the agent explicitly signals that it cannot fulfill the request based on verifiable criteria, rather than probabilistic guessing. This concept borrows heavily from systems programming e.g., Rust’s Result<T, E or Go’s error handling . The agent must validate its own output before returning it to the user. If validation fails, the agent does not return a "best effort" string; it returns a structured Failure object. The term "self-editing" implies a secondary LLM call or a deterministic check that reviews the primary output. It is a form of self-correction or refusal . There are two main types of self-editing gates: To build an agent that "refuses 96 times," we need a multi-step pipeline where the output of the generator is the input to a verifier. This is often called a critic-loop or verification-based generation . Before the LLM even generates text, define the strict shape of the expected answer. If your agent extracts data, it must output JSON. Use a library like Pydantic or Zod to enforce this. If the LLM outputs malformed JSON, the parser fails. This is not a hallucination; this is a syntax error. The agent should immediately fall back to a refusal state or retry with a stricter prompt, rather than attempting to parse a broken string. This is the core mechanism. We must introduce a verification step that checks if the generated answer is entailed by the context. We can implement this using a Logit Bias approach or a Second-Pass LLM . The second-pass approach is more robust for complex reasoning. Instead of trusting the first completion, we ask the LLM to evaluate its own work. The prompt structure changes from: Q: Question A: Answer To: Context: Context Q: Question A: Draft Answer Verify if A is fully supported by Context. Output TRUE or FALSE. If the verifier outputs FALSE , the agent enters the failure mode. It does not return the draft. It returns a standard refusal message. Here is a Python implementation of a self-editing agent using a verification loop: python from pydantic import BaseModel import openai import json class VerificationResult BaseModel : is supported: bool reasoning: str class AgentResponse BaseModel : answer: str | None status: str "success" or "refused" reason: str | None def verify answer context: str, question: str, draft: str - VerificationResult: prompt = f""" You are a rigorous fact-checker. Context: {context} Question: {question} Draft Answer: {draft} Task: Determine if the Draft Answer is strictly supported by the Context. - If the answer is in the context, output TRUE. - If the answer is a hallucination or not mentioned, output FALSE. - Do not use outside knowledge. Return JSON: {{"is supported": true/false, "reasoning": "..."}} """ response = openai.chat.completions.create model="gpt-4o-mini", Cost-effective verifier messages= {"role": "user", "content": prompt} , response format={"type": "json object"} return VerificationResult json.loads response.choices 0 .message.content async def self editing agent question: str, context: str - AgentResponse: 1. Generate draft draft prompt = f"Context:\n{context}\n\nAnswer: {question}" draft resp = openai.chat.completions.create model="gpt-4o", messages= {"role": "user", "content": draft prompt} draft answer = draft resp.choices 0 .message.content 2. Verify verdict = verify answer context, question, draft answer 3. Decide if verdict.is supported: return AgentResponse answer=draft answer, status="success", reason=None else: HARD FAILURE MODE: Refuse return AgentResponse answer=None, status="refused", reason=f"Insufficient context to answer confidently: {verdict.reason}" When the agent refuses, the response must be explicit. In the example above, the status is refused . The frontend or downstream service should treat this differently from a success response. If you have 96 refusals in a test suite, analyze them. Are they true positives the question was indeed unanswerable ? If so, your system is working correctly. If they are false positives the answer was in the context but the verifier missed it , you need to tune the verifier prompt or lower the temperature of the generator. A hard failure mode does not always mean immediate termination. In many systems, a "refusal" triggers a fallback strategy. This is where self-editing becomes powerful. If the verifier rejects the answer, the agent can attempt to re-generate with modified constraints. For example: This adds latency but drastically improves precision. For every 100 requests, you might accept 85, retry and accept 10, and refuse 5. The 96 refusals mentioned in the industry anecdotes often come from systems that skip the retry and go straight to refusal, which is actually safer. Traditional accuracy metrics are misleading here. You want to optimize for Precision when the agent speaks, it is right over Recall the agent answers everything . The "96 refusals" story is compelling because it highlights that the cost of a false positive in an AI agent is often orders of magnitude higher than the cost of a false negative refusal . In a medical diagnosis bot, a false positive hallucinating a treatment is catastrophic. A false negative refusing to answer is merely inconvenient. Beyond the basic verifier loop, there are advanced patterns used in production-grade agents like those at companies building autonomous coding assistants . If your agent uses tools APIs, databases , the result of the tool call is the ground truth. You can verify that the LLM's summary of the tool result matches the actual tool output. For example, if the agent calls get balance user id and receives 0.00 , the LLM must not write "The user has $100 in their account." A simple string comparison or checksum can verify this. This is a deterministic hard failure mode. Encourage the model to output its reasoning steps CoT before the final answer. The verifier can check the reasoning chain for logical gaps. If the reasoning is flawed but the answer is correct lucky guess , the verifier should still reject it. This enforces explainable accuracy. During the self-editing loop, you can dynamically adjust the temperature. If the first generation is rejected, retry with temperature=0 to minimize variance and force the model to stick closer to the context distribution. If your verifier is just another LLM call, it can also hallucinate. It might falsely reject a correct answer. To mitigate this: gpt-4o-mini or claude-3-haiku .Self-editing adds at least one extra LLM round-trip. This doubles latency and cost. The story of the agent refusing 96 times is a triumph of engineering integrity over product vanity. In the short term, a high refusal rate looks bad on a dashboard. In the long term, it builds trust. Users learn that when the agent speaks, they can trust it implicitly. Building self-editing agents with hard failure modes requires a shift in mindset: from generating content to verifying correctness . By implementing structured verification loops, enforcing schema strictness, and treating refusal as a valid state, we can build AI systems that are not just intelligent, but reliable. As we push toward more autonomous agents, the ability to say "no" is the defining characteristic of robustness. The code samples and patterns provided here serve as a foundation for transitioning your agent from a probabilistic chatbot to a deterministic worker. Q: How do I measure if my refusal rate is too high? A: Establish a "Golden Set" of questions with known answerable/unanswerable labels. Calculate your Refusal Accuracy : True Refusals / Total Refusals . If this is 95%, your refusal rate is healthy, regardless of the absolute number. If it is <80%, your verifier is too aggressive. Q: Can I use this pattern with non-LLM components? A: Yes. Hard failure modes apply to any stochastic system. If you have a heuristic classifier that sometimes confuses two classes, you can add a confidence threshold. If confidence < T, refuse. This is the same logic, just deterministic instead of LLM-based. Q: What is the best model for the verifier? A: The verifier does not need to be creative; it needs to be precise. Smaller models like gpt-4o-mini , claude-3-haiku , or even quantized local models like llama-3.2-3b often perform surprisingly well at verification tasks because they follow instructions closely without adding flair. Benchmark a few candidates on your specific data.