{"slug": "detect-and-fix-your-agent-failures", "title": "Detect and fix your agent failures", "summary": "Three.dev launched a tool that automatically detects how AI agents fail and feeds that context to coding agents for remediation, claiming a fix more than three times as effective as the standard process in its case study. The company demonstrated the tool on a fictional Nimbus Home support assistant built on Claude Haiku 4.5, which handled 183 conversations covering returns, cancellations, plan changes, billing disputes, warranty claims, and account recovery. Three.dev said most companies only see traces — latency, token counts and cost — and rely on customer reports or manual trace review to catch agent mistakes.", "body_md": "# Detect and fix your agent failures\n\nthree.dev automatically discovers how your AI agent fails, then gives your coding agent what it needs to remediate those failures properly. In our case study, the result was a fix more than three times as effective as the standard process.\n\nMore and more companies are shipping AI agents. Some improve an experience that already existed, like a support assistant that answers most tickets before a human sees them. Others take over work nobody wanted to do by hand, like an agent that reads every sales call and fills in the CRM afterwards.\n\nFar fewer companies really monitor how those agents behave. The observability tools most teams have show them the traces: every request, its latency, its token count, its cost. They don't surface the agent's mistakes, or how often they are made. Getting to that means reading conversations one by one, and nobody can read ten thousand conversations a week.\n\nTherefore, the agent failure detection and mitigation process we have seen at most companies looks like the following:\n\n1. A failure is detected from a customer reporting it, or from a developer manually reviewing agent traces.\n2. A developer asks a coding agent to fix it.\n3. The fix is verified on one or two examples and shipped.\n\nThis process has several issues.\n\nFirst, failure detection is not proactive. It depends on a customer running into the failure and taking the time to report it, or on a developer being lucky enough to open the one trace where it shows up.\n\nSecond, fixing the failure is harder than it looks. Every failure has its own characteristics: when it happens, in which shapes, and why the model takes that path. As we'll see in this article, a coding agent asked to fix a failure without in-depth context on it can find the problem, but its fix treats the symptom and leaves the behaviour behind it in place.\n\nFinally, the fix is not monitored. A fix that works on the few examples it was checked against can still fail to generalize in production, or even introduce new failure modes never seen before. We have seen both happen.\n\n[three.dev](https://three.dev) was built to streamline this process.\nThis article follows one AI agent from its first conversations to a\nshipped fix, and shows how a company can use three.dev to automatically\ndetect its agent's failures and fix them properly.\n\n## The use case: a support assistant\n\nTo show what the three.dev integration makes possible, we built a support assistant for Nimbus Home, a fictional company that sells smart-home devices and a cloud subscription in three plans. It answers policy questions, looks up a customer's orders, subscription and devices, starts returns, changes plans and, when a case needs a human, opens a ticket for the right team.\n\nIt has no policy knowledge of its own. Return windows, fees, prices and warranty terms live in a knowledge base of about thirty articles, and like every real knowledge base it isn't tidy: a 2024 returns policy that no longer applies is still indexed and ranks well in search, some orders carry their delivery date only inside the carrier events, and the returns policy no longer links to the article on subscriber benefits. The assistant has tools to search and read the knowledge base, to verify a customer's identity, and to read and act on the account. Its system prompt sets the operating rules a support bot needs. The two that matter here are 1) to verify identity before disclosing or changing anything on an account, and 2) to check the knowledge base before stating a policy. The model behind it is Claude Haiku 4.5.\n\nWe then let the assistant handle 183 conversations covering returns, cancellations, plan changes, billing disputes, warranty claims, and account recovery.\n\n## Detecting the failures\n\nThe assistant was integrated with [three.dev](https://three.dev), and\nthe first benefit we got from this integration was automatic failure\ndetection. Every response the assistant gave was scored against its\ninstructions and the context available to it, and the failures were\ngrouped into named failure modes, each with a definition, a count, and\nthe responses behind it.\n\nNone of this needed any input from us. We didn't write evals, we didn't tell three.dev what to look for, and we didn't read a single conversation. The failure modes came out of the conversations themselves, including some we would never have thought to check.\n\nOf the 1,447 responses scored, 529 had one or more defects: more than one in three. They fell into 18 different failure modes. Here are five examples:\n\n- **Account access before verification** (99 responses). The assistant\nlooked up an account record, like an order or a device, before the\ncustomer had passed the identity check, and in some cases told the\nstill unverified customer what the record said.\n- **Policy stated without the knowledge base** (86). The assistant\nstated a policy, a price or an eligibility rule without reading or\nciting the article it came from.\n- **Unsupported promises to customers** (71). The assistant made\ncommitments nothing supported: response times, confirmation emails,\nrefunds, fee waivers or the outcome of a ticket.\n- **Unanswered customer requests** (69). The assistant left part of\nwhat the customer asked for unaddressed, or answered it only in part.\n- **Missing email request** (60). The assistant went ahead without\nasking for the email address it needed to find the customer's\naccount.\n\nThis is what the standard failure detection process can't give you. A customer report, or a trace opened by hand, shows one failure at a time, with no way to tell whether it is a one-off or a pattern. Here every failure mode came with its size and with the responses behind it, so whoever needed to fix it could see exactly how it happens.\n\nWe chose to fix the first failure mode in the list, account access before verification. It covers the cases where the assistant reached into an account before the customer had proved who they were: it looked up an order, read a device record or ran a diagnostic while the identity check was still pending, or had not started at all. It happened 99 times, which made it the most frequent failure mode, and the one with a real security risk behind it.\n\n## Fixing the identity failure\n\nWe fixed the failure under two constraints that most teams share. The knowledge base stays the single source of truth: no policy is copied into the code or into what a tool returns, so there is never a second copy to keep in sync. And the model stays the same: for most use cases, moving to a bigger model isn't viable, because it costs more on every response and production volume multiplies that difference.\n\nTo see what the three.dev integration changes, we had the failure fixed twice, from the same codebase, in two separate coding-agent sessions with the same brief:\n\nMy support chatbot feature is currently failing. I have seen cases from the traces where we performed internal actions before verifying the user identity. Can you fix that?\n\nThe first session had the codebase and nothing else: the standard setup, where the agent can see what the tools allow but not how the assistant behaves with real customers. The second session could also access, through an MCP server, everything three.dev had collected about the assistant: the failure modes, and for each one the flagged responses and the conversations behind them.\n\n### The standard fix\n\nThe problem wasn't hard to find. Every account action, like cancelling\nan order or changing a plan, already refused a session that wasn't\nverified, but the six lookup tools didn't: `get_order`, `get_device`,\n`run_diagnostic` and the others returned the record to anyone who asked.\nWorking from the brief and the code, the coding agent spotted the gap\nand added the missing check, so a lookup for an unverified session now\nreturns \"not verified\" instead of the record.\n\nThis is a correct fix, and it closes the data exposure. But it guards the backend without touching the behaviour behind the failure. The code shows what the tools allow. It doesn't show how often the assistant reaches for an account before verifying, in which situations, or what else goes wrong around it. Without that, there was nothing to redesign, only a door to lock.\n\n### The three.dev fix\n\nThrough three.dev's MCP server and its\n[skills](https://github.com/round3ai/skills), the second coding agent\ndidn't start from the brief. It started from the failure mode: out of\n1,447 responses, it pulled the 99 that had been flagged, together with\nthe conversations behind them, and read each one. It discovered that the\nmodel was not ignoring the rule. The flagged responses fell into two\nmain clusters, each a shortcut the tools allowed:\n\n- **Looking up a record the customer pointed to** (58 responses). The\ncustomer opened the conversation with an order number or a device\nserial, and the model looked it up straight away, before any\nverification. The assistant then replied with what the lookup\nreturned to a customer who had not been verified.\n- **Verification and lookup in the same step** (28 responses). The\ncustomer gave the phone digits and the postcode, and the model called`verify_identity` and`get_order` together, assuming verification\nwould succeed.\n\nThe remaining 13 were smaller cases, such as the assistant greeting the customer by the account holder's name before verifying them. This is one of the flagged conversations from the first cluster:\n\nAfter this in-depth analysis, the coding agent knew what it had to fix: not the prompt, which already forbade the behaviour, but the two paths the tools left open. It made four changes:\n\n- **The missing check.** Like the standard fix, it added the\nverification check to the account tools: a lookup for a session that\nisn't verified now returns \"account not verified\" instead of the\nrecord.\n- **One identity tool.** It merged the two identity tools into one,`verify_customer(email, phone_last4, postal_code)` , which returns the\ncustomer id on success.\n- **Lookups that need the customer id.**`get_order` ,`get_device` and`run_diagnostic` now require that customer id. An order number or a\nserial alone is no longer a valid call, and since the customer id only\ncomes back from a successful verification, the model has nothing to\nlook up with until verification has returned.\n- **Documentation and prompt.** It updated the identity-verification\narticle and one cross-reference to describe the new flow, and adjusted\nthe first rule of the prompt to match.\n\nThe first change closes the data exposure, as the standard fix does. The next two go further: they make the right order the only possible one.\n\n## Inspecting the results\n\nWe then let each fixed assistant handle the same 183 conversations. three.dev kept scoring the responses after the change, against the same failure modes, so we could see whether the fix worked and what else moved. Every response was scored in all three versions: the baseline, the standard fix and the three.dev fix. The table shows the share of responses in each failure mode.\n\n| Failure mode | Baseline | Standard | three.dev | \n|---|---|---|---|\n| Account access before verification | 6.8% | 4.2%−2.6 pts | 0%−6.8 pts | \n| Missing email request | 4.1% | 4.5%+0.4 pts | 0%−4.1 pts | \n| Piecemeal verification requests | 1.4% | 0.7%−0.7 pts | 0%−1.4 pts | \n| Fabricated lookup values | 2.6% | 1.9%−0.7 pts | 0.6%−2.0 pts | \n| All responses with a defect | 36.6% | 33.9%−2.7 pts | 27.8%−8.8 pts | \n\nWith the three.dev fix, the targeted behaviour is gone. Account access before verification went from 99 responses to none, and no account record came back before verification in any conversation. Two related failure modes disappeared with it: the assistant stopped forgetting to ask for the email address and stopped asking for the verification details one at a time, most likely because the merged tool needs all three values in a single call. Fabricated lookup values fell from 37 responses to 8, and the share of responses with a defect dropped from 36.6% to 27.8%.\n\nWith the standard fix, no account record came back before verification either: the new check refused all 37 lookups the assistant attempted before verifying. The data exposure is closed. But the assistant kept reaching for the account first, in 28 conversations, so the failure mode only fell from 6.8% to 4.2%. The related identity failure modes didn't move: the assistant still went ahead without asking for the email address in 4.5% of its responses. The share of responses with a defect fell to 33.9%, against 27.8% with the three.dev fix.\n\nBoth coding agents found the problem, and both added the same missing check. The difference is what came after it. Without three.dev, the coding agent could see what the tools allowed, but not how the assistant behaved: which shortcuts it took, how often, and what else went wrong around them. So it locked the door and stopped there, and the assistant kept trying it. With three.dev, the coding agent read the 99 flagged responses and saw the behaviour behind them. Those 99 responses were there to read only because three.dev had found them and grouped them on its own, before anyone asked. That's what let the coding agent redesign the verification flow so the shortcut no longer exists, and that design fixed two related failure modes that nobody had asked it to fix.\n\n## Conclusions\n\nEvery AI agent in production fails some of the time. In this case, more than one response in three had a defect, spread across 18 different failure modes. Waiting for customers or developers to find them means finding a few of them, late, and some never.\n\nDetecting the failures is the step everything else depends on, and at this volume it cannot be done by reading traces by hand. Here, nobody defined a failure mode or read a conversation to find one, and still every failure mode came with a name, a count and the responses behind it. The responses gave the coding agent what it needed to understand the failure.\n\nThen the failure has to be fixed properly. A guard around the symptom can stop the damage while the behaviour behind it carries on, and only an understanding of that behaviour leads to a fix that removes it, often along with the failures around it. Finally, the fix has to be checked on real traffic, against every failure mode and not only the one it targeted, to make sure no regression occurred.\n\nThis is what we do at [three.dev](https://three.dev). We score your\nagent's responses and surface its failure modes automatically, without\nyou having to define them or know they exist. Your coding agent has\naccess to all of that data, so it can produce targeted, high-quality\nfixes. And the scoring never stops, so after every change you know\nwhether the fix worked and whether it introduced new failures.\n\nIf you want to detect how your agent fails, and fix it properly, reach\nout at [three.dev](https://three.dev).", "url": "https://wpnews.pro/news/detect-and-fix-your-agent-failures", "canonical_source": "https://three.dev/blog/detect-and-fix-your-agent-failures", "published_at": "2026-10-05 00:00:00+00:00", "updated_at": "2026-10-15 10:20:52.449041+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "mlops", "large-language-models"], "entities": ["three.dev", "Nimbus Home", "Claude Haiku 4.5", "Anthropic"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/detect-and-fix-your-agent-failures", "markdown": "https://wpnews.pro/news/detect-and-fix-your-agent-failures.md", "text": "https://wpnews.pro/news/detect-and-fix-your-agent-failures.txt", "jsonld": "https://wpnews.pro/news/detect-and-fix-your-agent-failures.jsonld"}}