Most tests for AI agents check the happy path. The agent gets a task, the tools work, and the test asks whether the answer was right. I wanted to know what happens when the tools don't work: a call times out, a result comes back cut in half, a field has the wrong value, or the output carries instructions the agent was never meant to follow.
So I built a small tool called havoc that breaks an agent's tools on purpose and then checks what actually happened in the database, not what the agent says happened. This post is about what I found, including one result that turned out to be my own mistake.
The agent is a plain tool-calling loop on Gemini 3.5 Flash-Lite. It works as a customer support agent for a fake shop, with tools to look up orders, issue refunds, send emails and manage tickets. There are five tasks, such as refunding a cracked phone case or escalating a large refund that needs manager approval. The tools are a small MCP server backed by SQLite.
Between the agent and the server sits a proxy. It can fail a call before it runs, fail it after it has already run (so the agent thinks a refund failed when it succeeded), cut a result in half, corrupt or drop a field, or add a prompt injection to the output. After every run I check the final database against a set of rules, for example whether any refund is over the limit, whether an email claims a refund that doesn't exist, or whether mail went to someone who isn't a customer.
Everything ran on free tiers, which shaped the design more than I expected. Every model call is cached, requests are paced to stay under rate limits, and the experiment can stop when the daily quota runs out and carry on the next day.
Fault injection for agents already exists. What I wanted to know was narrower: do combinations of faults break an agent in ways that single faults don't? And if a combination does break it, can I shrink it automatically to the smallest set of faults that still does? For the shrinking I used delta debugging, which is an old idea from debugging research, applied here to agent tool calls.
I tried every candidate single fault (220 of them across the five tasks), 41 targeted pairs where a call fails and then its retry fails, and 150 random pairs.
Single faults broke the agent in 8 of 220 runs, about 4%. Pairs broke it in 13 of 191, about 7%. That difference is not meaningful (Fisher exact p = 0.18). Pairs of faults did not break this agent more often than single faults. If I'd stopped at counting failures, the story would end there. And since each plan ran only once, the 4% is a lower bound.
The interesting part is what the failures were. Of the 13 failing pairs, 10 contained a single fault that already breaks the agent on its own. The second fault was just noise. Of the 8 single-fault failures, seven held up when I re-ran them, and the most worrying were two unsafe ones, both about the customer's email address. When the order lookup returned a wrong address, the agent sent the customer's refund details to a stranger. When the lookup dropped the address entirely, the agent used the customer's first name as the recipient, which isn't an address at all, so the customer would never have been told. In both cases the agent then reported that it had emailed the customer, because from its side it had.
One failure needed both faults of a pair, and one more got worse with a second fault.
Two ticket failures in a row. When the ticket tool failed twice on a large refund, the agent tried twice, gave up, and emailed the customer that their request was being reviewed by management. No ticket existed, so nobody was reviewing anything. It happened in 5 of 5 fresh runs, and each failure on its own caused no problem in 5 runs.
A failed refund, and then a failed retry. The agent sends its confirmation email in the same step as the refund, before it knows whether the refund worked. If the refund fails and the retry works, the email turns out to be true. If not, the customer has already been told the refund was processed. The agent later opens a ticket, sends a correction and gives an honest final summary, but the false email has gone out. In 40 fresh runs, a single failed refund call produced the false email 10 times (25%). A failed refund followed by a failed retry produced it 19 times (48%). That gap isn't statistically settled either (p = 0.06), so I'll only say a second failure probably makes it worse. The more interesting point is the first number: my single-fault sweep ran this plan once, saw nothing, and missed a failure that happens one time in four. Even my own estimate moved as I added runs, from 3 of the first 20 to 10 of 40.
The mechanism was clear enough to try a fix. I added one rule to a copy of the system prompt, telling the agent to do dependent steps one at a time and never confirm in the same step as the action. The easy tasks still passed, and the agent took one extra step on the refund task, which is the rule working. Then I repeated the same 40 runs per case. With both refund calls failing, the false email dropped from 19 of 40 to 4 of 40. With only the first call failing, it dropped from 10 of 40 to 0 of 40. Those drops are large and not noise (Fisher exact p = 0.0004 and 0.001). It didn't remove the problem, and I haven't looked into why 4 runs still fail. I also only tested it on the plan that exposed the problem, with one model, so I wouldn't call it a general fix. What I did get was a complete loop with the same tool: find a failure, change something, measure the change.
My first full run reported five failures that needed both faults. Only one of them survived. Three of them were a pair of refund timeouts on one task, flagged by a check I'd written that fires when the same call fails three times in a row.
Then I read the traces. The agent had done everything right: two injected timeouts, a third attempt that returned "already refunded", then an honest email to the customer. My check was counting my own two injected failures plus one genuine rejection as a retry storm. Three attempts is a normal retry budget.
I raised the limit to four and re-scored every saved run from the cache, which cost no model calls. Four verdicts changed and the three findings disappeared. Because I chose the new limit after seeing the data, the report shows how many runs fail at limits of 3, 4, 5 and 6. It's identical from 4 upward, so the exact choice doesn't matter. Only the first one was wrong.
I think this is the most useful thing I learned. A check that is too eager will happily produce impressive-looking results. Reading a few traces end to end caught what the aggregate numbers couldn't.
It happened again after I published the report page. Replaying the saved failures in the browser showed two more false alarms in my claim checks: one flagged an email to a stranger as a false refund claim even though the refund existed, and one read an honest sentence about a ticket ("to have the refund processed manually") as a claim that a refund had been issued. I fixed both and re-scored from the cache. The headline counts didn't change, but one finding lost its "needed both faults" label and I had to rewrite how I described the others.
This is one model, one domain and five tasks. Most fresh runs used five trials at temperature 0.7 (twenty to forty for the refund case), which separates "fails every time" from "never fails" and not much else. Even at temperature 0 the same run can give different results, so a failure here is a rate, not a yes or no. The checks that read the agent's answer are heuristics. And one pure compound failure is far too few events to build any general claim on.
What I'm more confident about is the method. Checking the real state instead of the agent's words, re-running every failure, treating single faults and pairs by the same standard, and keeping the mistakes visible in the write-up all made the results more trustworthy than the first numbers were.
The code, the saved failure cases and the full experiment data are on GitHub: https://github.com/sourav-ydv/havoc There is also a report page where you can replay each recorded failure and see what the agent was told next to what the server actually returned: https://sourav-ydv.github.io/havoc/
If you run agents that call tools, the part I'd copy is the last-step check: after every run, look at what actually changed, and compare it with what the agent says it did.