{"slug": "ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production", "title": "AI Agent Testing: Why a 77% Pass Rate Can Mean 53% in Production", "summary": "A developer's invoice-triage agent that passed 22 test cases three days in a row later filed the same PDF under two different vendors, illustrating what IBM Research calls the \"consistency gap.\" In a new arXiv paper, Evelyn Duesterwald and colleagues measured a ReAct agent on the AppWorld benchmark with GPT-4.1 and found a 77% per-run pass rate but only a 53% all-five-runs pass rate, a 24-point gap. Their Consistency Analyzer and Guideline Generator, which stores targeted step-level instructions as episodic memory, raised the all-five pass rate by 16 points on the same tasks and 13 points on unseen ones.", "body_md": "Short version for the impatient: if your agent passes 77% of your test cases, the chance it passes the same case five times in a row might be closer to 53%. That's the number I want you to carry around. If you want to know where it comes from and what I changed in my own testing because of it, read on.\n\nI ran into this the embarrassing way. Last spring I shipped an invoice-triage agent for a client. Twenty-two test cases, all green, three days in a row. I wrote \"stable\" in the handover doc. Two weeks later their ops person sent me a screenshot: the same PDF, uploaded twice ten minutes apart, filed under two different vendors. Nothing had changed. No prompt edit, no model update, no new data. The agent just took a different path the second time.\n\nI didn't have a name for that failure until a paper from IBM Research landed on arXiv this week. They call it the consistency gap, and they measured it properly, which I never had.\n\nThe paper is [Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course](https://arxiv.org/abs/2609.08832) by Evelyn Duesterwald and colleagues. The setup is simple enough that I'm annoyed I didn't do it myself. Take a ReAct agent on the [AppWorld benchmark](https://appworld.dev/) using GPT-4.1. Run every task five times. Count two things: the average pass rate per run, and the fraction of tasks that pass all five runs.\n\nPer-run pass rate: 77%. All-five pass rate: 53%. A 24 point gap, on a benchmark, with no environmental noise to blame.\n\nRead that again with your own dashboard in mind. My twenty-two green tests were one sample each. If the underlying agent had a 77% per-run rate, the odds of all twenty-two passing on a given day are small, so in hindsight I probably got lucky on the day I wrote \"stable\". Or, more likely, my cases were easier than the client's real invoices and the per-run rate was higher, which hides the gap without closing it.\n\nThe gap matters more than the pass rate for one reason. A human can live with a system that fails a known 23% of inputs, because you can route those inputs elsewhere. Nobody can live with a system that handles the same input differently on Tuesday than on Monday, because there's nothing to route on.\n\nWhere does the flip come from? Temperature is the obvious suspect and the wrong one. Set it to zero and you still get variation from the provider side (batching, hardware, model updates behind a stable name), and more importantly the agent loop amplifies tiny differences. One slightly different tool call on step three means a different observation on step four, and by step eight you're on a different trajectory entirely.\n\nThe paper's contribution isn't the diagnosis, it's what they do with it. They built a Consistency Analyzer that looks at the five trajectories for a task and finds the specific step where they diverge. Then a Guideline Generator writes a short targeted instruction about that step (\"when the search returns multiple contacts, filter on the email domain before picking one\" is the kind of thing) and stores it as episodic memory. The next time the agent sees a similar task, that guideline gets injected.\n\nResult on AppWorld: all-five pass rate up 16 points on the same tasks, and up 13 points on similar-but-unseen tasks. They don't claim to close the gap. They claim to narrow it, and the generalisation number is the one I find more convincing, since anyone can overfit guidelines to a fixed task list.\n\nI'd push back on one thing. The paper treats \"succeeds in all five runs\" as the target. For a lot of business automation, five is too few. If the agent runs 400 times a day and a 2% flip rate means eight wrong vendor assignments, five-run consistency won't catch that. Pick your N from your volume, not from the paper.\n\nI've changed three things in how I test agents for clients, and none of them required the paper's framework. They just required admitting that a single run tells you almost nothing.\n\nFirst, every eval case runs N times, and the report shows both numbers. Here's the shape of the harness, stripped down:\n\n``` python\nfrom collections import Counter\n\ndef run_case(agent, case, n=5):\n    results = [agent.run(case.input) for _ in range(n)]\n    passes = [case.check(r) for r in results]\n    return {\n        \"case\": case.id,\n        \"per_run\": sum(passes) / n,\n        \"all_pass\": all(passes),\n        \"paths\": Counter(tuple(r.tool_calls) for r in results),\n    }\n\ndef report(agent, cases, n=5):\n    rows = [run_case(agent, c, n) for c in cases]\n    per_run = sum(r[\"per_run\"] for r in rows) / len(rows)\n    consistent = sum(r[\"all_pass\"] for r in rows) / len(rows)\n    print(f\"per-run pass: {per_run:.0%}  all-{n} pass: {consistent:.0%}\")\n    for r in rows:\n        if 0 < r[\"per_run\"] < 1:\n            print(r[\"case\"], \"FLAKY\", dict(r[\"paths\"]))\n```\n\nThat `paths` counter is the part that earns its keep. A case that passes 3 of 5 with two different tool-call sequences tells you exactly where to look. On the invoice agent, the flaky cases all diverged at the same step: a vendor lookup that sometimes returned two matches, and the model picked differently each time. One line in the system prompt fixed it. I'd never have found it from a pass/fail table.\n\nSecond, I stopped reporting the per-run rate to clients as \"accuracy\". I report the all-N rate as the headline and the per-run rate as a footnote, because the all-N rate is the one that predicts support tickets. Clients don't love hearing 61% instead of 84%. They love it more than the screenshot of two vendors for one PDF.\n\nThird, flaky cases block the release. A case that passes 4 of 5 used to be \"basically fine\". Now it's a bug with a known reproduction, and it gets the same treatment I described in [when not to use AI automation](https://abrarqasim.com/blog/when-not-to-use-ai-automation-the-refund-bot-that-cost-me-a-client/): either the step gets deterministic (a rule, a lookup, a hard filter) or the whole path gets a human approval gate.\n\nThe paper's actual mechanism, turning unstable steps into remembered guidelines, is worth trying and I've started to. My version is much dumber than theirs. When the harness flags a flaky case, I write the guideline by hand, drop it into a per-task notes file, and the agent's system prompt loads notes matching the task type. No analyser, no generator, one developer with a text editor.\n\nThat works for a handful of task types. It stops working around twenty, which is roughly where I'd want their automated version. Something I keep in mind from [my post on agent memory](https://abrarqasim.com/blog/ai-agent-memory-is-a-dose-not-a-switch/): every guideline you inject is context the model has to weigh against everything else, and I've watched a pile of well-meaning rules make an agent worse at the cases that were never flaky. The paper doesn't report the cost side of that trade, or I missed it, and I'd like to see it.\n\nThe other caveat is the model. Their numbers are on GPT-4.1 with a ReAct loop. I haven't seen a published consistency gap for the current frontier models, and I'd guess it's smaller but not gone. I ran a quick five-repeat on my own invoice cases with Claude Sonnet 5 last night: per-run 91%, all-five 79%. Twelve points. Smaller gap, same shape, and that's one evening's data on a tiny set, so take it as an anecdote and not a finding.\n\nIf you want a broader lens on where agents fail, another paper from the same arXiv batch, [AgentAudit](https://arxiv.org/abs/2609.09875), evaluates the whole trace (planning, tool selection, tool execution, memory) and attributes failures to a stage rather than a pass/fail. Different goal, same underlying point: the interesting failures are inside the trajectory, and a single end-to-end score hides them.\n\nYou don't need the framework. You need one loop.\n\nTake the ten test cases you already have for whatever agent you've shipped, run each one five times tonight, and count how many pass all five. Put both numbers side by side. If the gap is under 5 points, good, you've earned the word \"stable\" and I'm a little jealous. If it's 15 or more, find the cases that pass 3 of 5, dump the tool-call sequences from each run, and look for the step where they fork. In my experience it's the same step across most of the flaky cases, and it's usually a lookup that returns more than one thing.\n\nThen decide, per flaky step, whether it becomes a rule or a human checkpoint. Neither option is glamorous. Both beat writing \"stable\" in a doc you'll have to explain later.\n\nI build and test this kind of agent for clients as part of my [freelance work](https://abrarqasim.com), and the five-run harness above is now the first thing I set up on a new project, before the prompt, before the tools. It has changed what I'm willing to promise.\n\n*Originally published at [abrarqasim.com](https://abrarqasim.com/blog/ai-agent-testing-the-consistency-gap-77-percent-pass-53-percent-repeat/). I write there about React, PHP, Rust, Go and the AI tooling around them.*", "url": "https://wpnews.pro/news/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production", "canonical_source": "https://dev.to/abyzgenic/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production-3ln", "published_at": "2026-09-14 06:40:20+00:00", "updated_at": "2026-09-14 07:02:04.285123+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-safety", "ai-tools"], "entities": ["IBM Research", "Evelyn Duesterwald", "AppWorld", "GPT-4.1", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production", "markdown": "https://wpnews.pro/news/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production.md", "text": "https://wpnews.pro/news/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production.txt", "jsonld": "https://wpnews.pro/news/ai-agent-testing-why-a-77-pass-rate-can-mean-53-in-production.jsonld"}}