# How to Calculate AI Agent Task Success Rate Before You Promise an SLA > Source: > Published: 2026-11-15 09:15:03+00:00 *Your agent can look reliable on every single step and still let customers down on a big share of jobs. Work out the real odds before they end up in a contract.* - Multiplying per-step reliability across a task shows how fast it decays, and the 99% needed per step for a 90% finish over ten steps is higher than most agents deliver. - Grade a random sample of real production runs against a written outcome, because a 90% result on 50 runs only supports a claim of about 79% at 95% confidence. - Separate loud failures your system catches from silent failures it doesn't, since silent ones are what trigger SLA credits and legal exposure. - Sierra's tau-bench found agents that pass a task once often fail it when repeated, so test consistency by running each task several times. - Write the SLA around the lower confidence bound, per task type, with human handoff counted as its own outcome. Ten steps at 95% each sounds like a rounding error away from perfect. Multiply it out and you get 59.9%, which means your agent fails about four tasks in every ten. That's the whole case for learning how to calculate ai agent task success rate before anyone signs a contract with your name on it. Most teams quote one accuracy figure, measured on a single step or a single answer, and the customer hears it as a promise about the entire job. The formula takes one line. Multiply each step's success rate together. If every step succeeds 99% of the time, ten steps give you 90.4%. At 98% you're at 81.7%. At 90% per step you land on 34.9%, which is worse than a coin flip with extra steps. Stretch the 95% agent to twenty steps and it finishes just 35.8% of its tasks. Run it backwards and the demand gets steep. To reach 95% end to end over ten steps, every step has to succeed about 99.5% of the time. Very few agents doing open-ended tool calls get there without help. Frankly, the accuracy figure on most pitch decks is a step-level or answer-level number, picked because it's the biggest one available. A customer reads it as the odds that the job gets done. Those are different quantities, and the gap widens with every step you add. [How to Calculate the Right AI Agent Error Budget Before You Ship](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) How to calculate ai agent error budget starts with picking a target below 100% reliability, then multiplying volume by allowed error rate; a 10,000-call agent at 99% gets a 100-mistake daily budget to spend. - [how to set error budget for AI agents](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) - [AI agent reliability metrics before shipping product](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) The multiplication has limits. It assumes each step fails independently and that nothing catches an error once it happens. Real agents break both assumptions. A bad retrieval at step two poisons steps three through nine, so failures cluster instead of spreading evenly, and a verification step can claw some of the loss back. Use the formula to see what's coming, then measure the actual number from actual runs. ## Count steps from your traces, not your architecture diagram Your diagram says the support agent has four stages. The trace says otherwise. Open a sample of production runs and count every model call, tool call and handoff. A refund task might look up the account, fetch the order, check the policy, draft a reply, call the payments API and write a ticket note. That's six steps on a good day and fourteen when the agent retries a search twice and stops to ask itself a clarifying question. Pull step counts for a few hundred runs and look at the spread, not the average. A median of seven steps and a 90th percentile of twenty are two different products with two different sets of odds. Group runs by task type and compute success for each group separately. A short lookup and a multi-system refund shouldn't share a number, and a customer who only buys the lookup shouldn't be quoted the refund's figure. This is also where you find out which step does the damage. Plot failures by position in the run. If a third of your lost tasks die at the same retrieval call, you don't have a ten-step reliability problem. You have one bad step, and fixing it is cheaper than anything else on this page. ## How to measure AI agent accuracy in production Define success as an outcome, and write the definition down before you look at results. 'Refund issued for the right amount against the right order, ticket closed, no human edit' can be checked. 'The reply looked fine' can't. Then grade a random sample of real runs, not the demo cases your team remembers. An LLM grader is fine for triage, but have a person check a slice of its verdicts, because graders tend to share blind spots with the agent they're grading. Sample size matters more than most founders expect. Say you grade 50 runs and 45 pass. That's 90%, and you'll want to quote 90%. The 95% Wilson confidence interval on 45 of 50 runs from about 79% to 96%. The same 90% measured on 500 runs tightens to roughly 87% to 92%. When you're about to sign, a bigger sample is worth more than a better prompt. The number that goes in the contract is the bottom of the range. If you promise the point estimate, you're betting that you were lucky when you measured, and you'll be wrong about half the time. [How to Build an Eval Set for AI Agents Before You Trust One in Production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) Air Canada was ordered to pay about C$812 in February 2024 after its chatbot invented a refund policy. This guide covers where to find test cases, how to score them with code before an LLM judge, and why each case should run three to five times. - [build an eval set for AI agents](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) - [AI agent test cases before production](https://startupfortune.com/how-to-build-an-eval-set-for-ai-agents-before-you-trust-one-in-production/) Next, split your failures into two piles. Loud failures are tool errors, timeouts, loops and malformed outputs. Your system sees them and can retry or hand off to a person. Silent failures are the agent confidently issuing the wrong refund, quoting the wrong policy or updating the wrong record. Nothing flags them, and they're what SLA credits and lawsuits are made of. If 6% of tasks fail loudly and get routed to a human while 2% fail silently, the figure that matters most for your risk is the 2%, plus the cost of handling the 6%. Loud failures also make retries look better than they are. One independent retry on a step that fails 5% of the time cuts its failure rate to 0.25%, and ten such steps then finish 97.5% of the time. But a retry only fires when something notices the failure. Silent errors sail past it, so the gain shows up in your loud-failure pile and barely touches the other one. ## What the published benchmarks already show You don't have to take the decay on faith. Sierra, the customer service agent company, published tau-bench in 2024, a test that puts agents in simulated retail and airline support conversations with real tool calls and policy rules. Its researchers reported that even a strong model like GPT-4o completed fewer than half of the tasks, and they introduced a metric called pass^k, which runs each task k times and counts it only if every run passes. On the retail tasks, they found, GPT-4o's pass^8 fell below 25%. Carnegie Mellon's TheAgentCompany benchmark, which simulates a small software firm with tasks across browsing, coding and messaging coworkers, showed the same thing from another angle. When it launched, the best agent in the study finished roughly a quarter of the tasks on its own. Both results come from older models, and scores have moved up since. The shape hasn't changed. Single-run accuracy flatters an agent, and consistency across repeated runs of the same task is much lower. The arithmetic is simple: a task an agent passes 90% of the time passes five times in a row only 59% of the time. If your customer sends the same kind of request every morning, pass^k is closer to their experience than your headline accuracy is. So run each task in your test set several times, not once. If a task passes on four of five runs, don't count it as 80% solved. Write it down as a task you can't yet promise. ## Writing an SLA you can keep Air Canada learned what an agent's wrong answer costs. In February 2024, a British Columbia tribunal ruled the airline was responsible for incorrect bereavement fare information its website chatbot gave a passenger, and ordered it to pay about C$812 in damages and fees. The airline had argued the chatbot was a separate entity. The tribunal didn't buy it. The amount was small, but the principle isn't: what your agent says and does is what you said and did. That's why an SLA on 'accuracy' is a trap. It has no denominator and no definition, and a customer's lawyer will happily supply both. Commit to three narrower things instead. First, an autonomous completion rate: the share of in-scope tasks the agent finishes correctly without a person touching them. Second, a safe-handoff rate: the share that get escalated cleanly with the context attached, counted as their own outcome and not as failures. Third, a ceiling on silent wrong actions, the one that carries the credits. Here's the method with made-up but realistic numbers. You grade 200 runs of a ten-step task and 176 complete correctly, which is 88%. The Wilson lower bound sits near 83%. Of the 24 that don't complete, 17 are caught and handed off, and 7 are silent errors, about 3.5%, with an upper bound of roughly 7% at the same confidence. A defensible contract says 80% autonomous completion and a 5% cap on silent errors, with credits kicking in below either line. You've left yourself room for the model to have a bad month, and you haven't promised a number you've only seen on a good day. Scope it by task type. Ten-step tasks and three-step tasks get separate commitments, because the first group's math looks nothing like the second's. A customer who wants the long workflow should pay for the extra checks it needs, not get it bundled at the price of the easy one. ## Moving the number before you sign You have three levers, and the cheapest is usually cutting steps. Take the 95% agent from the top. Trim it from ten steps to six and it finishes 73.5% of the time instead of 59.9%, without touching the model. Merge tool calls, hard-code the lookups that never change, and stop letting the agent re-plan between steps that have only one sensible next move. The second lever is a check that catches errors early. A verifier that spots 80% of a step's mistakes turns a 5% failure rate into 1%, and ten steps at 99% give you 90.4%. Be honest about the catch: a verifier is another model call with its own error rate, and if it shares the agent's blind spots it'll wave the same bad answers through. Test it the way you'd test the agent. The third is operational. Pin the model version you measured, and rerun the sample whenever the vendor ships a new one or you change a prompt. A provider update can move your end-to-end rate by several points in either direction, and you won't see it unless you graded the same tasks before and after. Put a monthly regrade on the calendar, and put the date of the last one in the contract file. Before you quote anything, check how many clean runs you actually have. If your test of 30 runs produced zero failures, the standard rule of three says the true failure rate could still be as high as 10%. **Also read:** [How to set an AI agent context window budget before long tasks fall apart](https://startupfortune.com/how-to-set-an-ai-agent-context-window-budget-before-long-tasks-fall-apart/) • [How to calculate burn rate variance startup boards actually check](https://startupfortune.com/how-to-calculate-burn-rate-variance-startup-boards-actually-check/) • [How to Calculate Sales Cycle Length by Deal Size Before You Forecast](https://startupfortune.com/how-to-calculate-sales-cycle-length-by-deal-size-before-you-forecast/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](https://startupfortune.com/community/) Almost there. Sign in and your reply posts straight away.