Reliability, Cost, and Latency A lesson on agent reliability evaluation argues that teams should report pass^k — the probability that all k attempts succeed — rather than pass@k for user-facing agents, citing τ-bench as the benchmark that popularized the metric. The lesson's worked example shows an agent with pass@1 of 0.84 and pass@4 of 0.96 but pass^4 of only 0.61, the rate users actually experience across repeat contacts, and it recommends Wilson score intervals, which give 43/50 as a range of 0.738 to 0.930 versus 0.827 to 0.888 for 430/500. The lesson also advises sorting flaky cases by pass rate and comparing failing and passing traces to find the first divergent step, and measuring cost and latency per task alongside quality. · Agentic AI · 8 min read 📋 Prerequisites - Safety, Red-Teaming, and Adversarial Evaluation previous lesson 🎯 What You'll Learn - Distinguish pass@k from pass^k and choose the right one for a product - Report pass rates with confidence intervals and estimate the sample size needed to detect a change - Compare two agent versions with a paired test instead of eyeballing averages - Measure and budget cost and latency per task alongside quality “It Passed” Is Not a Reliability Claim Run a Quill case once and it passes. Run it ten times and it passes seven. Which is the truth? Both, and neither. The agent doesn’t have a pass/fail property per case; it has a probability of passing . Every eval number you report is an estimate of probabilities like that, and estimates come with uncertainty. This lesson is the statistics you need to report agent quality honestly. It’s not much math, but skipping it is how teams ship regressions they “measured” as improvements. pass@k vs pass^k Run each case k times. Two very different metrics come out of that. pass@k : the probability that at least one of k attempts succeeds. Useful when a human or verifier picks the best attempt, as in code generation with tests. pass^k “pass hat k” : the probability that all k attempts succeed. This is the one that matters for agents serving users. Each user gets one attempt, and you need the agent to succeed for every user. pass^k was popularized by the τ-bench agent benchmark precisely because pass@1 hid how inconsistent agents were. If a case has per-trial success probability p : - pass@k = 1 − 1 − p ^k - pass^k = p^k python def per case estimates results: dict str, list bool , k: int : """results: case id - list of n trial outcomes, n = k. Unbiased estimators.""" from math import comb pass at k, pass hat k = , for trials in results.values : n, c = len trials , sum trials pass at k.append 1 - comb n - c, k / comb n, k pass hat k.append comb c, k / comb n, k return sum pass at k / len pass at k , sum pass hat k / len pass hat k A real example of how much these diverge: pass@1 0.84 pass@4 0.96 ← looks great pass^4 0.61 ← what users actually experience across repeat contacts An agent with pass@1 of 84% can have only 61% of cases that work reliably . The other cases work sometimes, which in production means some users get help and others with the identical problem don’t. Report pass^k for any user-facing agent. It’s the metric most sensitive to the inconsistency that erodes trust. Finding Flaky Cases Per-case pass rates across trials split the dataset into three groups: - Always pass : stable capability - Always fail : a consistent bug, usually easy to diagnose - Flaky passes sometimes : the most important group. These are where small prompt or model changes swing your aggregate score, and where users experience inconsistency. flaky = {cid: sum t / len t for cid, t in results.items if 0 < sum t < len t } Sort flaky cases by pass rate and read their failing traces next to their passing traces. The divergence point the first step where the two trajectories differ is usually the decision to fix. Confidence Intervals A pass rate of 86% on 50 cases is not the same claim as 86% on 500. Always report an interval. For proportions, the Wilson score interval behaves well even near 0% and 100%: php from math import sqrt def wilson ci successes: int, n: int, z: float = 1.96 - tuple float, float : if n == 0: return 0.0, 1.0 p = successes / n denom = 1 + z 2 / n center = p + z 2 / 2 n / denom half = z sqrt p 1 - p / n + z 2 / 4 n 2 / denom return center - half, center + half print wilson ci 43, 50 0.738, 0.930 print wilson ci 430, 500 0.827, 0.888 43/50 is “somewhere between 74% and 93%.” That’s a wide range. If yesterday’s version scored 82% on the same 50 cases, you can’t claim today’s is better. Multiple trials per case aren’t independent samples. Ten trials of 50 cases isn’t 500 independent observations; trials of the same case are correlated. Use a cluster bootstrap : resample cases with replacement keeping all their trials together , recompute the metric, and take percentiles. python import random def cluster bootstrap ci results: dict str, list bool , metric, n boot=2000, seed=0 : rng = random.Random seed ids = list results stats = for in range n boot : sample = {f"{i} {j}": results rng.choice ids for j, i in enumerate ids } stats.append metric sample stats.sort return stats int 0.025 n boot , stats int 0.975 n boot mean pass = lambda r: sum sum t / len t for t in r.values / len r Comparing Two Versions: Use a Paired Test The question you usually need answered isn’t “what’s the pass rate?” but “ is version B better than version A? ” Comparing two independent confidence intervals is wasteful. Both versions ran on the same cases, so use that pairing. For each case, compute the difference in per-case pass rate B minus A . Then bootstrap the mean difference: python def paired bootstrap a: dict str, list bool , b: dict str, list bool , n boot=5000, seed=0 - dict: rng = random.Random seed ids = sorted set a & set b diffs = sum b i / len b i - sum a i / len a i for i in ids observed = sum diffs / len diffs boots = sorted sum rng.choice diffs for in diffs / len diffs for in range n boot lo, hi = boots int 0.025 n boot , boots int 0.975 n boot return {"delta": observed, "ci95": lo, hi , "verdict": "better" if lo 0 else "worse" if hi < 0 else "inconclusive", "improved": sum d 0 for d in diffs , "regressed": sum d < 0 for d in diffs } Report this table for every comparison: version B vs A 200 cases × 5 trials Δ pass rate +3.4 pts 95% CI +0.9, +6.0 → better cases improved 23 cases regressed 9 ← read these before merging Always read the regressed cases , even when the aggregate improves. A change that fixes 23 low-risk cases and breaks 9 refund cases may not be a change you want. For binary single-trial comparisons, McNemar’s test on the discordant pairs cases where exactly one version passed is the classic equivalent. How Many Cases and Trials? To detect a difference of size d in pass rate with reasonable power, you need roughly: php def cases needed p: float, d: float, z alpha=1.96, z beta=0.84 - int: """Approximate n per version for an unpaired comparison of two proportions.""" p2 = p + d var = p 1 - p + p2 1 - p2 return int z alpha + z beta 2 var / d 2 + 1 for d in 0.10, 0.05, 0.02 : print f"detect {d:.0%} change at 85% baseline → ~{cases needed 0.85, d } cases" detect 10% change at 85% baseline → ~138 cases detect 5% change at 85% baseline → ~683 cases detect 2% change at 85% baseline → ~4716 cases Paired designs and multiple trials reduce these numbers substantially, often by half or more, because they remove between-case variance. But the lesson stands: with 50 cases, you can only detect large changes. That’s fine for a smoke suite. It isn’t fine for deciding whether to adopt a new model. Practical guidance: | Purpose | Cases | Trials | |---|---|---| | Smoke catch breakage | 20–30 | 1 | | PR regression catch 5–10 pt drops | 150–300 | 2–3 | | Release / model adoption decision | 300–1000 | 3–5 | | Flakiness analysis | Any | 5–10 | Cost: Measure It Per Task Every eval run already has token counts on its LLM spans Lesson 2 . Turn them into dollars: PRICES PER MTOK = { illustrative; load from config, prices change "model-large": {"input": 3.00, "output": 15.00}, "model-small": {"input": 0.80, "output": 4.00}, } def estimate cost trace, default model: str - float: total = 0.0 for s in trace.of kind "llm" : m = s.attributes.get "gen ai.request.model", default model p = PRICES PER MTOK m total += s.attributes.get "tokens.input", 0 / 1e6 p "input" total += s.attributes.get "tokens.output", 0 / 1e6 p "output" return total Report cost as a distribution , not just a mean. Agents have long tails: most tasks take four LLM calls, but a looping run takes forty. The p95 cost and the max are where budget surprises come from. cost / task mean $0.021 p50 $0.017 p95 $0.058 max $0.31 Then look at cost per successful task : total cost divided by number of successes. A cheaper model that fails more often can be more expensive per resolved ticket once retries and escalations are counted. Latency: End-to-End and Per Step Measure wall-clock latency per run, and break it down by span kind: latency / task p50 4.1s p95 11.8s llm 71% tool 26% overhead 3% Latency in agents is driven mostly by the number of steps , so trajectory efficiency Lesson 7 is also a latency metric. If p95 latency jumps after a prompt change, check whether mean tool calls went up before looking at infrastructure. The Pareto View Quality, cost, and latency trade off. When comparing configurations models, prompts, max steps , plot them together and look for the Pareto frontier : configurations where nothing else is better on all three axes. config pass^3 $/task p95 latency large model, v4 prompt 0.78 $0.041 12.3s small model, v4 prompt 0.71 $0.009 6.1s ← Pareto large model, v3 prompt 0.74 $0.044 13.0s ← dominated small model + large for refunds only 0.77 $0.016 7.4s ← Pareto The routing configuration in the last row, a small model by default and a large one for money-moving decisions, is often where the best tradeoff lives. You can only find it if every eval report includes cost and latency next to quality. Key Takeaways - Agents have a probability of passing each case; one run isn’t a measurement - Report pass^k for user-facing agents; pass@k overstates reliability - Identify flaky cases and compare their passing and failing traces - Always report confidence intervals ; use a cluster bootstrap when you have multiple trials per case - Compare versions with a paired test, and always read the regressed cases - 50 cases detects only large changes; size the suite to the decision it supports - Report cost and latency as distributions, plus cost per successful task, and look for the Pareto frontier Exercise 1. Run your regression split with 5 trials per case 2. Report pass@1, pass^3, and pass^5 with cluster-bootstrap confidence intervals 3. List the flaky cases and diagnose the top three by comparing passing and failing traces 4. Make one change prompt, model, or max steps , rerun, and produce the paired comparison table 5. Add cost mean, p95, per success and latency p50, p95 to your report