{"slug": "reliability-cost-and-latency", "title": "Reliability, Cost, and Latency", "summary": "A lesson on agent reliability evaluation argues that teams should report pass^k — the probability that all k attempts succeed — rather than pass@k for user-facing agents, citing τ-bench as the benchmark that popularized the metric. The lesson's worked example shows an agent with pass@1 of 0.84 and pass@4 of 0.96 but pass^4 of only 0.61, the rate users actually experience across repeat contacts, and it recommends Wilson score intervals, which give 43/50 as a range of 0.738 to 0.930 versus 0.827 to 0.888 for 430/500. The lesson also advises sorting flaky cases by pass rate and comparing failing and passing traces to find the first divergent step, and measuring cost and latency per task alongside quality.", "body_md": "· Agentic AI · 8 min read\n\n### 📋 Prerequisites\n\n- Safety, Red-Teaming, and Adversarial Evaluation (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Distinguish pass@k from pass^k and choose the right one for a product\n- Report pass rates with confidence intervals and estimate the sample size needed to detect a change\n- Compare two agent versions with a paired test instead of eyeballing averages\n- Measure and budget cost and latency per task alongside quality\n\n## “It Passed” Is Not a Reliability Claim\n\nRun a Quill case once and it passes. Run it ten times and it passes seven. Which is the truth? Both, and neither. The agent doesn’t have a pass/fail property per case; it has a **probability of passing**. Every eval number you report is an estimate of probabilities like that, and estimates come with uncertainty.\n\nThis lesson is the statistics you need to report agent quality honestly. It’s not much math, but skipping it is how teams ship regressions they “measured” as improvements.\n\n## pass@k vs pass^k\n\nRun each case *k* times. Two very different metrics come out of that.\n\n**pass@k**: the probability that *at least one* of *k* attempts succeeds. Useful when a human or verifier picks the best attempt, as in code generation with tests.\n\n**pass^k** (“pass hat k”): the probability that *all k* attempts succeed. This is the one that matters for agents serving users. Each user gets one attempt, and you need the agent to succeed for *every* user. pass^k was popularized by the τ-bench agent benchmark precisely because pass@1 hid how inconsistent agents were.\n\nIf a case has per-trial success probability *p*:\n\n- pass@k = 1 − (1 − p)^k\n- pass^k = p^k\n\n``` python\ndef per_case_estimates(results: dict[str, list[bool]], k: int):\n    \"\"\"results: case_id -> list of n trial outcomes, n >= k. Unbiased estimators.\"\"\"\n    from math import comb\n    pass_at_k, pass_hat_k = [], []\n    for trials in results.values():\n        n, c = len(trials), sum(trials)\n        pass_at_k.append(1 - comb(n - c, k) / comb(n, k))\n        pass_hat_k.append(comb(c, k) / comb(n, k))\n    return sum(pass_at_k) / len(pass_at_k), sum(pass_hat_k) / len(pass_hat_k)\n```\n\nA real example of how much these diverge:\n\n```\npass@1   0.84\npass@4   0.96     ← looks great\npass^4   0.61     ← what users actually experience across repeat contacts\n```\n\nAn agent with pass@1 of 84% can have only 61% of cases that work *reliably*. The other cases work sometimes, which in production means some users get help and others with the identical problem don’t. **Report pass^k for any user-facing agent.** It’s the metric most sensitive to the inconsistency that erodes trust.\n\n### Finding Flaky Cases\n\nPer-case pass rates across trials split the dataset into three groups:\n\n- **Always pass** : stable capability\n- **Always fail** : a consistent bug, usually easy to diagnose\n- **Flaky** (passes sometimes): the most important group. These are where small prompt or model changes swing your aggregate score, and where users experience inconsistency.\n\n```\nflaky = {cid: sum(t) / len(t) for cid, t in results.items() if 0 < sum(t) < len(t)}\n```\n\nSort flaky cases by pass rate and read their failing traces next to their passing traces. The divergence point (the first step where the two trajectories differ) is usually the decision to fix.\n\n## Confidence Intervals\n\nA pass rate of 86% on 50 cases is not the same claim as 86% on 500. Always report an interval. For proportions, the **Wilson score interval** behaves well even near 0% and 100%:\n\n``` php\nfrom math import sqrt\n\ndef wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:\n    if n == 0:\n        return (0.0, 1.0)\n    p = successes / n\n    denom = 1 + z**2 / n\n    center = (p + z**2 / (2 * n)) / denom\n    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom\n    return (center - half, center + half)\n\nprint(wilson_ci(43, 50))    # (0.738, 0.930)\nprint(wilson_ci(430, 500))  # (0.827, 0.888)\n```\n\n43/50 is “somewhere between 74% and 93%.” That’s a wide range. If yesterday’s version scored 82% on the same 50 cases, you can’t claim today’s is better.\n\n**Multiple trials per case aren’t independent samples.** Ten trials of 50 cases isn’t 500 independent observations; trials of the same case are correlated. Use a **cluster bootstrap**: resample *cases* with replacement (keeping all their trials together), recompute the metric, and take percentiles.\n\n``` python\nimport random\n\ndef cluster_bootstrap_ci(results: dict[str, list[bool]], metric, n_boot=2000, seed=0):\n    rng = random.Random(seed)\n    ids = list(results)\n    stats = []\n    for _ in range(n_boot):\n        sample = {f\"{i}_{j}\": results[rng.choice(ids)] for j, i in enumerate(ids)}\n        stats.append(metric(sample))\n    stats.sort()\n    return stats[int(0.025 * n_boot)], stats[int(0.975 * n_boot)]\n\nmean_pass = lambda r: sum(sum(t) / len(t) for t in r.values()) / len(r)\n```\n\n## Comparing Two Versions: Use a Paired Test\n\nThe question you usually need answered isn’t “what’s the pass rate?” but “**is version B better than version A?**” Comparing two independent confidence intervals is wasteful. Both versions ran on the *same* cases, so use that pairing.\n\nFor each case, compute the difference in per-case pass rate (B minus A). Then bootstrap the mean difference:\n\n``` python\ndef paired_bootstrap(a: dict[str, list[bool]], b: dict[str, list[bool]],\n                     n_boot=5000, seed=0) -> dict:\n    rng = random.Random(seed)\n    ids = sorted(set(a) & set(b))\n    diffs = [sum(b[i]) / len(b[i]) - sum(a[i]) / len(a[i]) for i in ids]\n    observed = sum(diffs) / len(diffs)\n    boots = sorted(sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(n_boot))\n    lo, hi = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)]\n    return {\"delta\": observed, \"ci95\": (lo, hi),\n            \"verdict\": \"better\" if lo > 0 else \"worse\" if hi < 0 else \"inconclusive\",\n            \"improved\": sum(d > 0 for d in diffs), \"regressed\": sum(d < 0 for d in diffs)}\n```\n\nReport this table for every comparison:\n\n```\nversion B vs A (200 cases × 5 trials)\nΔ pass rate     +3.4 pts   95% CI [+0.9, +6.0]   → better\ncases improved   23\ncases regressed   9   ← read these before merging\n```\n\n**Always read the regressed cases**, even when the aggregate improves. A change that fixes 23 low-risk cases and breaks 9 refund cases may not be a change you want.\n\nFor binary single-trial comparisons, **McNemar’s test** on the discordant pairs (cases where exactly one version passed) is the classic equivalent.\n\n## How Many Cases and Trials?\n\nTo detect a difference of size *d* in pass rate with reasonable power, you need roughly:\n\n``` php\ndef cases_needed(p: float, d: float, z_alpha=1.96, z_beta=0.84) -> int:\n    \"\"\"Approximate n per version for an unpaired comparison of two proportions.\"\"\"\n    p2 = p + d\n    var = p * (1 - p) + p2 * (1 - p2)\n    return int(((z_alpha + z_beta) ** 2 * var) / d**2) + 1\n\nfor d in (0.10, 0.05, 0.02):\n    print(f\"detect {d:.0%} change at 85% baseline → ~{cases_needed(0.85, d)} cases\")\ndetect 10% change at 85% baseline → ~138 cases\ndetect 5% change at 85% baseline → ~683 cases\ndetect 2% change at 85% baseline → ~4716 cases\n```\n\nPaired designs and multiple trials reduce these numbers substantially, often by half or more, because they remove between-case variance. But the lesson stands: **with 50 cases, you can only detect large changes.** That’s fine for a smoke suite. It isn’t fine for deciding whether to adopt a new model.\n\nPractical guidance:\n\n| Purpose | Cases | Trials | \n|---|---|---|\n| Smoke (catch breakage) | 20–30 | 1 | \n| PR regression (catch 5–10 pt drops) | 150–300 | 2–3 | \n| Release / model adoption decision | 300–1000 | 3–5 | \n| Flakiness analysis | Any | 5–10 | \n\n## Cost: Measure It Per Task\n\nEvery eval run already has token counts on its LLM spans (Lesson 2). Turn them into dollars:\n\n```\nPRICES_PER_MTOK = {           # illustrative; load from config, prices change\n    \"model-large\":  {\"input\": 3.00, \"output\": 15.00},\n    \"model-small\":  {\"input\": 0.80, \"output\": 4.00},\n}\n\ndef estimate_cost(trace, default_model: str) -> float:\n    total = 0.0\n    for s in trace.of_kind(\"llm\"):\n        m = s.attributes.get(\"gen_ai.request.model\", default_model)\n        p = PRICES_PER_MTOK[m]\n        total += s.attributes.get(\"tokens.input\", 0) / 1e6 * p[\"input\"]\n        total += s.attributes.get(\"tokens.output\", 0) / 1e6 * p[\"output\"]\n    return total\n```\n\nReport cost as a **distribution**, not just a mean. Agents have long tails: most tasks take four LLM calls, but a looping run takes forty. The p95 cost and the max are where budget surprises come from.\n\n```\ncost / task    mean $0.021   p50 $0.017   p95 $0.058   max $0.31\n```\n\nThen look at **cost per successful task**: total cost divided by number of successes. A cheaper model that fails more often can be more expensive per resolved ticket once retries and escalations are counted.\n\n## Latency: End-to-End and Per Step\n\nMeasure wall-clock latency per run, and break it down by span kind:\n\n```\nlatency / task   p50 4.1s   p95 11.8s\n  llm            71%\n  tool           26%\n  overhead        3%\n```\n\nLatency in agents is driven mostly by the **number of steps**, so trajectory efficiency (Lesson 7) is also a latency metric. If p95 latency jumps after a prompt change, check whether mean tool calls went up before looking at infrastructure.\n\n## The Pareto View\n\nQuality, cost, and latency trade off. When comparing configurations (models, prompts, max steps), plot them together and look for the **Pareto frontier**: configurations where nothing else is better on all three axes.\n\n```\nconfig                  pass^3   $/task   p95 latency\nlarge model, v4 prompt   0.78    $0.041     12.3s\nsmall model, v4 prompt   0.71    $0.009      6.1s   ← Pareto\nlarge model, v3 prompt   0.74    $0.044     13.0s   ← dominated\nsmall model + large for refunds only\n                         0.77    $0.016      7.4s   ← Pareto\n```\n\nThe routing configuration in the last row, a small model by default and a large one for money-moving decisions, is often where the best tradeoff lives. You can only find it if every eval report includes cost and latency next to quality.\n\n## Key Takeaways\n\n- Agents have a *probability* of passing each case; one run isn’t a measurement\n- Report **pass^k** for user-facing agents; pass@k overstates reliability\n- Identify **flaky cases** and compare their passing and failing traces\n- Always report **confidence intervals** ; use a cluster bootstrap when you have multiple trials per case\n- Compare versions with a **paired** test, and always read the regressed cases\n- 50 cases detects only large changes; size the suite to the decision it supports\n- Report **cost** and**latency** as distributions, plus cost per successful task, and look for the Pareto frontier\n\n## Exercise\n\n1. Run your regression split with 5 trials per case\n2. Report pass@1, pass^3, and pass^5 with cluster-bootstrap confidence intervals\n3. List the flaky cases and diagnose the top three by comparing passing and failing traces\n4. Make one change (prompt, model, or max steps), rerun, and produce the paired comparison table\n5. Add cost (mean, p95, per success) and latency (p50, p95) to your report", "url": "https://wpnews.pro/news/reliability-cost-and-latency", "canonical_source": "https://superml.org/tutorials/agent-eval-reliability-cost-latency", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:34.333541+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research"], "entities": ["Quill", "τ-bench", "Wilson score interval"], "also_reported_by": [], "takeaway": "Report pass^k, not pass@k, for user-facing agents: an agent with pass@1 of 0.84 and pass@4 of 0.96 had pass^4 of just 0.61, and Wilson intervals put 43/50 at 0.738–0.930 versus 0.827–0.888 for 430/500.", "recommendation": {"advice": "Run each eval case k times and report pass^k with Wilson confidence intervals; sort flaky cases by pass rate and diff their failing versus passing traces to find the first divergent step.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/reliability-cost-and-latency", "markdown": "https://wpnews.pro/news/reliability-cost-and-latency.md", "text": "https://wpnews.pro/news/reliability-cost-and-latency.txt", "jsonld": "https://wpnews.pro/news/reliability-cost-and-latency.jsonld"}}