# Reliability, Cost, and Latency

> Source: <https://superml.org/tutorials/agent-eval-reliability-cost-latency>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Run each eval case k times and report pass^k with Wilson confidence intervals; sort flaky cases by pass rate and diff their failing versus passing traces to find the first divergent step.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

· Agentic AI · 8 min read

### 📋 Prerequisites

- Safety, Red-Teaming, and Adversarial Evaluation (previous lesson)

### 🎯 What You'll Learn

- Distinguish pass@k from pass^k and choose the right one for a product
- Report pass rates with confidence intervals and estimate the sample size needed to detect a change
- Compare two agent versions with a paired test instead of eyeballing averages
- Measure and budget cost and latency per task alongside quality

## “It Passed” Is Not a Reliability Claim

Run a Quill case once and it passes. Run it ten times and it passes seven. Which is the truth? Both, and neither. The agent doesn’t have a pass/fail property per case; it has a **probability of passing**. Every eval number you report is an estimate of probabilities like that, and estimates come with uncertainty.

This lesson is the statistics you need to report agent quality honestly. It’s not much math, but skipping it is how teams ship regressions they “measured” as improvements.

## pass@k vs pass^k

Run each case *k* times. Two very different metrics come out of that.

**pass@k**: the probability that *at least one* of *k* attempts succeeds. Useful when a human or verifier picks the best attempt, as in code generation with tests.

**pass^k** (“pass hat k”): the probability that *all k* attempts succeed. This is the one that matters for agents serving users. Each user gets one attempt, and you need the agent to succeed for *every* user. pass^k was popularized by the τ-bench agent benchmark precisely because pass@1 hid how inconsistent agents were.

If a case has per-trial success probability *p*:

- pass@k = 1 − (1 − p)^k
- pass^k = p^k

``` python
def per_case_estimates(results: dict[str, list[bool]], k: int):
    """results: case_id -> list of n trial outcomes, n >= k. Unbiased estimators."""
    from math import comb
    pass_at_k, pass_hat_k = [], []
    for trials in results.values():
        n, c = len(trials), sum(trials)
        pass_at_k.append(1 - comb(n - c, k) / comb(n, k))
        pass_hat_k.append(comb(c, k) / comb(n, k))
    return sum(pass_at_k) / len(pass_at_k), sum(pass_hat_k) / len(pass_hat_k)
```

A real example of how much these diverge:

```
pass@1   0.84
pass@4   0.96     ← looks great
pass^4   0.61     ← what users actually experience across repeat contacts
```

An agent with pass@1 of 84% can have only 61% of cases that work *reliably*. The other cases work sometimes, which in production means some users get help and others with the identical problem don’t. **Report pass^k for any user-facing agent.** It’s the metric most sensitive to the inconsistency that erodes trust.

### Finding Flaky Cases

Per-case pass rates across trials split the dataset into three groups:

- **Always pass** : stable capability
- **Always fail** : a consistent bug, usually easy to diagnose
- **Flaky** (passes sometimes): the most important group. These are where small prompt or model changes swing your aggregate score, and where users experience inconsistency.

```
flaky = {cid: sum(t) / len(t) for cid, t in results.items() if 0 < sum(t) < len(t)}
```

Sort flaky cases by pass rate and read their failing traces next to their passing traces. The divergence point (the first step where the two trajectories differ) is usually the decision to fix.

## Confidence Intervals

A pass rate of 86% on 50 cases is not the same claim as 86% on 500. Always report an interval. For proportions, the **Wilson score interval** behaves well even near 0% and 100%:

``` php
from math import sqrt

def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return (center - half, center + half)

print(wilson_ci(43, 50))    # (0.738, 0.930)
print(wilson_ci(430, 500))  # (0.827, 0.888)
```

43/50 is “somewhere between 74% and 93%.” That’s a wide range. If yesterday’s version scored 82% on the same 50 cases, you can’t claim today’s is better.

**Multiple trials per case aren’t independent samples.** Ten trials of 50 cases isn’t 500 independent observations; trials of the same case are correlated. Use a **cluster bootstrap**: resample *cases* with replacement (keeping all their trials together), recompute the metric, and take percentiles.

``` python
import random

def cluster_bootstrap_ci(results: dict[str, list[bool]], metric, n_boot=2000, seed=0):
    rng = random.Random(seed)
    ids = list(results)
    stats = []
    for _ in range(n_boot):
        sample = {f"{i}_{j}": results[rng.choice(ids)] for j, i in enumerate(ids)}
        stats.append(metric(sample))
    stats.sort()
    return stats[int(0.025 * n_boot)], stats[int(0.975 * n_boot)]

mean_pass = lambda r: sum(sum(t) / len(t) for t in r.values()) / len(r)
```

## Comparing Two Versions: Use a Paired Test

The question you usually need answered isn’t “what’s the pass rate?” but “**is version B better than version A?**” Comparing two independent confidence intervals is wasteful. Both versions ran on the *same* cases, so use that pairing.

For each case, compute the difference in per-case pass rate (B minus A). Then bootstrap the mean difference:

``` python
def paired_bootstrap(a: dict[str, list[bool]], b: dict[str, list[bool]],
                     n_boot=5000, seed=0) -> dict:
    rng = random.Random(seed)
    ids = sorted(set(a) & set(b))
    diffs = [sum(b[i]) / len(b[i]) - sum(a[i]) / len(a[i]) for i in ids]
    observed = sum(diffs) / len(diffs)
    boots = sorted(sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(n_boot))
    lo, hi = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)]
    return {"delta": observed, "ci95": (lo, hi),
            "verdict": "better" if lo > 0 else "worse" if hi < 0 else "inconclusive",
            "improved": sum(d > 0 for d in diffs), "regressed": sum(d < 0 for d in diffs)}
```

Report this table for every comparison:

```
version B vs A (200 cases × 5 trials)
Δ pass rate     +3.4 pts   95% CI [+0.9, +6.0]   → better
cases improved   23
cases regressed   9   ← read these before merging
```

**Always read the regressed cases**, even when the aggregate improves. A change that fixes 23 low-risk cases and breaks 9 refund cases may not be a change you want.

For binary single-trial comparisons, **McNemar’s test** on the discordant pairs (cases where exactly one version passed) is the classic equivalent.

## How Many Cases and Trials?

To detect a difference of size *d* in pass rate with reasonable power, you need roughly:

``` php
def cases_needed(p: float, d: float, z_alpha=1.96, z_beta=0.84) -> int:
    """Approximate n per version for an unpaired comparison of two proportions."""
    p2 = p + d
    var = p * (1 - p) + p2 * (1 - p2)
    return int(((z_alpha + z_beta) ** 2 * var) / d**2) + 1

for d in (0.10, 0.05, 0.02):
    print(f"detect {d:.0%} change at 85% baseline → ~{cases_needed(0.85, d)} cases")
detect 10% change at 85% baseline → ~138 cases
detect 5% change at 85% baseline → ~683 cases
detect 2% change at 85% baseline → ~4716 cases
```

Paired designs and multiple trials reduce these numbers substantially, often by half or more, because they remove between-case variance. But the lesson stands: **with 50 cases, you can only detect large changes.** That’s fine for a smoke suite. It isn’t fine for deciding whether to adopt a new model.

Practical guidance:

| Purpose | Cases | Trials | 
|---|---|---|
| Smoke (catch breakage) | 20–30 | 1 | 
| PR regression (catch 5–10 pt drops) | 150–300 | 2–3 | 
| Release / model adoption decision | 300–1000 | 3–5 | 
| Flakiness analysis | Any | 5–10 | 

## Cost: Measure It Per Task

Every eval run already has token counts on its LLM spans (Lesson 2). Turn them into dollars:

```
PRICES_PER_MTOK = {           # illustrative; load from config, prices change
    "model-large":  {"input": 3.00, "output": 15.00},
    "model-small":  {"input": 0.80, "output": 4.00},
}

def estimate_cost(trace, default_model: str) -> float:
    total = 0.0
    for s in trace.of_kind("llm"):
        m = s.attributes.get("gen_ai.request.model", default_model)
        p = PRICES_PER_MTOK[m]
        total += s.attributes.get("tokens.input", 0) / 1e6 * p["input"]
        total += s.attributes.get("tokens.output", 0) / 1e6 * p["output"]
    return total
```

Report cost as a **distribution**, not just a mean. Agents have long tails: most tasks take four LLM calls, but a looping run takes forty. The p95 cost and the max are where budget surprises come from.

```
cost / task    mean $0.021   p50 $0.017   p95 $0.058   max $0.31
```

Then look at **cost per successful task**: total cost divided by number of successes. A cheaper model that fails more often can be more expensive per resolved ticket once retries and escalations are counted.

## Latency: End-to-End and Per Step

Measure wall-clock latency per run, and break it down by span kind:

```
latency / task   p50 4.1s   p95 11.8s
  llm            71%
  tool           26%
  overhead        3%
```

Latency in agents is driven mostly by the **number of steps**, so trajectory efficiency (Lesson 7) is also a latency metric. If p95 latency jumps after a prompt change, check whether mean tool calls went up before looking at infrastructure.

## The Pareto View

Quality, cost, and latency trade off. When comparing configurations (models, prompts, max steps), plot them together and look for the **Pareto frontier**: configurations where nothing else is better on all three axes.

```
config                  pass^3   $/task   p95 latency
large model, v4 prompt   0.78    $0.041     12.3s
small model, v4 prompt   0.71    $0.009      6.1s   ← Pareto
large model, v3 prompt   0.74    $0.044     13.0s   ← dominated
small model + large for refunds only
                         0.77    $0.016      7.4s   ← Pareto
```

The routing configuration in the last row, a small model by default and a large one for money-moving decisions, is often where the best tradeoff lives. You can only find it if every eval report includes cost and latency next to quality.

## Key Takeaways

- Agents have a *probability* of passing each case; one run isn’t a measurement
- Report **pass^k** for user-facing agents; pass@k overstates reliability
- Identify **flaky cases** and compare their passing and failing traces
- Always report **confidence intervals** ; use a cluster bootstrap when you have multiple trials per case
- Compare versions with a **paired** test, and always read the regressed cases
- 50 cases detects only large changes; size the suite to the decision it supports
- Report **cost** and**latency** as distributions, plus cost per successful task, and look for the Pareto frontier

## Exercise

1. Run your regression split with 5 trials per case
2. Report pass@1, pass^3, and pass^5 with cluster-bootstrap confidence intervals
3. List the flaky cases and diagnose the top three by comparing passing and failing traces
4. Make one change (prompt, model, or max steps), rerun, and produce the paired comparison table
5. Add cost (mean, p95, per success) and latency (p50, p95) to your report
