Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests

A developer published a stdlib-Python approach for gating LLM changes on paired, statistically powered eval tests rather than raw average scores, arguing that most eval suites cannot detect the changes teams actually ship. The method bootstraps per-example differences between baseline and candidate on the same examples and ships only when the confidence interval clears zero, with a helper for computing required sample size from the disagreement rate. In the worked example, a 15% disagreement rate over 200 paired examples leaves a 2-point bump invisible, requiring roughly 400 examples to detect a 5-point change and about 1,100 for 3 points.

I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved, and two weeks later a specific intent is failing in production. The regression was real. The improvement was not. Nobody did anything careless — the eval reported a number that looked like a measurement and was actually a coin flip. The unglamorous part of LLMOps is not prompt management or tracing. It is that most eval suites cannot detect the changes teams ship on. Fixing that is a half-day of statistics and one CI job. Before you compare two configurations, know what is moving: You cannot remove all three. You can remove the first one almost entirely by comparing paired results: run baseline and candidate on the same examples and analyze the per-example difference. Example difficulty cancels out. A prompt that fixes one hard case and breaks another now shows up as one +1 and one -1, not as an aggregate you have to reason about. For a paired comparison of pass/fail outcomes, the standard error of the mean difference depends on the disagreement rate — how often the two configs disagree on the same example — not on the accuracy. If 15% of examples flip between your two configurations and you have 200 paired examples: So on that setup, a 2-point bump is invisible. To see a 5-point change under the same assumptions you need roughly 400 paired examples; to see 3 points, about 1,100. These are arithmetic, not benchmarks — plug in your own disagreement rate, which is usually higher than you expect for prompt changes. Two consequences follow. First, stop gating on raw averages. Second, if your golden set is 200 examples and you cannot grow it, your gate should be "no regression beyond X" rather than "improvement required." The whole thing is about thirty lines of stdlib Python. Bootstrap the paired differences, look at the interval, and ship only when the interval clears zero. python """paired gate.py — decide whether an LLM change is real or noise. Run: python paired gate.py Deps: none stdlib only .""" import random def paired bootstrap baseline, candidate, iters=5000, seed=7 : """Scores aligned on the SAME example ids, 0/1 or 0..1. Returns mean delta, 2.5th pct, 97.5th pct .""" diffs = c - b for b, c in zip baseline, candidate n = len diffs rng = random.Random seed fixed seed: the gate must be reproducible means = sorted sum diffs rng.randrange n for in range n / n for in range iters return sum diffs / n, means int 0.025 iters , means int 0.975 iters def required n disagreement rate, mde, z alpha=1.96, z power=0.84 : """Paired binary comparison: examples needed per side.""" se = mde / z alpha + z power p = disagreement rate return int p 1 - p / se 2 + 1 if name == " main ": 200 paired examples; candidate looks "+3 points" better. baseline = 1 160 + 0 40 candidate = 1 166 + 0 34 delta, lo, hi = paired bootstrap baseline, candidate print f"delta