Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests

I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved, and two weeks later a specific intent is failing in production. The regression was real. The improvement was not. Nobody did anything careless — the eval reported a number that looked like a measurement and was actually a coin flip.

The unglamorous part of LLMOps is not prompt management or tracing. It is that most eval suites cannot detect the changes teams ship on. Fixing that is a half-day of statistics and one CI job.

Before you compare two configurations, know what is moving:

You cannot remove all three. You can remove the first one almost entirely by comparing paired results: run baseline and candidate on the same examples and analyze the per-example difference. Example difficulty cancels out. A prompt that fixes one hard case and breaks another now shows up as one +1 and one -1, not as an aggregate you have to reason about.

For a paired comparison of pass/fail outcomes, the standard error of the mean difference depends on the disagreement rate — how often the two configs disagree on the same example — not on the accuracy.

If 15% of examples flip between your two configurations and you have 200 paired examples:

So on that setup, a 2-point bump is invisible. To see a 5-point change under the same assumptions you need roughly 400 paired examples; to see 3 points, about 1,100. These are arithmetic, not benchmarks — plug in your own disagreement rate, which is usually higher than you expect for prompt changes.

Two consequences follow. First, stop gating on raw averages. Second, if your golden set is 200 examples and you cannot grow it, your gate should be "no regression beyond X" rather than "improvement required."

The whole thing is about thirty lines of stdlib Python. Bootstrap the paired differences, look at the interval, and ship only when the interval clears zero.

python
"""paired_gate.py — decide whether an LLM change is real or noise.
Run: python paired_gate.py    Deps: none (stdlib only)."""
import random

def paired_bootstrap(baseline, candidate, iters=5000, seed=7):
    """Scores aligned on the SAME example ids, 0/1 or 0..1.
    Returns (mean delta, 2.5th pct, 97.5th pct)."""
    diffs = [c - b for b, c in zip(baseline, candidate)]
    n = len(diffs)
    rng = random.Random(seed)  # fixed seed: the gate must be reproducible
    means = sorted(
        sum(diffs[rng.randrange(n)] for _ in range(n)) / n
        for _ in range(iters)
    )
    return sum(diffs) / n, means[int(0.025 * iters)], means[int(0.975 * iters)]

def required_n(disagreement_rate, mde, z_alpha=1.96, z_power=0.84):
    """Paired binary comparison: examples needed per side."""
    se = mde / (z_alpha + z_power)
    p = disagreement_rate
    return int((p * (1 - p)) / se**2) + 1

if __name__ == "__main__":
    baseline  = [1] * 160 + [0] * 40
    candidate = [1] * 166 + [0] * 34
    delta, lo, hi = paired_bootstrap(baseline, candidate)
    print(f"delta