{"slug": "your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests", "title": "Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests", "summary": "A developer published a stdlib-Python approach for gating LLM changes on paired, statistically powered eval tests rather than raw average scores, arguing that most eval suites cannot detect the changes teams actually ship. The method bootstraps per-example differences between baseline and candidate on the same examples and ships only when the confidence interval clears zero, with a helper for computing required sample size from the disagreement rate. In the worked example, a 15% disagreement rate over 200 paired examples leaves a 2-point bump invisible, requiring roughly 400 examples to detect a 5-point change and about 1,100 for 3 points.", "body_md": "I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved, and two weeks later a specific intent is failing in production. The regression was real. The improvement was not. Nobody did anything careless — the eval reported a number that looked like a measurement and was actually a coin flip.\n\nThe unglamorous part of LLMOps is not prompt management or tracing. It is that most eval suites cannot detect the changes teams ship on. Fixing that is a half-day of statistics and one CI job.\n\nBefore you compare two configurations, know what is moving:\n\nYou cannot remove all three. You can remove the first one almost entirely by comparing **paired** results: run baseline and candidate on the *same* examples and analyze the per-example difference. Example difficulty cancels out. A prompt that fixes one hard case and breaks another now shows up as one +1 and one -1, not as an aggregate you have to reason about.\n\nFor a paired comparison of pass/fail outcomes, the standard error of the mean difference depends on the **disagreement rate** — how often the two configs disagree on the same example — not on the accuracy.\n\nIf 15% of examples flip between your two configurations and you have 200 paired examples:\n\nSo on that setup, a 2-point bump is invisible. To see a 5-point change under the same assumptions you need roughly 400 paired examples; to see 3 points, about 1,100. These are arithmetic, not benchmarks — plug in your own disagreement rate, which is usually higher than you expect for prompt changes.\n\nTwo consequences follow. First, stop gating on raw averages. Second, if your golden set is 200 examples and you cannot grow it, your gate should be \"no regression beyond X\" rather than \"improvement required.\"\n\nThe whole thing is about thirty lines of stdlib Python. Bootstrap the paired differences, look at the interval, and ship only when the interval clears zero.\n\n```\npython\n\"\"\"paired_gate.py — decide whether an LLM change is real or noise.\nRun: python paired_gate.py    Deps: none (stdlib only).\"\"\"\nimport random\n\ndef paired_bootstrap(baseline, candidate, iters=5000, seed=7):\n    \"\"\"Scores aligned on the SAME example ids, 0/1 or 0..1.\n    Returns (mean delta, 2.5th pct, 97.5th pct).\"\"\"\n    diffs = [c - b for b, c in zip(baseline, candidate)]\n    n = len(diffs)\n    rng = random.Random(seed)  # fixed seed: the gate must be reproducible\n    means = sorted(\n        sum(diffs[rng.randrange(n)] for _ in range(n)) / n\n        for _ in range(iters)\n    )\n    return sum(diffs) / n, means[int(0.025 * iters)], means[int(0.975 * iters)]\n\ndef required_n(disagreement_rate, mde, z_alpha=1.96, z_power=0.84):\n    \"\"\"Paired binary comparison: examples needed per side.\"\"\"\n    se = mde / (z_alpha + z_power)\n    p = disagreement_rate\n    return int((p * (1 - p)) / se**2) + 1\n\nif __name__ == \"__main__\":\n    # 200 paired examples; candidate looks \"+3 points\" better.\n    baseline  = [1] * 160 + [0] * 40\n    candidate = [1] * 166 + [0] * 34\n    delta, lo, hi = paired_bootstrap(baseline, candidate)\n    print(f\"delta\n```\n\n", "url": "https://wpnews.pro/news/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests", "canonical_source": "https://dev.to/dima_kruglov_608741d9a416/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests-j7p", "published_at": "2026-10-25 22:00:49+00:00", "updated_at": "2026-10-25 22:13:33.445624+00:00", "lang": "en", "topics": ["mlops", "ai-tools", "large-language-models", "developer-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests", "markdown": "https://wpnews.pro/news/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests.md", "text": "https://wpnews.pro/news/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests.txt", "jsonld": "https://wpnews.pro/news/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests.jsonld"}}