# My Agent Found Real Improvements. The Statistics Still Killed the Promotion. > Source: > Published: 2026-09-02 09:34:17+00:00 **Previously:** [9 Bugs That All Looked Like a Working System](https://dev.to/debashish_ghosal/9-bugs-that-all-looked-like-a-working-system-25mg) · [I Built an AI That Rewrites Its Own Prompts](//02-i-built-an-ai-that-rewrites-its-own-prompts-its-safety-gate-rejected-every-single-edit.md) · [The Edit That Fixed 4 Tasks and Broke 1](//03-the-edit-that-fixed-4-tasks-and-broke-1.md) · [The Gate Is the Product](https://dev.to/debashish_ghosal/i-gave-an-llm-the-keys-to-rewrite-its-own-prompt-then-built-a-gate-that-said-no-4150-times-1h46) · [The Doctor Who Diagnosed Every Patient](//07-the-doctor-who-diagnosed-every-patient-with-a-headache.md) · [4 Models, 0 Promotable Edits](//08-three-models-same-mistakes-why-i-stopped-blaming-the-llm.md) In v0.1.0, an edit fixed 4 tasks and broke 1. Net +3 on 26 tasks. p=0.23. Gate rejected. The ceiling was clear: if you do not move enough tasks, the gate should say no. In v0.2.0, we expanded the A/B corpus to 40 tasks. We fixed the pipeline bugs. We added rejection context. We tested stronger models. The math got cleaner, not kinder. The ceiling shifted. It did not disappear. The edit was real. It fixed 4 tasks and broke 1: | Task | Prompt A | Prompt B | Expected | Change | |---|---|---|---|---| | classify-015 | technical | urgent | urgent | FIXED | | classify-023 | security | urgent, security | urgent, security | FIXED | | classify-024 | feature | feature, billing | feature, billing | FIXED | | classify-029 | feature | other | other | FIXED | | classify-014 | technical | feature | technical | BROKEN | Net: +3. Mean delta = 0.115. p=0.23. The permutation test computes this by shuffling task labels 1,000 times and counting how often random chance produces a delta ≥ 0.115. 23% of the time — above the 5% threshold. The sign-test floor with 5 discordant pairs out of 26 is ~0.031 one-sided. Even a flawless edit that fixed all 5 would barely clear p<0.05 two-sided. We expanded to 40 tasks. We should have more power. Here's what Mistral 24B, our strongest analyzer, produced: | Iter | p-value | Mean delta | Accuracy | |---|---|---|---| | 1 | 0.55 | +0.025 | 64% | | 2 | 0.52 | +0.025 | 64% | | 3 | 1.0 | 0.0 | 64% | | 4 | 0.52 | +0.025 | 64% | | 5 | 0.77 | -0.025 | 64% | Mistral produced positive deltas in 3 of 5 iterations. That is real signal. But the delta is +0.025 — 2.5% improvement on 40 tasks. At p~0.5, there is roughly a coin-flip chance this is noise. The ceiling did not disappear. It moved: with 40 tasks, the sign-test floor for a flawless edit that moves 5 tasks is ~0.016 one-sided — clearable. But Mistral did not move 5 tasks with perfect accuracy. It moved roughly 1-2 tasks, directionally, inconsistently. A mean delta of +0.025 on 40 tasks is about one additional correct task. The naive assumption was: more tasks = more statistical power = easier for edits to clear the gate. That's true, but it assumes the analyzer finds edits proportional to the task set. It didn't. The analyzer's edits still moved 1-3 tasks out of 40 — the same absolute count as 26 tasks. The effect size shrank from 11.5% to 2.5%. More tasks actually made the problem **harder** because the analyzer diluted its signal across a larger set without finding more movable tasks. The statistical power argument works only if you hold effect size constant. We did not. The analyzer's effect size collapsed when we added more tasks. The bottleneck is not task count. It is the analyzer's inability to find edits that move enough tasks. The v0.1.0 edit had a side effect: it added "urgent" to tasks where the trigger keywords appeared but urgent didn't belong. The ExactMatch scorer punished it. v0.2.0 introduced label-set-aware scorers. This helped — "urgent, security" no longer fails when the expected is "security, urgent." But the fundamental tension remains: any rule that adds labels will struggle under strict scoring. Mistral's edits showed the same pattern. It kept trying to tighten urgency rules, which moved some tasks in the right direction but over-corrected on others. The analyzer never found an edit that was purely additive — no side effects, no regressions. The search space is too large and the analyzer has no exploration mechanism. | Factor | v0.1.0 (26 tasks) | v0.2.0 (40 tasks) | |---|---|---| | Corpus size | 26 | 40 | | Movable tasks per edit | 5 | 1-2 | | Mean delta | 0.115 | 0.025 | | p-value | 0.23 | 0.5+ | | Gate result | reject (confidence) | reject (effect size, confidence) | | Rejection context | none | wired into staged analyzer | | Analyzer diversity | repeated same edit | Mistral explored variants | | Promotable edit found? | No | No | **The statistical ceiling is real at any corpus size.** More tasks help only if the analyzer finds edits proportional to the task set. It won't. The fix is better search, not more data. **A positive control oracle edit would have saved us months.** If you hand the gate an edit that fixes every failure, and it still rejects, the problem is the task set. We never ran this test. We should have. **The permutation test is honest, not conservative.** p=0.55 doesn't mean "maybe." It means "55% chance this is noise." Mistral's +0.025 mean delta at p=0.55 is the gate being transparent about weak evidence. **v0.2.0 proved the pipeline is correct.** The math is the math. No bugs are masking the result. The only remaining problem is that the analyzer can't find edits strong enough to clear the gate. That's a search problem, not a math problem. The math killed every promotion. That's the most valuable thing it did. v0.1.0's p=0.23 was confusing. Was the gate too strict? Was the task set too small? Was the analyzer too weak? We couldn't tell. The math gave us an answer but not a diagnosis. v0.2.0's math gave us the diagnosis. With 40 tasks and Mistral 24B producing +0.025 mean delta at p~0.5, the numbers are unambiguous: the analyzer finds real but tiny improvements. The gate correctly rejects them. The task set is big enough. The pipeline is correct. The search is the bottleneck. We now know the exact mathematical relationship between corpus size, movable tasks, and the p-value floor. We know that 40 tasks with 5 movable pairs can clear p<0.05 if the edit is flawless. We know that Mistral moves 1-2 tasks, not 5. We know the gap between "what the analyzer produces" and "what the gate requires" is 3-8 movable tasks. That gap is measurable. It's tractable. It's the kind of problem that disappears with a better search algorithm — not a bigger model, not a bigger corpus, not a lower threshold. The math told us exactly where the bottleneck is and exactly what it would take to fix it. The permutation test did not kill our promotion. It saved us from shipping noise as improvement. That is the gate's job, and it did it exactly right.