One seeded bug, 26 AI agents: all passed the tests, all stayed broken

A developer-run experiment found that 26 AI agents — spanning a 4-bit quantized 7B model up to Opus 5 — all passed the visible test suite for a seeded greedy-regex bug while all 26 remained broken, with ten runs across two labs and three model tiers producing ten hidden failures. The author reports that handing one agent the hidden checker as its visible spec fixed the task on the first try, writing re.sub(r"</?[A-Za-z!?][^>]*>", "", html), and argues the specification rather than the model was the binding constraint. The full account is published in docs/CONVERGENCE.md in the vyang472/five-bugs repository, which ships five single-file Python bugs with a verifier runnable via python3 scripts/verify.py.

26 agents, from a 4-bit quantized 7B model up to Opus 5, were given the same one-line bug and its test suite. All 26 made the tests pass. All 26 stayed broken. They wrote four different regexes to do it: re.sub r"< ^ ", "", html 14x Codex CLI, Opus 5, 4 repeats, 8 local re.sub r"< ^ + ", "", html 8x Claude Code, Sonnet 5, 1 repeat, 5 local re.sub r"<. ? ", "", html 3x local re.sub r"<.+? ", "", html 1x Haiku 4.5 All four turn the sentence a < b and c d into a d . Nobody was wrong to write them. The test suite they were handed contained only well-formed HTML, so that was the entire specification, and every one of them satisfied it exactly. Scaling the model across two orders of magnitude did not help, because the model was never the binding constraint. Handing one agent a better test fixed it on the first try. The full account — what was ruled out, and what it does and does not support — is in docs/CONVERGENCE.md https://github.com/vyang472/five-bugs/blob/main/docs/CONVERGENCE.md . Verify it in two seconds, without running an agent or spending anything: git clone https://github.com/vyang472/five-bugs cd five-bugs python3 scripts/verify.py Five single-file Python bugs. Each ships with the checker the agent is given, and a second checker, written afterwards, that no agent ever sees . The gap between those two is the whole experiment. Everything was run on one laptop: the tasks, the patch every agent actually wrote, raw timings, token counts and costs, and the scripts that produced them. Nothing here asks you to trust a number I typed. Claude Code and Codex CLI both fixed all five bugs, 5/5 on the checker they were given. It looked like a tie and a clean sweep. Then I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again. task agent visible hidden t1 claude PASS PASS t1 codex PASS PASS t2 claude PASS PASS t2 codex PASS PASS t3 claude PASS PASS t3 codex PASS PASS t4 claude PASS FAIL t4 codex PASS FAIL t5 claude PASS PASS t5 codex PASS PASS Four of five held. On the fifth, both agents failed in exactly the same way. The bug was a greedy regex: re.sub r"<.+ ", "", html the seeded bug So I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and Opus 5. Five agent configurations, two labs, three model tiers. They wrote three different patterns: | agent | patch | hidden | |---|---|---| | Claude Code default | < ^ + | FAIL | | Codex CLI | < ^ | FAIL | | Haiku 4.5 | <.+? | FAIL | | Sonnet 5 | < ^ + | FAIL | | Opus 5 | < ^ | FAIL | Three distinct fixes. One identical wrong answer: php strip tags "a < b and c d" - "a d" all five None of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given, and all five satisfied it exactly. So I ran two more experiments to find out whether the model or the spec was the binding constraint. Is it reproducible, or did I get unlucky? Same task, same weak checker, same prompt, five more independent runs: run visible hidden pattern 1 PASS FAIL < ^ + 2 PASS FAIL < ^ 3 PASS FAIL < ^ 4 PASS FAIL < ^ 5 PASS FAIL < ^ Ten runs on this task tonight across two labs and three model tiers. Ten hidden failures. Not variance — a fixed point. What happens if I fix the spec instead of the model? I handed the agent the hidden checker as its visible one. Nothing else changed: same default model, same prompt, same one attempt. All five tasks passed, t4 included. It wrote: re.sub r"</? A-Za-z ? ^ ", "", html and documented why, unprompted: "A < not followed by a name character as in a < b is literal text and is left alone." Ten runs could not buy that fix. One better test did. The test suite is the spec, and the agent optimises for exactly what you wrote down. Reproduce all of it with python3 scripts/verify.py — no agents run, nothing is billed. Wall time: quote a range, not a median. Same five tasks, one run each, timed end to end including CLI startup. | | median | range | spread | |---|---|---|---| | Claude Code | 21.3s | 20.4–25.0s | 1.2x | | Codex CLI | 29.6s | 23.6–58.1s | 2.5x | The 58s outlier was the easiest task in the set. With n=5 that is an anecdote about variance, not a property of either tool — which is the point: the median hid it completely. That spread is across tasks. Repeating one task five times, run-to-run variance was almost nil: wall time within 3%, cost within 1%, output tokens within 9% results/repeat.csv . The unpredictability lives in which task you hand the agent, not in rerunning the same one. The patches converged. Two of five came back byte-identical between the two agents. On money rounding both independently reached for the same non-obvious Decimal str x .quantize ..., ROUND HALF UP . For bugs this standard, model choice looks like it matters less than the arguing suggests. See patches/ . Output compression, measured. The caveman https://github.com/JuliusBrussee/caveman skill advertises a 65% token cut. On these five agentic-coding tasks I measured 21.7% 8,299 → 6,494 output tokens , at no cost to quality — 5/5 either way, and the same 4/5 on the hidden tests, failing the same case. Not a false claim; a different workload. It compresses prose, and in agentic coding most output tokens are tool calls and code, which it deliberately leaves alone. results/caveman.csv . Same five bugs, same prompt, three Claude model tiers, one run each. | tier | visible | hidden | cost | output tokens | median time | |---|---|---|---|---|---| | Haiku 4.5 | 5/5 | 4/5 | $0.28 | 11,689 | 22.9s | | Sonnet 5 | 5/5 | 4/5 | $0.61 | 5,812 | 20.4s | | Opus 5 | 5/5 | 4/5 | $1.53 | 5,369 | 17.6s | Identical scores on both test sets, for 5.4x the cost. Two things surprised me: - The cheap tier was the slow one. Haiku took 1.37x the total wall time of Opus, because it wrote 2.2x more output tokens to get to the same patch. Cheaper per token is not the same as cheaper per task, and it is not faster. - The overhead dwarfs the prompt. Across all 15 runs I typed 328 tokens of prompt. The runs read 3,216,012 tokens of cached context to answer them — roughly 9,800 read for every 1 written. The cheapest possible run in the whole set still cost $0.04. You are not paying for your question. On this workload the tier decision is a latency decision, not a quality one. Five bugs is nowhere near enough to generalise that — but it is enough to stop assuming the expensive tier is automatically the right default. results/models.csv . A reader @bnjmn marie https://x.com/bnjmn marie reported that on DeepSWE, two runs of a quantized model scored an identical 31.86% while disagreeing on 42 of 113 tasks. This is the smallest version of that experiment I could run. Same model Qwen2.5-Coder 7B , same prompt, two quantizations, 8 runs per task, 80 runs total. Single-turn code generation, not agentic — a 7B model cannot drive the tool loop the other agents used. | | visible | hidden | distinct answers | median time | |---|---|---|---|---| | Q4 K M | 24/40 | 16/40 | 16 | 2.3s | | Q8 0 | 24/40 | 16/40 | 17 | 5.2s | The scores are identical. The answers are not. On 2 of the 5 tasks the two quantizations produced completely different answer sets. Q8 took 2.3x longer to score the same. Distinct-answer count tracks difficulty better than the score does: t2 1 answer 8/8 pass the model knows this one t1 2 answers 8/8 pass t4 3 answers 8/8 pass ← and 0/16 on the hidden test t3 4 answers 0/8 pass t5 7 answers 0/8 pass the model is guessing t4 is the dangerous row: low spread, perfect score, uniformly wrong. The local model converged on < ^ 8 runs , < ^ + 5 and <. ? 3 . Two of those are exactly what the frontier agents wrote; the third differs from Haiku's <.+? by one character, which is the kind of difference this whole repo exists to notice. Counting last night's runs, 26 independent attempts spanning a 4-bit quantized 7B model and five frontier agents produced four patterns and zero hidden-test passes. All four collapse a < b and c d to a d . On t3 the model either left the bug untouched 10 of 16 runs returned round amount, 2 unchanged or invented API that does not exist round amount, 2, rounding method=ROUND HALF EVEN . Both failure modes show up as spread before they show up as a score. results/quantization.csv , patches/quant/ one file per distinct answer , scripts/run quant.py . The five bugs matter less than the shape. Pick five mistakes you already understand, write the obvious checker, then write the mean one afterwards. php scripts/run agents.sh Claude Code vs Codex CLI - results/tools.csv scripts/run models.sh haiku vs sonnet vs opus - results/models.csv scripts/run caveman.sh output-compression A/B - results/caveman.csv scripts/run spec.sh hidden tests handed over as the spec - results/spec.csv scripts/run repeat.sh one task, five times - results/repeat.csv scripts/run quant.py local model, two quantizations - results/quantization.csv scripts/verify.py replay archived patches, no agents run The runner scripts invoke real agents and consume real quota; verify.py does not. tasks/ five buggy files, each with the checker the agent is given hidden/ the checkers no agent ever saw patches/ what each agent actually wrote, including all five t4 attempts results/ raw csv: timings, token counts, cost, pass/fail on both test sets scripts/ the runners, and verify.py to replay the archive docs/ METHOD.md — how the runs were made and what they cannot tell you CONVERGENCE.md — the 26-run result in full n = 5, one machine, one attempt per cell, textbook bugs already localised to a single function. Sampling is non-deterministic; close cells are ties. This cannot be extrapolated to large refactors or unfamiliar codebases, and it never tests the hard part of real debugging, which is finding the bug. Full caveats in docs/METHOD.md https://github.com/vyang472/five-bugs/blob/main/docs/METHOD.md . MIT licensed. Built by @vyang472 https://x.com/vyang472 .