{"slug": "one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken", "title": "One seeded bug, 26 AI agents: all passed the tests, all stayed broken", "summary": "A developer-run experiment found that 26 AI agents — spanning a 4-bit quantized 7B model up to Opus 5 — all passed the visible test suite for a seeded greedy-regex bug while all 26 remained broken, with ten runs across two labs and three model tiers producing ten hidden failures. The author reports that handing one agent the hidden checker as its visible spec fixed the task on the first try, writing re.sub(r\"</?[A-Za-z!?][^>]*>\", \"\", html), and argues the specification rather than the model was the binding constraint. The full account is published in docs/CONVERGENCE.md in the vyang472/five-bugs repository, which ships five single-file Python bugs with a verifier runnable via python3 scripts/verify.py.", "body_md": "**26 agents, from a 4-bit quantized 7B model up to Opus 5, were given the same\none-line bug and its test suite. All 26 made the tests pass. All 26 stayed\nbroken.**\n\nThey wrote four different regexes to do it:\n\n```\nre.sub(r\"<[^>]*>\", \"\", html)   # 14x  Codex CLI, Opus 5, 4 repeats, 8 local\nre.sub(r\"<[^>]+>\", \"\", html)   #  8x  Claude Code, Sonnet 5, 1 repeat, 5 local\nre.sub(r\"<.*?>\",   \"\", html)   #  3x  local\nre.sub(r\"<.+?>\",   \"\", html)   #  1x  Haiku 4.5\n```\n\nAll four turn the sentence `a < b and c > d` into `a  d`.\n\nNobody was wrong to write them. The test suite they were handed contained only well-formed HTML, so that was the entire specification, and every one of them satisfied it exactly. Scaling the model across two orders of magnitude did not help, because the model was never the binding constraint.\n\n**Handing one agent a better test fixed it on the first try.**\n\nThe full account — what was ruled out, and what it does and does not support —\nis in [docs/CONVERGENCE.md](https://github.com/vyang472/five-bugs/blob/main/docs/CONVERGENCE.md).\n\nVerify it in two seconds, without running an agent or spending anything:\n\n```\ngit clone https://github.com/vyang472/five-bugs\ncd five-bugs\npython3 scripts/verify.py\n```\n\nFive single-file Python bugs. Each ships with the checker the agent is given,\nand a second checker, written afterwards, that **no agent ever sees**. The gap\nbetween those two is the whole experiment.\n\nEverything was run on one laptop: the tasks, the patch every agent actually wrote, raw timings, token counts and costs, and the scripts that produced them. Nothing here asks you to trust a number I typed.\n\nClaude Code and Codex CLI both fixed all five bugs, 5/5 on the checker they were given. It looked like a tie and a clean sweep.\n\nThen I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again.\n\n```\ntask agent    visible   hidden\nt1   claude   PASS      PASS\nt1   codex    PASS      PASS\nt2   claude   PASS      PASS\nt2   codex    PASS      PASS\nt3   claude   PASS      PASS\nt3   codex    PASS      PASS\nt4   claude   PASS      FAIL\nt4   codex    PASS      FAIL\nt5   claude   PASS      PASS\nt5   codex    PASS      PASS\n```\n\nFour of five held. On the fifth, **both agents failed in exactly the same\nway.** The bug was a greedy regex:\n\n```\nre.sub(r\"<.+>\", \"\", html)      # the seeded bug\n```\n\nSo I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and\nOpus 5. Five agent configurations, two labs, three model tiers. They wrote\nthree *different* patterns:\n\n| agent | patch | hidden | \n|---|---|---|\n| Claude Code (default) | `<[^>]+>` | FAIL | \n| Codex CLI | `<[^>]*>` | FAIL | \n| Haiku 4.5 | `<.+?>` | FAIL | \n| Sonnet 5 | `<[^>]+>` | FAIL | \n| Opus 5 | `<[^>]*>` | FAIL | \n\nThree distinct fixes. One identical wrong answer:\n\n``` php\nstrip_tags(\"a < b and c > d\")   # -> \"a  d\"    (all five)\n```\n\nNone of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given, and all five satisfied it exactly.\n\nSo I ran two more experiments to find out whether the model or the spec was the binding constraint.\n\n**Is it reproducible, or did I get unlucky?** Same task, same weak checker,\nsame prompt, five more independent runs:\n\n```\nrun  visible  hidden  pattern\n1    PASS     FAIL    <[^>]+>\n2    PASS     FAIL    <[^>]*>\n3    PASS     FAIL    <[^>]*>\n4    PASS     FAIL    <[^>]*>\n5    PASS     FAIL    <[^>]*>\n```\n\nTen runs on this task tonight across two labs and three model tiers. Ten hidden failures. Not variance — a fixed point.\n\n**What happens if I fix the spec instead of the model?** I handed the agent\nthe hidden checker as its visible one. Nothing else changed: same default\nmodel, same prompt, same one attempt.\n\nAll five tasks passed, t4 included. It wrote:\n\n```\nre.sub(r\"</?[A-Za-z!?][^>]*>\", \"\", html)\n```\n\nand documented why, unprompted: *\"A `<` not followed by a name character\n(as in `a < b`) is literal text and is left alone.\"*\n\nTen runs could not buy that fix. One better test did. **The test suite is\nthe spec, and the agent optimises for exactly what you wrote down.**\n\nReproduce all of it with `python3 scripts/verify.py` — no agents run,\nnothing is billed.\n\n**Wall time: quote a range, not a median.** Same five tasks, one run each,\ntimed end to end including CLI startup.\n\n|  | median | range | spread | \n|---|---|---|---|\n| Claude Code | 21.3s | 20.4–25.0s | 1.2x | \n| Codex CLI | 29.6s | 23.6–58.1s | 2.5x | \n\nThe 58s outlier was the *easiest* task in the set. With n=5 that is an\nanecdote about variance, not a property of either tool — which is the point:\nthe median hid it completely.\n\nThat spread is *across* tasks. Repeating one task five times, run-to-run\nvariance was almost nil: wall time within 3%, cost within 1%, output tokens\nwithin 9% (`results/repeat.csv`). The unpredictability lives in which task\nyou hand the agent, not in rerunning the same one.\n\n**The patches converged.** Two of five came back byte-identical between the\ntwo agents. On money rounding both independently reached for the same\nnon-obvious `Decimal(str(x)).quantize(..., ROUND_HALF_UP)`. For bugs this\nstandard, model choice looks like it matters less than the arguing suggests.\nSee `patches/`.\n\n**Output compression, measured.** The [caveman](https://github.com/JuliusBrussee/caveman)\nskill advertises a 65% token cut. On these five agentic-coding tasks I\nmeasured **21.7%** (8,299 → 6,494 output tokens), at no cost to quality —\n5/5 either way, and the same 4/5 on the hidden tests, failing the same case.\nNot a false claim; a different workload. It compresses prose, and in agentic\ncoding most output tokens are tool calls and code, which it deliberately\nleaves alone. `results/caveman.csv`.\n\nSame five bugs, same prompt, three Claude model tiers, one run each.\n\n| tier | visible | hidden | cost | output tokens | median time | \n|---|---|---|---|---|---|\n| Haiku 4.5 | 5/5 | 4/5 | $0.28 | 11,689 | 22.9s | \n| Sonnet 5 | 5/5 | 4/5 | $0.61 | 5,812 | 20.4s | \n| Opus 5 | 5/5 | 4/5 | $1.53 | 5,369 | 17.6s | \n\nIdentical scores on both test sets, for 5.4x the cost. Two things surprised me:\n\n- **The cheap tier was the slow one.** Haiku took 1.37x the total wall time\nof Opus, because it wrote 2.2x more output tokens to get to the same\npatch. Cheaper per token is not the same as cheaper per task, and it is\nnot faster.\n- **The overhead dwarfs the prompt.** Across all 15 runs I typed 328 tokens\nof prompt. The runs read**3,216,012 tokens** of cached context to answer\nthem — roughly 9,800 read for every 1 written. The cheapest possible run\nin the whole set still cost $0.04. You are not paying for your question.\n\nOn this workload the tier decision is a latency decision, not a quality one.\nFive bugs is nowhere near enough to generalise that — but it is enough to\nstop assuming the expensive tier is automatically the right default.\n`results/models.csv`.\n\nA reader ([@bnjmn_marie](https://x.com/bnjmn_marie)) reported that on DeepSWE,\ntwo runs of a quantized model scored an identical 31.86% while disagreeing on\n42 of 113 tasks. This is the smallest version of that experiment I could run.\n\nSame model (Qwen2.5-Coder 7B), same prompt, two quantizations, 8 runs per task, 80 runs total. Single-turn code generation, not agentic — a 7B model cannot drive the tool loop the other agents used.\n\n|  | visible | hidden | distinct answers | median time | \n|---|---|---|---|---|\n| Q4_K_M | 24/40 | 16/40 | 16 | 2.3s | \n| Q8_0 | 24/40 | 16/40 | 17 | 5.2s | \n\n**The scores are identical. The answers are not.** On 2 of the 5 tasks the two\nquantizations produced completely different answer sets. Q8 took 2.3x longer\nto score the same.\n\n**Distinct-answer count tracks difficulty better than the score does:**\n\n```\nt2  1 answer    8/8 pass     the model knows this one\nt1  2 answers   8/8 pass\nt4  3 answers   8/8 pass     ← and 0/16 on the hidden test\nt3  4 answers   0/8 pass\nt5  7 answers   0/8 pass     the model is guessing\n```\n\nt4 is the dangerous row: **low spread, perfect score, uniformly wrong.** The\nlocal model converged on `<[^>]*>` (8 runs), `<[^>]+>` (5) and `<.*?>` (3).\nTwo of those are exactly what the frontier agents wrote; the third differs\nfrom Haiku's `<.+?>` by one character, which is the kind of difference this\nwhole repo exists to notice.\n\nCounting last night's runs, **26 independent attempts spanning a 4-bit\nquantized 7B model and five frontier agents produced four patterns and zero\nhidden-test passes.** All four collapse `a < b and c > d` to `a  d`.\n\nOn t3 the model either left the bug untouched (10 of 16 runs returned\n`round(amount, 2)` unchanged) or invented API that does not exist\n(`round(amount, 2, rounding_method=ROUND_HALF_EVEN)`). Both failure modes show\nup as spread before they show up as a score.\n\n`results/quantization.csv`, `patches/quant/` (one file per distinct answer),\n`scripts/run_quant.py`.\n\nThe five bugs matter less than the shape. Pick five mistakes *you* already\nunderstand, write the obvious checker, then write the mean one afterwards.\n\n``` php\nscripts/run_agents.sh     # Claude Code vs Codex CLI      -> results/tools.csv\nscripts/run_models.sh     # haiku vs sonnet vs opus       -> results/models.csv\nscripts/run_caveman.sh    # output-compression A/B        -> results/caveman.csv\nscripts/run_spec.sh       # hidden tests handed over as the spec -> results/spec.csv\nscripts/run_repeat.sh     # one task, five times          -> results/repeat.csv\nscripts/run_quant.py      # local model, two quantizations -> results/quantization.csv\nscripts/verify.py         # replay archived patches, no agents run\n```\n\nThe runner scripts invoke real agents and consume real quota; `verify.py`\ndoes not.\n\n```\ntasks/      five buggy files, each with the checker the agent is given\nhidden/     the checkers no agent ever saw\npatches/    what each agent actually wrote, including all five t4 attempts\nresults/    raw csv: timings, token counts, cost, pass/fail on both test sets\nscripts/    the runners, and verify.py to replay the archive\ndocs/       METHOD.md — how the runs were made and what they cannot tell you\n            CONVERGENCE.md — the 26-run result in full\n```\n\nn = 5, one machine, one attempt per cell, textbook bugs already localised to\na single function. Sampling is non-deterministic; close cells are ties. This\ncannot be extrapolated to large refactors or unfamiliar codebases, and it\nnever tests the hard part of real debugging, which is finding the bug.\nFull caveats in [docs/METHOD.md](https://github.com/vyang472/five-bugs/blob/main/docs/METHOD.md).\n\nMIT licensed. Built by [@vyang472](https://x.com/vyang472).", "url": "https://wpnews.pro/news/one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken", "canonical_source": "https://github.com/vyang472/five-bugs", "published_at": "2026-09-16 01:48:53+00:00", "updated_at": "2026-09-16 02:07:35.185130+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools", "ai-safety"], "entities": ["Claude Code", "Codex CLI", "Claude Haiku 4.5", "Claude Sonnet 5", "Claude Opus 5", "vyang472/five-bugs"], "alternates": {"html": "https://wpnews.pro/news/one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken", "markdown": "https://wpnews.pro/news/one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken.md", "text": "https://wpnews.pro/news/one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken.txt", "jsonld": "https://wpnews.pro/news/one-seeded-bug-26-ai-agents-all-passed-the-tests-all-stayed-broken.jsonld"}}