26 agents, from a 4-bit quantized 7B model up to Opus 5, were given the same one-line bug and its test suite. All 26 made the tests pass. All 26 stayed broken.
They wrote four different regexes to do it:
re.sub(r"<[^>]*>", "", html) # 14x Codex CLI, Opus 5, 4 repeats, 8 local
re.sub(r"<[^>]+>", "", html) # 8x Claude Code, Sonnet 5, 1 repeat, 5 local
re.sub(r"<.*?>", "", html) # 3x local
re.sub(r"<.+?>", "", html) # 1x Haiku 4.5
All four turn the sentence a < b and c > d into a d.
Nobody was wrong to write them. The test suite they were handed contained only well-formed HTML, so that was the entire specification, and every one of them satisfied it exactly. Scaling the model across two orders of magnitude did not help, because the model was never the binding constraint.
Handing one agent a better test fixed it on the first try.
The full account — what was ruled out, and what it does and does not support — is in docs/CONVERGENCE.md.
Verify it in two seconds, without running an agent or spending anything:
git clone https://github.com/vyang472/five-bugs
cd five-bugs
python3 scripts/verify.py
Five single-file Python bugs. Each ships with the checker the agent is given, and a second checker, written afterwards, that no agent ever sees. The gap between those two is the whole experiment.
Everything was run on one laptop: the tasks, the patch every agent actually wrote, raw timings, token counts and costs, and the scripts that produced them. Nothing here asks you to trust a number I typed.
Claude Code and Codex CLI both fixed all five bugs, 5/5 on the checker they were given. It looked like a tie and a clean sweep.
Then I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again.
task agent visible hidden
t1 claude PASS PASS
t1 codex PASS PASS
t2 claude PASS PASS
t2 codex PASS PASS
t3 claude PASS PASS
t3 codex PASS PASS
t4 claude PASS FAIL
t4 codex PASS FAIL
t5 claude PASS PASS
t5 codex PASS PASS
Four of five held. On the fifth, both agents failed in exactly the same way. The bug was a greedy regex:
re.sub(r"<.+>", "", html) # the seeded bug
So I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and Opus 5. Five agent configurations, two labs, three model tiers. They wrote three different patterns:
| agent | patch | hidden |
|---|---|---|
| Claude Code (default) | <[^>]+> |
FAIL |
| Codex CLI | <[^>]*> |
FAIL |
| Haiku 4.5 | <.+?> |
FAIL |
| Sonnet 5 | <[^>]+> |
FAIL |
| Opus 5 | <[^>]*> |
FAIL |
Three distinct fixes. One identical wrong answer:
strip_tags("a < b and c > d") # -> "a d" (all five)
None of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given, and all five satisfied it exactly.
So I ran two more experiments to find out whether the model or the spec was the binding constraint.
Is it reproducible, or did I get unlucky? Same task, same weak checker, same prompt, five more independent runs:
run visible hidden pattern
1 PASS FAIL <[^>]+>
2 PASS FAIL <[^>]*>
3 PASS FAIL <[^>]*>
4 PASS FAIL <[^>]*>
5 PASS FAIL <[^>]*>
Ten runs on this task tonight across two labs and three model tiers. Ten hidden failures. Not variance — a fixed point.
What happens if I fix the spec instead of the model? I handed the agent the hidden checker as its visible one. Nothing else changed: same default model, same prompt, same one attempt.
All five tasks passed, t4 included. It wrote:
re.sub(r"</?[A-Za-z!?][^>]*>", "", html)
and documented why, unprompted: "A < not followed by a name character
(as in a < b) is literal text and is left alone."
Ten runs could not buy that fix. One better test did. The test suite is the spec, and the agent optimises for exactly what you wrote down.
Reproduce all of it with python3 scripts/verify.py — no agents run,
nothing is billed.
Wall time: quote a range, not a median. Same five tasks, one run each, timed end to end including CLI startup.
| median | range | spread | |
|---|---|---|---|
| Claude Code | 21.3s | 20.4–25.0s | 1.2x |
| Codex CLI | 29.6s | 23.6–58.1s | 2.5x |
The 58s outlier was the easiest task in the set. With n=5 that is an anecdote about variance, not a property of either tool — which is the point: the median hid it completely.
That spread is across tasks. Repeating one task five times, run-to-run
variance was almost nil: wall time within 3%, cost within 1%, output tokens
within 9% (results/repeat.csv). The unpredictability lives in which task
you hand the agent, not in rerunning the same one.
The patches converged. Two of five came back byte-identical between the
two agents. On money rounding both independently reached for the same
non-obvious Decimal(str(x)).quantize(..., ROUND_HALF_UP). For bugs this
standard, model choice looks like it matters less than the arguing suggests.
See patches/.
Output compression, measured. The caveman
skill advertises a 65% token cut. On these five agentic-coding tasks I
measured 21.7% (8,299 → 6,494 output tokens), at no cost to quality —
5/5 either way, and the same 4/5 on the hidden tests, failing the same case.
Not a false claim; a different workload. It compresses prose, and in agentic
coding most output tokens are tool calls and code, which it deliberately
leaves alone. results/caveman.csv.
Same five bugs, same prompt, three Claude model tiers, one run each.
| tier | visible | hidden | cost | output tokens | median time |
|---|---|---|---|---|---|
| Haiku 4.5 | 5/5 | 4/5 | $0.28 | 11,689 | 22.9s |
| Sonnet 5 | 5/5 | 4/5 | $0.61 | 5,812 | 20.4s |
| Opus 5 | 5/5 | 4/5 | $1.53 | 5,369 | 17.6s |
Identical scores on both test sets, for 5.4x the cost. Two things surprised me:
- The cheap tier was the slow one. Haiku took 1.37x the total wall time of Opus, because it wrote 2.2x more output tokens to get to the same patch. Cheaper per token is not the same as cheaper per task, and it is not faster.
- The overhead dwarfs the prompt. Across all 15 runs I typed 328 tokens of prompt. The runs read3,216,012 tokens of cached context to answer them — roughly 9,800 read for every 1 written. The cheapest possible run in the whole set still cost $0.04. You are not paying for your question.
On this workload the tier decision is a latency decision, not a quality one.
Five bugs is nowhere near enough to generalise that — but it is enough to
stop assuming the expensive tier is automatically the right default.
results/models.csv.
A reader (@bnjmn_marie) reported that on DeepSWE, two runs of a quantized model scored an identical 31.86% while disagreeing on 42 of 113 tasks. This is the smallest version of that experiment I could run.
Same model (Qwen2.5-Coder 7B), same prompt, two quantizations, 8 runs per task, 80 runs total. Single-turn code generation, not agentic — a 7B model cannot drive the tool loop the other agents used.
| visible | hidden | distinct answers | median time | |
|---|---|---|---|---|
| Q4_K_M | 24/40 | 16/40 | 16 | 2.3s |
| Q8_0 | 24/40 | 16/40 | 17 | 5.2s |
The scores are identical. The answers are not. On 2 of the 5 tasks the two quantizations produced completely different answer sets. Q8 took 2.3x longer to score the same.
Distinct-answer count tracks difficulty better than the score does:
t2 1 answer 8/8 pass the model knows this one
t1 2 answers 8/8 pass
t4 3 answers 8/8 pass ← and 0/16 on the hidden test
t3 4 answers 0/8 pass
t5 7 answers 0/8 pass the model is guessing
t4 is the dangerous row: low spread, perfect score, uniformly wrong. The
local model converged on <[^>]*> (8 runs), <[^>]+> (5) and <.*?> (3).
Two of those are exactly what the frontier agents wrote; the third differs
from Haiku's <.+?> by one character, which is the kind of difference this
whole repo exists to notice.
Counting last night's runs, 26 independent attempts spanning a 4-bit
quantized 7B model and five frontier agents produced four patterns and zero
hidden-test passes. All four collapse a < b and c > d to a d.
On t3 the model either left the bug untouched (10 of 16 runs returned
round(amount, 2) unchanged) or invented API that does not exist
(round(amount, 2, rounding_method=ROUND_HALF_EVEN)). Both failure modes show
up as spread before they show up as a score.
results/quantization.csv, patches/quant/ (one file per distinct answer),
scripts/run_quant.py.
The five bugs matter less than the shape. Pick five mistakes you already understand, write the obvious checker, then write the mean one afterwards.
scripts/run_agents.sh # Claude Code vs Codex CLI -> results/tools.csv
scripts/run_models.sh # haiku vs sonnet vs opus -> results/models.csv
scripts/run_caveman.sh # output-compression A/B -> results/caveman.csv
scripts/run_spec.sh # hidden tests handed over as the spec -> results/spec.csv
scripts/run_repeat.sh # one task, five times -> results/repeat.csv
scripts/run_quant.py # local model, two quantizations -> results/quantization.csv
scripts/verify.py # replay archived patches, no agents run
The runner scripts invoke real agents and consume real quota; verify.py
does not.
tasks/ five buggy files, each with the checker the agent is given
hidden/ the checkers no agent ever saw
patches/ what each agent actually wrote, including all five t4 attempts
results/ raw csv: timings, token counts, cost, pass/fail on both test sets
scripts/ the runners, and verify.py to replay the archive
docs/ METHOD.md — how the runs were made and what they cannot tell you
CONVERGENCE.md — the 26-run result in full
n = 5, one machine, one attempt per cell, textbook bugs already localised to a single function. Sampling is non-deterministic; close cells are ties. This cannot be extrapolated to large refactors or unfamiliar codebases, and it never tests the hard part of real debugging, which is finding the bug. Full caveats in docs/METHOD.md.
MIT licensed. Built by @vyang472.