AIArticle A solo developer's $2,430 refactor of 189 files probes agentic coding's blind spot: changes with no test oracle.
Mariana Souza The hardest problem in agentic coding right now isn't getting a model to write code. It's knowing when the code is right without a human reading it. The industry's emerging answer — what Addy Osmani and others have been calling loop engineering — leans on oracles: a test suite, a compiler, a second agent with fresh context. That works beautifully when an oracle exists. The Bun rewrite, 535k lines of Zig ported to Rust by a fleet of Claude agents in 11 days, was anchored to a million-assertion test suite that predated the port. The tests were the ground truth; the agents just had to satisfy them.
A new arXiv case study goes after the case that framework can't touch: a change whose target behavior has never existed, so no test suite can encode it. And the author's answer — manufacture the oracle as a frozen specification before generating a line of code — is genuinely the most interesting protocol document I've read this year, even though the evidence behind it should be handled with tongs.
What actually happened #
Joël Abenhaïm, a solo developer in Paris, runs a 717,725-line proprietary TypeScript codebase: a VS Code extension called AICode that is itself an AI coding agent. He wanted to dismantle one of its founding invariants — that a UI panel stays open for the lifetime of an AI request — so a streaming generation could survive its panel closing and reattach on reopen with no token loss or duplication. That guarantee was load-bearing across request lifetime, memory ownership, cache restoration, and async event ordering. His assessment: infeasible as an incremental refactor, the kind of change that normally means rewriting the subsystem.
Instead, the agent (AICode running what the paper identifies as OpenAI's ChatGPT 5.6 Sol in max reasoning mode) worked a five-phase protocol: ideate, specify, refine, code, verify. The refine phase re-audited the draft specification against the real source code in fresh sessions — fourteen cycles, roughly 85 corrections, scope growing from 110 to 160 files as unnoticed dependencies surfaced. Once a cycle returned zero findings, the spec was frozen. Code was then generated against it and audited against it — seventeen verification cycles, 116 more corrections — until two consecutive passes found nothing. Only then did a human run the program for the first time.
Total: 189 files touched, 34,770 insertions across two commits, three days, $2,430 in inference. The pre-existing unit test suite passed, and across roughly thirty subsequent sessions no bug has been observed. The full session logs — 1,500-plus pages, in French — are published alongside the paper.
The idea worth taking seriously #
The insight here isn't "big refactor, no review." It's where the referent lives. Standard second-agent review, the kind Anthropic's own loop engineering guidance recommends, separates writer from checker by identity: a fresh agent audits the diff. Abenhaïm separates them by referent instead: the checker is the same agent, but it's comparing code against a document that was hammered into shape before the code existed and then frozen. That sidesteps the well-documented failure mode where a model revising its own output with nothing external to compare against doesn't improve — and it produces something a plain review loop never does: a durable, human-legible artifact of intent that outlives the diff.
There's a real economic argument buried in the cycle counts, too. A defect caught during spec refinement costs a paragraph edit. The same defect caught after generation costs a web of interdependent code changes across dozens of files. Fourteen cheap cycles before generation, seventeen expensive ones after — the protocol's whole bet is shifting defects leftward into the cheap column. Given that review time is the documented bottleneck of AI-assisted development (one 2026 telemetry study the paper cites found review times up as much as 91% while delivery metrics stayed flat), relocating quality control upstream of generation is at least aimed at the right problem.
Now read the caveats like your job depends on it #
Because they're structural. This is one task, one codebase, one operator — and that operator designed the agent, sells it, and wrote the paper. The competing interest is declared, but declared bias is still bias. The codebase is closed-source, so nobody can replay the operation. And "no bug observed" means first manual execution plus about thirty usage sessions plus a passing unit suite — a real signal, not a proof.
My sharper worry is one the paper doesn't fully confront: the convergence criterion is a fixed point of the model, not of reality. "Two consecutive verification passes with zero findings" can mean the code stopped having defects — or that the auditor ran out of ability to see them. Those 201 pre-execution corrections were all found by the same model that wrote the spec and the code. Without an independent oracle, convergence measures self-consistency. The published logs and the downloadable build (the behavior change shipped in v2.3.0 and any user can poke at it) are honest mitigations, and better transparency than most vendor case studies bother with. But the clearest next step is the one the author himself names: an independent operator running the protocol on a public codebase. Until that happens, treat the numbers as one data point from an interested party.
What you can steal today #
You don't need AICode to run this play; any agent harness with fresh-session support can. The mechanics: have the agent draft a formal spec from your natural-language intent. Then, in separate sessions, repeatedly audit that spec against the actual source until an audit returns nothing — expect the scope to grow, which is the process working. Freeze the document. Generate against it atomically. Then audit the code against the frozen spec in fresh sessions until you get two clean passes in a row. Keep compiler and test failures visible rather than letting the agent silently self-correct.
When is 31 audit passes worth it? Not for routine features — for those, tests-as-oracle loops are cheaper and stronger. This protocol earns its cost precisely where oracles can't exist: entangled, cross-cutting invariant changes where the new behavior has no precedent in the codebase. That's a narrow slice of engineering work, but it's the slice that currently makes teams choose rewrites over refactors. A $2,430, three-day alternative to a subsystem rewrite is a trade plenty of teams would take — once someone other than the vendor shows it converging on code we can all read.
Sources & further reading #
[AICode - AI Sovereign Labs](https://ai-code.ai)— ai-code.ai -
[Zig creator calls Bun's Claude Rust rewrite 'unreviewed slop'](https://www.theregister.com/devops/2026/07/14/zig-creator-calls-buns-claude-rust-rewrite-unreviewed-slop/5270743)— theregister.com -
[Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust)— bun.com
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.