A small model with a review loop matches a larger one at 58% of the cost

A build-and-check review loop using the small GPT-5.6 Luna model matched the single-shot score of the larger GPT-5.6 Sol on 20 ProgramBench tasks at a median 58% of Sol's cost, and at Sol's cost the loop scored higher on 15 of the 20 tasks, according to results published in October 2026. The loop ran as a two-node zeroshot graph with a builder and a fresh-session checker; a single Luna session told to play both roles stopped after one pass and scored no better than a plain single shot, and the loop never produced a badly failed run while every single-shot arm did. Luna costs $0.20 per million input tokens and $1.20 per million output tokens versus Sol's $4 and $20, and both models ran through OpenAI's Codex CLI at xhigh reasoning effort, with Sol run five times per task as a single shot and Luna's loop run five times per task.

October 2026 A small model with a review loop matches a larger one at 58% of the cost. On 20 ProgramBench tasks, GPT-5.6 Luna in a simple build-and-check loop reached the score of GPT-5.6 Sol's single shot for a median 58% of what Sol's run cost, and at Sol's cost it scored higher on 15 of the 20. A single Luna session told to play both roles stopped after one pass and scored no better than a plain single shot, and the loop never produced a badly failed run, where every single-shot arm did. What the loop is for Coding models are trained to finish fast and sound right, which makes them confident by default. Ask one to build a program and it comes back with something that compiles and a note saying it's done, from the same model that did the work. In the runs below, 100 Luna sessions that had been told to check their own work returned after 4 to 20 minutes with about half of the hidden tests passing. Every one of them stopped on its own, well short of the six hours it had. A review loop splits the two jobs. A builder works until it believes the task is done; then a checker, in a fresh session that hasn't seen the builder's reasoning, tries to break the result, and if it finds problems the builder gets the list and goes again. The checker is the same model with the same task statement; it just starts without the builder's transcript. ProgramBench is a good place to measure whether that helps. Each task gives an agent a compiled program svgbob, xz, samtools, direnv, 16 more and its documentation, offline, and asks for a reimplementation from observed behaviour alone. Hidden tests then score the rebuild: 262 to 1,425 of them per task here. It's a fair stand-in for everyday work, and on our 20 tasks no published run passes every test; the best passes between 38% and 99%. What we ran We ran the loop as a zeroshot graph with two nodes, a builder and a checker; zeroshot is our open-source runner for graphs of coding agents, and this is about the simplest graph it can run. One builder turn is a build, one checker turn a check, and the pair is a round. The builder keeps its session across rounds; every check starts fresh with only the task statement and the workspace. A rejection sends the checker's list of discrepancies back as the builder's feedback, and the loop ends when the checker accepts, after 50 rounds, or at six hours. We wrote two prompts and froze them before the first run; neither was tuned to any task. Everything else the model sees is ProgramBench's own task statement and the runner's standard instructions for a builder and a checker. builder Complete the task described in the input. Keep working and verifying your own work until you believe the result fully satisfies the task. If review feedback is provided, address every point without breaking what already works. checker Independently judge whether the current workspace satisfies the task. Derive your own checks from the task statement and whatever it makes available; do not rely on the builder's tests, notes, or claims. Run the checks. Accept only with concrete evidence that nothing is wrong. Otherwise reject and list the most important discrepancies, each with a minimal reproduction. Luna and Sol are two models in the GPT-5.6 line, Luna the small one and Sol the large one; we ran both through OpenAI's Codex CLI at its xhigh reasoning-effort setting. Luna costs $0.20 per million input tokens and $1.20 per million output tokens, Sol $4 and $20. Sol ran each task as a single shot one build, no checker five times. Luna's loop ran five times per task. 1.The loop reaches Sol's score at 58% of Sol's cost Why this comparison. If you're paying for the work, the question isn't which model scores higher but what a given level of quality costs. Luna costs about a twentieth of Sol per token, and the loop spends tokens on checks and repairs that a single shot never makes, so a loop's bill depends on how many rounds it takes. We asked, per task, what the loop had to spend to reach the score Sol's single shot gets there. The loop's five runs are scored build by build; the cost to match is the smallest budget at which the mean of their last builds within that budget reaches Sol's mean score, and we report it as a share of what Sol's single shot cost on the same task. What we expected. If the checker catches bugs the builder missed, the loop should reach Sol's score before it has spent what Sol spends, with a wide margin on tasks where a cheap first build already lands close. If the small model can't do a task, no budget gets there. We expected the loop to come in cheaper on most tasks, not all: where a cheap first build lands far from the mark, 50 rounds of a small model may never close the gap. What we found. On 15 of the 20 tasks the loop reached Sol's mean score for less than Sol's single shot cost, and across the 20 tasks the median is 58% of Sol's spend.<sup 1 n1 </sup On 3 tasks it needed more than Sol spent 130% to 193% , and on 2 datasurgeon, xz it hadn't reached Sol's score by its last build. The figure below shows each task. Fixing the cost instead of the score gives the same picture. With each loop scored at the last build that fits inside Sol's mean cost on the task, 11 to 50 rounds in, the loop scored higher on 15 of the 20 tasks.<sup 2 n2 </sup On 7 of them ditaa, parqeye, dstask, direnv, calcurse, gittype, gdu the 95% intervals of the two means don't overlap; on the 5 tasks Sol leads, they do.<sup 3 n3 </sup Averaged over the tasks, the loop passes 69.5% of the hidden tests to Sol's 65.6%. What the loop does pay is wall-clock time: a median 69 minutes to reach Sol's score and 2.3 hours to spend Sol's budget, against 11 minutes for Sol's single shot medians of the per-task averages . 2.A single session given both roles stops after one build Why this comparison. The cheapest explanation of the loop's lead is that the checker's instructions are good, and that any session told to build and then check would do as well. If that were true, the separate checker session would add little beyond the prompt. What we expected. Two outcomes seemed possible. Either a single session given both roles keeps going for about as long as the loop does and lands near the same score, or it judges its own work fine and stops early. We leaned toward the second: the builder prompt already tells the session to verify its own work before it stops, and a second instruction to do the same thing seemed unlikely to change when it stops. What we ran. We gave Luna the builder's and the checker's text above, verbatim, plus the runner's standard checker instructions and four sentences of glue; we call the whole thing the role prompt: glue Work in rounds. In each round, act first as the builder and then as the checker, following the instructions for each role below. If the checker rejects, its list of discrepancies is the review feedback for the builder in the next round. Repeat until the checker accepts, then return. The model, the task statement, the six-hour limit and the five runs per task were the loop's; the only change is one session instead of two. What we found. Luna didn't loop. All 100 sessions returned on their own after 4 to 20 minutes median 9 , at about the price of one build. They scored 50.5% on average, 2 points below Luna's plain single shot the loop's first build . At that spend the loop itself has finished exactly one build in all but one of the runs in the comparison,<sup 4 n4 </sup so this is a comparison of two prompts for one build, and the plain one wins it 14 tasks to 6. Everything the loop gains over a single shot comes from the extra rounds: 69.5% at Sol's budget, 71.1% after all its rounds. 5 n5 3.The loop is more reliable Why this comparison. If you ship agent-built software, the worst run matters as much as the average one. A single shot is one attempt, and attempts vary: the model commits to a design in its first minutes and may misread a flag along the way. A checker should catch the worst of that. What we expected. The loop's runs should land closer to each other than any single shot's, and Sol's single shots, from the stronger model, should vary less than Luna's. What we found. The figure below pools the 20 tasks, each run plotted as its score minus its arm's mean on that task; an arm is one of the four setups Sol's single shot, Luna's single shot, the role-prompt sessions, the loop at Sol's budget . Scored at Sol's budget, 86% of the loop's runs land within 5 points of their task's mean; so do 72% of Sol's single shots, 65% of Luna's single shots and 62% of the role-prompt sessions. 6 n6 The loop's lead over Sol is in the tails. Four of Sol's 100 runs ended more than 15 points from their task's mean; the loop had one such run in 100, on the high side.<sup 7 n7 </sup Leave those out and the two arms are equally steady: 85% of Sol's remaining runs land within 5 points, 87% of the loop's. What the loop never did was fail badly: no loop run fell more than 15 points below its task's mean, where Sol's single shot had three such runs, Luna's two and the role prompt's two. Conclusion The loop we measured is the simplest thing that can be called a review loop: two nodes, two generic prompts that never changed, and a checker whose only advantage over the builder is a fresh start. On these tasks that was enough for a small model to reach a large model's score at a median 58% of its cost, and to do it without the badly failed runs that every single-shot arm produced. The trade is wall-clock time: reaching that score took the loop about six times as long as Sol's single shot, which matters for some uses and not for others. A prompt asking one session to do both jobs didn't produce the same effect, because the session stopped when it believed it was done. That is a core hypothesis zeroshot is built on: an independent check catches what a builder's own assessment misses. The two-node loop is its smallest instance. A zeroshot graph can carry more than that, such as several builders working in parallel, checkers that specialize one runs the tests, one reads the diff, one compares against the spec , verifiers with their own tools, and prompts written for the job at hand. We expect graphs built for a task to bring the cost-to-match number well below what a generic two-node loop gets. Limitations - 20 tasks, not the whole benchmark. Five we chose by hand: svgbob and ditaa from a shortlist of programs that are small by code size and rarely solved by published runs, and calcurse, revive and fasttext by a rule at most 800 hidden tests over more than one test branch, Sol's published run between 15% and 80%, the best published run at least 5 points higher, one task per language not yet covered, and within each language the domain least like the two diagram renderers . The other 15 we drew at random, with a recorded seed and stratified by language, from the 72 ProgramBench tasks that met three conditions: Sol's published single run scores between 10% and 90%, the best published run is at least 5 points higher, and the hidden tests number at most 1,500 spread over more than one test branch. - One model family, one harness. Luna and Sol are two models in the GPT-5.6 line, driven by the Codex CLI. - Five runs per arm. The intervals in the figures are 95% bootstrap intervals over those five runs; some of them are fairly wide. Setup and what we changed Code, experiment files, every per-task result and the figure scripts are on the benchmark/programbench https://github.com/the-open-engine/zeroshot/tree/benchmark/programbench/benchmarks/programbench branch of the zeroshot repository. - Harness - zeroshot 10.8.0 driving Codex CLI 0.155.0 inside the official ProgramBench task image, instead of mini-SWE-agent, the scaffold the benchmark ships with. The task statement is the upstream mini-SWE-agent prompt with only its scaffold mechanics removed the persona line, the one-command protocol, the submit sentinel, the command examples ; our unit tests re-derive it byte for byte. - The reference binary - ProgramBench puts the reference at ./executable , which is also the build target. With the task unmodified, builders overwrote the reference, the checkers then had only a README that contradicts the binary in places and enforced it, and the loop gained nothing. So we moved the reference to a root-owned, execute-only path outside the workspace and pointed the task statement there. Every number in this post comes from that setup, both arms alike. The reference stays a black box that can be run but not read, and instrumenting it through the dynamic loader or ptrace disqualifies a run. - Documentation - As shipped, except on svgbob, where we corrected three README statements that contradict the binary, for both arms. - Network - Tool commands have no route out, as under upstream's --network none . Only Codex reaches the model API, through a per-attempt proxy that allows nothing else; web search is off. - Limits - Six hours per attempt, as upstream; three hours per build, one per check; no step limit. - Scoring - ProgramBench's own evaluator on the task image, hidden tests pinned, identical workspaces scored once. Each task's smoke test re-scores a published submission and asks for agreement within 2 points. Four tasks failed that gate crowbook, dust, samtools, tree-sitter and stayed in the study, because the causes touch both arms the same way.<sup 8 n8 </sup We compare arms within this setup and treat leaderboard numbers as context. - Audits - We scan every run's shell commands. Instrumenting the reference, a tool command calling the model API or reaching the network through Codex's proxy, reading process environments or searching the web disqualifies a run. All 300 runs in these numbers passed. We also diff each check against the preceding snapshot to confirm the checker edited nothing. Notes 1. The median is over all 20 tasks, with the two tasks the loop never reaches counted as the most expensive outcome; over the 18 it does reach, it is 49%. For xz we checked the loop's final builds, all below Sol's mean, rather than every intermediate build. ↩ r1 2. Two of Sol's five ditaa runs scored near zero, and under the benchmark's rules they count: javac rejects one run's source because of a non-ASCII character, and the other lost a test branch to the scorer's one-hour limit ProgramBench runs each task's hidden tests in separate groups, its test branches, each under that limit . Without those two runs Sol's ditaa mean is 29%, still 18 points behind the loop, and the loop reaches it for 7% of Sol's cost partly because that mean is so low. ↩ r2 3. The loop's samtools win depends on a Sol run that lost its largest test branch to the scorer's one-hour limit; without that run Sol would lead there by a point. On xz, Sol's mean rests on one strong run among four weak ones. ↩ r3 4. One direnv loop run is left out of this comparison because the scorer counted one extra test on its first build; another direnv run had finished two builds within the budget. ↩ r4 5. 91 of the 100 loop runs ran all 50 rounds; the other 9 stopped between rounds 23 and 44. ↩ r5 6. The labelled outliers in the figure are mostly the runs in notes 2 and 3: Sol's are the two ditaa runs and the samtools run, plus one unusually good xz run, and Luna's single shot has one each on igrep and keifu. The role-prompt sessions' are all on keifu, where three sessions landed at 68 to 77% and two near 30%; the loop's one is datasurgeon, where three runs plateau near 53% and two reach 75 to 82%. ↩ r6 7. Sol's four: the two broken ditaa runs, the stalled samtools run and the very strong xz run. The loop's one: a datasurgeon run 19 points above its task's mean. ↩ r7 8. On crowbook the published run fails tests that pass here; on dust the golden files the expected outputs bake in filesystem block sizes and one test branch's pytest plugin won't load; on tree-sitter 33 tests pass only here; on samtools one test branch stalls past the evaluator's one-hour limit. ↩ r8