Browse sections #
Updated 15 October 2026: Claude Sonnet 5.5 and Opus 5.5 added to the BIRD comparison, and a recall test with a renamed copy of the questions: the models know many of BIRD’s recorded answers but barely rely on them, and the ranking holds. The method table lists what changed.
A finance team wants an agent to answer database questions and edit workbooks. Which version should it deploy? In our BIRD tests, one database answer had the right rows but extra columns, breaking the required format. Another returned no rows because the agent trusted a hint with the wrong letter case. Should the team clarify instructions, let the agent inspect results, or pay for a stronger model?
We tested database workflow changes on BIRD and compared workbook agents on SpreadsheetBench. Public cases make the comparisons inspectable. They can also be learned by heart: on BIRD, the strongest models reproduced most recorded answers without seeing the database, though a renamed copy showed that they barely relied on it. Only the team’s own cases can show its likely repair work and cost.
Dvergr compares complete workflows on the same cases: model, instructions, tools, data route and checker. It records failures, cost and time. Simmis supports reviewing proposed changes to accepted managed state. Connecting benchmark evidence directly to that adoption path remains in progress.
Fix instructions before buying a stronger model #
First, test whether a clearer instruction fixes the output. BIRD gives an agent a database schema, a question and a hint, then asks for one SQL query. We compared its published prompt with the same prompt plus one instruction: return exactly the columns the question asks for. A third workflow let the agent run SQL, inspect results or errors, and retry for up to twenty turns. BIRD graded all three on the same 100 questions:
| Workflow | GPT-5.6 Luna | Claude Haiku 5.5 | Tokens per question (Luna) |
|---|---|---|---|
| BIRD’s prompt, one reply | 51 | 53 | 1,200 |
| BIRD’s prompt plus the column instruction | 61 | 64 | 1,200 |
| Dvergr: run queries, see results and errors, then submit | 65 | 69 | 8,200 |
One instruction raised Luna from 51 to 61 and Haiku from 53 to 64 correct answers, without extra tokens (p = 0.013 and 0.003). On question 781, Luna returned the right heights plus names, so BIRD marked it wrong. The instruction fixed that answer. Extra columns accounted for seven of fifteen cases Luna solved with Dvergr but missed with BIRD’s original prompt. Fixed report columns could spare staff a repair step, though BIRD does not measure staff time.
Question 817 needed a different fix. Its hint used ‘blue’ and ‘male’, but the database stored ‘Blue’ and ‘Male’. A one-reply query found nothing. In Dvergr, Luna saw that empty result, retried without case sensitivity and found three matching heroes. Checking the result recovered an answer that the first query missed.
Retries solved four or five more questions but used roughly seven times the tokens. Dvergr’s workflow already included the column instruction, so those answers isolate what exploration added. The gain remains uncertain on 100 cases (p = 0.34 and 0.30). The improved one-reply workflow cost about one third as much per correct answer as Dvergr’s retry workflow. Whether retries earn their cost depends on the damage an empty answer causes.
Compare candidates without changing the starting line #
Next, candidates need the same starting conditions. Otherwise a score may reflect different cases or data changed by another attempt. Dvergr records those conditions and isolates each comparison:
A Dvergr case pack holds cases, recorded answers and a checker. Certification rejects duplicate ids, conflicting outcomes and answers the checker rejects. It kept 390 of 400 SpreadsheetBench tasks before model runs. Excluding ungradable tasks saves spend and prevents bad cases from shaping the choice.
Runs take place in a Dvergr room holding files, database state and history. Each candidate gets a separate fork for each case. Unchanged state is shared, while workbook and file edits stay inside that attempt. Database reads use a fresh read-only connection or immutable snapshot. The checker records a verdict, then Dvergr discards the fork. Fork overhead takes a fraction of a second; model answers take ten to seventy seconds. One attempt therefore cannot change another’s starting conditions.
Dvergr runs the comparison. Yggdrasil supplies copy-on-write branches across Git and Datahike. Spindel supports forkable execution. Datahike retains queryable history. Together, these layers preserve the starting state and basis for each verdict. Simmis supports proposal review and authorized adoption of supported managed changes. A direct path from benchmark report to that decision is still in progress.
When a model or instruction changes, the recorded setup makes another comparison possible:
- Same cases. Every candidate answers every case, so the team can compare their disagreements.
- Failures distinguished. A provider outage or timeout triggers a rerun; a wrong answer or no answer receives a verdict.
- Comparable price. Costs use public token list prices, including lower cached-input rates, whether a subscription or the organization’s own key paid. They exclude staff repair time, delays and self-hosted hardware.
- New setup, new experiment. Interrupted runs resume, but changing cases, candidates, instructions, checker or software versions starts another experiment.
Check the score before trusting it #
The checker must judge work as staff would use it. Otherwise a score can favour the wrong workflow. SpreadsheetBench exposed that risk with workbook formulas.
SpreadsheetBench shows each sheet’s first five rows and asks for a Python program that edits the workbook. Its multi-round version returns program output or errors for up to five turns. We ran both with GPT-5.6 Luna on 59 tasks. Programs ran in Docker with access limited to their input file; the benchmark graded results. Dvergr’s agent instead read and wrote cells through its sandboxed spreadsheet engine, then inspected computed values.
SpreadsheetBench’s grader reads stored formula values without calculating them. Its instructions say to reopen workbooks in LibreOffice or Excel before grading. We used LibreOffice on Linux, and that choice changed the comparison.
Task 56563 asks for a sum of monthly amounts despite an error in one cell. Both workflows wrote =AGGREGATE(9,6,C2:C13), which skips the error and should produce 77,772.6. Two calculation paths gave that same cell opposite verdicts:
| Graded by | What it found in the cell | Verdict |
|---|---|---|
| LibreOffice, following benchmark instructions | #NAME? : neither workflow wrote the_xlfn.AGGREGATE prefix LibreOffice needs to recognise this newer Excel function |
wrong |
| Dvergr’s spreadsheet engine | 77,772.6; LibreOffice agrees once the prefix is written | right |
Across all 59 tasks, the recalculation choice changed the comparison:
| Workflow | Recalculated by LibreOffice (the benchmark’s instructions) | Tokens per task (median) | Median time |
|---|---|---|---|
| SpreadsheetBench, one reply with code | 34 | 2,200 | 15 s |
| SpreadsheetBench, up to five turns with execution | 41 | 4,300 | 17 s |
| Dvergr: read and write cells, see what they compute, submit | 41 as written, 48 as Excel stores them | 5,700 | 24 s |
Under LibreOffice grading, the multi-round reference and Dvergr each score 41, though they solve eight different tasks. Adding Excel’s function prefix raises Dvergr to 48; its Rechentafel engine scores 51. Rechentafel matches Excel’s saved values on 390 of 400 recorded answers. If staff use Excel, grading with another program could lead the team to buy the wrong workflow. The checker must match staff’s software or use an engine verified against it.
Check the answer key too #
Even a consistent checker can follow a wrong answer key. A team may then blame its model and pay more when the benchmark is wrong. Both case sets contained such problems.
On BIRD, four initial models all missed the same twenty of 100 questions in both SQL and Datalog. For many of those questions, models agreed with each other but differed from the recorded answer:
| Question (BIRD id) | Recorded answer | What the models answered |
|---|---|---|
| atoms of molecule TR346 and its bond types (309) | a query for molecule TR000 | molecule TR346 |
| the German type of a card (482) | the English type | the German type |
| patients with abnormal CRP and no recorded data (1256) | 208, the number of lab rows | 25 patients; the only answer graded right, GPT-6 Luna’s in Datalog, repeated the mistake |
| expenses of the budget with the lowest remaining (1365) | one row, cut off by LIMIT 1 |
both expenses of that budget |
| difference of two percentages (1458) | 12.12 | 0.1212, following the hint’s formula, which has no ×100 |
| eye colours of Marvel heroes by popularity (728) | colour, count and a rank column | colour and count |
We found about ten questions with a wrong recorded answer and four more with a contradictory hint. Other disputes concern requested columns or wording, and a careful model can win those: Claude Opus, measured later, answered seven of the twenty in SQL or Datalog, and all four of its SQL answers held on a renamed copy of the questions, where recall cannot help (see the recall test below). The wrong recorded answers are another matter. On two of them Opus was graded right only for repeating the recorded mistake, in Datalog. On question 185 its Datalog query counted clients by their home district, as the recorded query does, instead of by their accounts’ branch; on 951 it returned the number of races for each constructor instead of the number of constructors. Its SQL answers to both questions did what was asked, and were graded wrong. On this sample, wrong recorded answers cap the attainable score near 90 rather than 100. Paired comparisons remain useful when candidates lose the same flawed cases. Still, flawed cases shrink visible differences and can deny credit for answering the question as asked. Our grader matched BIRD’s official grader on all 200 SQL answers we checked. Every BIRD score here follows BIRD’s rules, including their defects.
On SpreadsheetBench:
- The published evaluation script cannot score the published tasks. On the repository’s main branch it compares each task’sinput workbook with the recorded answer (the line that reads the model’s output is commented out), and its file names are those of an earlier release. Every published score on the 400 verified tasks comes from a modified script that is not part of the benchmark.
- Formulas are graded as empty unless the workbook is recalculated first, as described above. The benchmark’s instructions include that step, but the program that recalculates changes the score, and the step also recalculates the recorded answers, so after it they are that program’s results too.
- One task can never pass. The answer position of task 45944 contains spaces, and the checker fails on it whatever the answer.
- Some recorded answers are wrong or misfiled. Task 118-50’s answer misses a pair its own rule finds in the input, and the two strongest models were graded wrong for finding it. Task 42930’s answer file carries another task’s number.
Certification set aside ten of 400 spreadsheet tasks before model runs. Those exclusions avoid spending tokens on known defects and keep those defects from selecting a workflow.
Check the benchmark runner itself
Our runner also needed correction before its scores could guide a decision. Four AI review agents checked execution, fork isolation, documentation and reported numbers. They found nine problems, including a provider outage counted as a model failure and cached input priced at the full rate. We added failing tests, fixed the problems, then repeated or withdrew affected runs. Otherwise both quality and cost comparisons could have misled a buyer.
Check whether the models have seen the answers #
A public benchmark publishes its questions and its recorded answers. A model trained after publication may have read both. Its score could then measure memory as well as the skill a team would deploy.
BIRD’s development questions and recorded queries have been public since 2023. Claude Opus 5.5 said so itself while it worked: “Known BIRD question; expected answer: SELECT T1.bond_id, T2.atom_id, T2.atom_id2 FROM bond T1 INNER JOIN connected T2 …”, before it had looked at the data. So we tested every model for recall. Each received only a question’s text and the name of its database, with no schema, no hint and no data, and was asked for BIRD’s recorded query. We ran the reply on the database and compared its rows with the recorded answer’s.
We then checked whether the models rely on that memory. We built a renamed copy of the same 100 questions: every database, table and column has a new name, each recorded query was rewritten to the new names and verified to return the same rows, and every question and hint was reworded, the meaning checked by a second model, and three of them by hand. The data and the recorded answers are unchanged, so only recall is taken away:
| Model | Correct without the database (of 100) | Recorded query reproduced, ignoring spacing and case | With the database, BIRD, SQL | With the database, renamed copy, SQL |
|---|---|---|---|---|
| Claude Opus 5.5 | 86 | 68 | 82 | 80 |
| Claude Sonnet 5.5 | 73 | 38 | 75 | 72 |
| GPT-6.1 Sol | 60 | 20 | 74 | 69 |
| GPT-6 Astra | 58 | 18 | 75 | 71 |
| Claude Haiku 5.5 | 58 | 20 | 69 | 67 |
| DeepSeek V4.1 Flash (open weights) | 43 | 9 | 66 | 66 |
| GLM 5.3 Flash (open weights) | 33 | 7 | 64 | 69 |
| GPT-5.6 Luna | 26 | 5 | 65 | 59 |
| GPT-6 Luna | 25 | 3 | 68 | 63 |
Without seeing a single table, Opus produced the recorded answer to 86 questions, more than the 82 it reached with the database, and wrote 68 of the recorded queries word for word, 63 of them character for character. Sonnet, Sol, Astra and Haiku produced 58 to 73. Some correct answers without the database are guesses on well-known schemas; the exact copies are not. On the renamed copy a recalled query no longer runs: without the database, no model answered more than two of the 100. Several still recognised the reworded questions: for 43 of them Opus wrote out BIRD’s original query, with the old names.
With the database, recall barely mattered. On the renamed copy, Opus answered 80 questions instead of 82. The other models answered between six fewer and five more; none of these changes is larger than chance. Seven of the nine scored slightly lower on the copy; that could be a little reliance on memory or rewordings that are slightly harder, and these runs cannot tell which. Of the 86 questions Opus could reproduce from memory, it still answered 73 on the renamed copy, against 75 on BIRD. The models read the data and answer from it, even when they know the recorded query. The ranking held as well: on the renamed copy Opus beat every other model, counted among its eight comparisons, including GPT-6 Astra (nine questions to none) and Claude Sonnet (nine to one). Counted among all 36 pairs of models, its margins over Sonnet, Astra and GLM fall short.
The recall test alone would have led us to the wrong conclusion. It shows what a model can reproduce, not what it relies on; only a control on cases the model cannot have memorised shows that. A team’s own cases have never been published, so they need no such control. For public benchmarks, we will run comparisons on a renamed copy beside the original. We have not yet tested SpreadsheetBench this way.
What does each usable result cost? #
Token price alone misses failures and waiting time. We compared success, list-price cost per correct result and time under the same stated rules.
Each dot represents a model; each bar shows uncertainty from sampled cases, not repeated runs. Around two thirds correct on 100 questions, the interval spans roughly nine points either way. Those wide intervals make a small apparent lead weak evidence for a purchase.
On the same cases, twelve wins against two losses say more than six against five. We report a p-value for how surprising each split would be under equal performance. Values below 0.05 merit attention, though many comparisons increase false positives. Dvergr reports intervals; we calculated paired counts and p-values from recorded verdicts.
Provider load changes, so timing comparisons are strongest when candidates run together.
Spreadsheet work: price varied more than quality
For workbook edits, compare cost per passing task before comparing token prices. We held Dvergr’s workflow and spreadsheet engine constant across 59 of the 390 certified tasks. The public SpreadsheetBench leaderboard uses other tools, so its results are not directly comparable:
| Model | Tasks | Passed | 95 % interval | Cost per pass | Median time |
|---|---|---|---|---|---|
| GPT-5.6 Luna | 59 | 51 | 76–93 % | $0.0026 | 24 s |
| GPT-5.6 Sol | 59 | 52 | 78–95 % | $0.033 | 27 s |
| Claude Opus 5.5 (Claude Code subscription) | 59 | 52 | 78–95 % | $0.062 | 17 s |
| Claude Haiku 5.5 (Claude Code subscription) | 59 | 47 | 68–88 % | $0.0046 | 22 s |
| Claude Sonnet 5.5 (Claude Code subscription) | 59 | 46 | 66–87 % | $0.33 | 19 s |
| GPT-6 Luna | 59 | 48 | 70–90 % | $0.0008 | 10 s |
| GPT-6.1 Sol | 59 | 55 | 85–98 % | $0.010 | 9 s |
| GPT-6 Astra | 59 | 55 | 85–98 % | $0.050 | 9 s |
Against GPT-5.6 Luna, none of the GPT-5.6 or Claude models showed a clear quality gain on these cases. Opus won six cases and lost five; Sol won three and lost two; Haiku won four and lost eight; Sonnet won three and lost nine (the closest comparison, p = 0.15). Cost per passing task differed sharply: Haiku cost roughly twice as much as Luna, Sol thirteen times, Opus twenty-four times and Sonnet more than a hundred times. Sonnet’s slowest tenth of tasks took over nine minutes. That delay matters when staff wait for a workbook before continuing.
GPT-6.1 Sol and GPT-6 Astra each passed 55 tasks. They disagreed on four, split two each way, while Astra cost five times as much per pass. Sol 6.1 beat GPT-6 Luna on eight cases and lost one (p = 0.039); against GPT-5.6 Luna the split was five to one (p = 0.22). The newer models ran about three times faster. They were tested a few days later, so provider conditions may explain part of that difference.
At least one model solved every task but one. Most failures therefore identify candidate weaknesses, not tasks no model could solve.
Database routes: similar answers, different bills
Database access can change an agent’s cost even when accuracy barely moves. BIRD covers 11 databases. We compared SQL on SQLite, SQL on Datahike through pg-datahike, and Datalog on Datahike. Before involving a model, we checked that pg-datahike returned exactly SQLite’s rows for 1372 of BIRD’s 1534 reference queries. That check helps separate engine differences from agent differences.
GPT-5.6 Luna then answered 100 fresh questions held out from tuning:
| Engine | Correct (of 100) | 95 % interval | Cost per correct answer | Input tokens per question | Median / slowest tenth |
|---|---|---|---|---|---|
| SQL on SQLite | 65 | 55–74 % | $0.0023 | 7,839 | 10 s / 23 s |
| SQL on Datahike (pg-datahike) | 65 | 55–74 % | $0.0026 | 10,136 | 10 s / 41 s |
| Datalog on Datahike | 64 | 54–73 % | $0.0044 | 15,061 | 13 s / 42 s |
All three routes scored similarly. Each pair disagreed on seven to ten questions, nearly evenly split. Datalog used more input tokens while the model examined the schema, roughly doubling list-price cost per correct answer against SQL. A team could pay nearly twice as much per correct answer without gaining accuracy.
We also ran nine models on the same 100 questions:
| Model | SQL on SQLite (of 100) | Datalog (of 100) | SQL, renamed copy | Cost per correct answer, SQL / Datalog | Median time, SQL |
|---|---|---|---|---|---|
| GPT-5.6 Luna | 65 | 64 | 59 | $0.0023 / $0.0044 | 10 s |
| Claude Haiku 5.5 | 69 | 68 | 67 | $0.0026 / $0.0047 | 9 s |
| DeepSeek V4.1 Flash (open weights) | 66 | 64 | 66 | $0.0074 / $0.0195 | 8 s |
| GLM 5.3 Flash (open weights) | 64 | 58 | 69 | $0.0042 / $0.0106 | 6 s |
| GPT-6 Luna | 68 | 60 | 63 | $0.0009 / $0.0014 | 5 s |
| GPT-6.1 Sol | 74 | 71 | 69 | $0.015 / $0.019 | 8 s |
| GPT-6 Astra | 75 | 70 | 71 | $0.067 / $0.091 | 6 s |
| Claude Sonnet 5.5 | 75 | 72 | 72 | $0.046 / $0.038 | 7 s |
| Claude Opus 5.5 | 82 | 81 | 80 | $0.054 / $0.073 | 8 s |
Among the first four models, no pair showed a reliable advantage after considering all comparisons. Pairs disagreed on 10 to 18 questions. Even the most uneven split, GLM against Haiku in Datalog (p = 0.013), can arise somewhere among 28 comparisons by chance. One favourable split is weak evidence for buying a model.
GPT-6.1 Sol and GPT-6 Astra answered 74 and 75 SQL questions, against Luna 5.6’s 65; their paired wins and losses against Luna were eleven to two and twelve to two. Claude Opus 5.5, measured a few days later, answered 82 SQL and 81 Datalog questions, the most of any model here. It beat every other model in paired comparisons except Astra in SQL (nine to two, p = 0.065), and on the renamed copy it beat Astra too, counted among its own comparisons. Claude Sonnet answered 75, level with Sol and Astra. Per correct SQL answer, Opus cost less than Astra and about three and a half times as much as Sol.
The two open-weight models, served here by Fireworks, answered about as many questions as GPT-5.6 Luna and Claude Haiku. Their list-price cost per correct answer was two to three times higher because they used more tokens and received no cached-input discount. An organization could run those weights on its own machines to keep data there. We did not measure that option’s hardware and operating costs.
Test the proposed change on new cases #
A promising candidate still needs a fresh test before rollout. Cases used to improve it cannot show whether that improvement carries over to unseen work.
Dvergr keeps the task, instructions, checker, cases and sandbox code in its room. A person or agent can propose another instruction, step or model. Each candidate runs in separate forks on tuning cases, then unseen cases. Checker promotion requires a host administrator’s connection outside the room, beyond an agent’s write access.
We tried this by hand with a Datalog workflow using Datahike to answer BIRD questions. After inspecting failures, we added two sentences explaining Datalog, then measured the revision on 100 unseen questions:
| Workflow | Correct (of 100) | Input tokens per question |
|---|---|---|
| SQL on SQLite | 59 | 8,880 |
| Datalog, revised description | 63 | 14,042 |
| Datalog, previous description | 58 | 16,684 |
The revision answered five more questions with fewer input tokens, but chance remains plausible (p = 0.23). Dvergr measures variants proposed by people or agents today. We are building toward agents running that propose-and-measure loop before handing candidates to owners. Fresh cases must still catch regressions before a team changes a production workflow.
What these tests cannot decide for your team #
Each model answered each case once, so these runs do not measure variation across repeats. Intervals concern sampled cases. A tie on 59 tasks does not prove equality, and 100 cases can miss a small regression. Providers can also change a model behind a familiar name, and a model may have learned a public benchmark’s answers, as most did on BIRD. Public cases show how instructions, tools, engines and checkers affect results. They cannot estimate the finance team’s repair cost, acceptable failure rate or next month’s case mix. For that, the team needs checked cases from its own history and reruns after workflow or model changes.
Method and scope
| Models | GPT-5.6 Luna and Sol, GPT-6 Luna, GPT-6.1 Sol and GPT-6 Astra (Codex subscription); Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 (Claude Code subscription, each pinned to its exact version); GLM 5.3 Flash and DeepSeek V4.1 Flash (Fireworks). Each answers every case once. |
| Cases | BIRD dev, the held-out third of each database’s questions, fresh samples of 100; SpreadsheetBench verified, held-out tasks among the 390 certified |
| Grading | BIRD: the returned rows equal the recorded query’s rows; SpreadsheetBench: the answer cells equal the recorded workbook’s values |
| Statistics | Jeffreys 95 % intervals for success rates, as Dvergr’s report gives them; exact McNemar test on the cases two candidates disagree on, computed from the recorded per-case verdicts |
| Cost | list price per million tokens on the day of measurement, cached input at the cache rate |
| Reference setups | BIRD’s prompt from its published gpt_request.py and its evaluation script; SpreadsheetBench’s single- and multi-round drivers and its comparison code, the generated programs run in a Docker container; both reach the model through Dvergr’s model connection:benchmarks/reference/ |
| Code and data | BIRD and SpreadsheetBench are public; Dvergr and both benchmark adapters are open source: github.com/replikativ/dvergr |
| Recall test | each model received a BIRD question’s text and its database’s name only, and was asked for BIRD’s recorded query; the reply was run and its rows compared with the recorded answer’s, as in grading |
| Renamed copy | the same 100 questions with every database, table and column renamed, the recorded queries rewritten and verified to return the same rows, the questions and hints reworded (checked by Claude Sonnet 5.5, three by hand); data and recorded answers unchanged; SQL on SQLite only |
| Updated | 2026-10-15: Claude Sonnet 5.5 and Opus 5.5 added to the BIRD comparison; a recall test and a renamed copy of the BIRD questions added; the estimate of what BIRD’s recorded answers allow revised |
Decide on your own cases #
The finance team’s decision rests on its own checked history: past bookings, coded invoices or workbook edits with recorded outcomes. Those cases show which failures staff must repair, what a usable result costs and whether a change improves its work.
Dvergr can run that comparison today on the organization’s own machine. Start with one workflow and checked historical cases. Export each input, recorded outcome and grading rule. Dvergr certifies the case pack, naming records it cannot grade before model runs. It then compares candidates on the same cases and reports failures, success intervals, list-price cost and time. Technical teams can follow running benchmarks and case-pack setup.
Dvergr runs the comparison. Simmis keeps proposed changes separate from accepted managed state until authorized adoption. Connecting Dvergr’s benchmark report directly to that review path remains in progress.
If your team has an agent to evaluate, we offer a fixed-fee six-week pilot on one workflow. You receive a benchmark and a report on model, instructions, steps and measured cost. Send us a note with your workflow and checked examples.