# Meta-Evaluation: Which Tool Builds the Better Agent > Source: > Published: 2026-09-17 00:00:00+00:00 # Meta-Evaluation: Which Tool Builds the Better Agent There are well over a hundred agent benchmarks now. The useful question isn't which agent tops one of them. It's which method to build an agent with in that domain, and what that method costs in dollars and in a person's time. We compare Mutagent Helix with Claude Code across twenty of them. Hundreds of runs of the same loop, on public benchmarks and on other problems, each one starting from a blank playbook and ending at a score: spec, build, evaluate, diagnose, optimize. That’s the data Helix is tuned on. None of those runs was about winning a leaderboard. Each was a question about the loop itself: when you point Helix at a task it was never designed around, does it get an agent to a good score with less money and less of a person’s time than doing the same work by hand in a coding agent? That question is a meta-evaluation. This post is what we’ve learned from running it. ## Method: one calibration pass, run two ways An evaluation scores an agent. A meta-evaluation scores the thing that builds agents. The unit of measurement is one calibration pass: take a rough first draft of an agent, a benchmark with a scorer, and a model that has a realistic shot at the top of the board, and run the loop until the score stops moving. Then count two things. What it cost in dollars, and how many times a human had to interact with the tool to get there. Benchmarks are the stand-in for the real situation, which is a problem you own, some data or traces for it, and a way to tell good from bad. A benchmark packages all three, plus a scorer somebody else wrote, so the calibration can’t quietly grade itself. It’s not a claim that the agent beats the benchmark. It’s a controlled way to compare two methods of building agents on the same task, so that on the next real problem we know which tool to reach for. The right analogy is an optimizer like Adam. Nobody faults Adam for fitting the training set. The question is whether it converges, fast and cheap, on whatever you point it at, and that is the property a calibration pass measures. Keeping a held-out set is data discipline. It sits with whoever runs the loop, not with the loop, and none of the three showcase agents claims to have kept one. We run the same calibration two ways. Once with Helix, where the loop runs the stages and a person answers its questions. Once by hand in Claude Code, where the same person drives every stage: reads the traces, writes the criteria, decides what broke, applies the fix, reruns. Same model underneath, same benchmark, same scorer. ## A taxonomy of agent benchmarks More than anyone has counted. The [ACL 2026 survey on evaluating LLM-based agents](https://aclanthology.org/2026.findings-acl.1330/) names more than sixty across nine categories, and its [companion repository](https://github.com/Asaf-Yehudai/LLM-Agent-Evaluation-Survey) lists over a hundred and seventy papers and frameworks. A narrower [mini-survey from February 2026](https://arxiv.org/pdf/2602.05073) restricted itself to text-only LLM agents and still counted forty-four. Those counts stop at the general-purpose ones, and the growth now is in domain benchmarks with expert-written rubrics that arrive monthly. Put both together and the honest number is well past a hundred, with no single list that holds all of them. A count that large only becomes useful once it’s sorted, and two axes do most of the work. The first is what the benchmark tests: a general capability every agent needs, or expertise in one domain. The second is how it scores: against a checkable ground truth, against a rubric an expert wrote, or by a model judging a transcript. Where a benchmark sits on those two axes decides what a calibration pass against it looks like, because it decides what the loop can verify on its own and what needs a person. **Coding** SWE-bench VerifiedSWE-bench ProSWE-bench MultimodalTerminal-BenchSWE-LancerAider PolyglotLiveCodeBench **Tool use** BFCLAppWorldMCP-UniverseMCPMarkACEBench **Computer and web** OSWorldAndroidWorldWindowsAgentArenaMind2WebBrowseComp **Research engineering** MLE-benchRE-Bench **Long horizon** Vending-BenchAgentBenchBALROG **CAD** Parametric CAD BenchCADGenBenchText2CAD-BenchRealCADBenchBenchCADP3D-Bench **Security** CybenchNYU CTF BenchCyberGym **Science** ScienceAgentBenchDiscoveryBenchCORE-BenchLAB-BenchChemBench **Data science** DSBenchDA-CodeDABstep **Finance** FinGAIAInvestorBenchFinAgentBench **Healthcare** EHRAgentMedAgentGym **Legal** LegalAgentBench **Office** SheetCopilotSpreadsheetBench 2 **Research** PaperBench **Knowledge work** GDPval **Legal** Harvey Legal Bench **Finance** Finance Agent Bench **Science** AstaBenchBixBench **Simulated user** tau-benchtau2-benchTheAgentCompany **Model judge** GAIAWebArenaWebVoyagerToolBench **Healthcare** MedAgentBenchHealthAgentBenchAgentClinic **Automotive** CAR-bench Color is the family: software and data science and research engineering regulated professions security general. The number in each cell is how many are listed there. The grid holds about sixty, which is a sample of the field, not a census. The rows tell you how much of the loop’s knowledge has to come from the domain: a general benchmark is mostly harness, a domain one is harness plus conventions the benchmark authors assumed. The columns tell you where the reward comes from. A checkable result lets the loop verify every attempt itself, which is the same property that made coding agents work. A rubric moves the check to a judge reading against written criteria, which is reliable but not free. A judged transcript puts a second model in the loop as the counterpart, and its behavior becomes part of what’s being measured. The third thing that varies is what a single run costs, and it varies by orders of magnitude. The EvalEval coalition’s [cost survey](https://huggingface.co/blog/evaleval/eval-costs-bottleneck) puts one GAIA pass anywhere from $7.80 to $2,829 depending on the agent, tau-bench airline from $0.31 to $180, and a full PaperBench evaluation near $9,500. A [study of SWE-bench Verified runs](https://arxiv.org/pdf/2603.23749) found a median of $163 per run with per-task costs spanning $0.08 to $32 across models. Princeton’s [HAL leaderboard](https://arxiv.org/abs/2510.11977) spent about forty thousand dollars running nine models through nine benchmarks. Those spreads are the reason a meta-evaluation has to be run across the table, not in one cell: any single benchmark is a corner case. ## What one calibration pass costs We took twenty of those benchmarks, spanning the cheap tool-use corner to the expensive research end, and priced the two numbers for each: the dollars one calibration pass costs, and the human interactions it takes, once with Helix and once by hand in Claude Code. “Calibrated” needs a stopping rule, or the cost of a pass is whatever you feel like spending. Ours is one line: a pass is done when the agent scores in the top ten percent of the benchmark’s public leaderboard, on the full task set, with a model that has a realistic shot at getting there. It starts from a first draft that runs end to end, and a budget cap sits behind it, so a pass that hits the cap first is reported as unfinished. That rule is what makes the two arms comparable. Both stop at the same line, so the dollars and the interactions count the same amount of progress. Each position starts from what one full evaluation run costs with a top-of-board model, taken from the leaderboards and papers that publish it: Princeton’s HAL cost columns for tau-bench, GAIA, SciCode, Online Mind2Web, and SWE-bench Verified, the PaperBench, SWE-Lancer, MLE-bench, and RE-Bench papers, Harvey’s and gNucleus’s published costs, and our own runs where nothing is published. A pass is then priced as a fixed overhead plus one and a half full runs, with everything past a thousand dollars per run discounted, because on an expensive benchmark nobody runs the full suite every iteration. You smoke-test on five tasks, then twenty, and only run the whole thing to confirm. That’s why the dollar band is narrower than the raw per-run costs would suggest: a cheap tool-use benchmark and an expensive research benchmark end up about a factor of twenty apart on the plot, not a thousand. The interaction count is where the two tools separate. With Helix, one calibration pass takes between five and fifty human interactions, most of them in the low teens. By hand, the same pass takes three to ten times that. The reason is mechanical, not magical: every read of a trace batch, every review of a design, every diagnosis of a failing trial, and every rerun after a fix is something a person has to ask for, read, and act on. Helix runs those stages and comes back with a verdict and a question. The person answers the question. ## Where the difference actually comes from The model is the same in both arms, so the difference has to come from what’s wrapped around it. Claude Code is a general coding agent. It does what you ask, one session at a time, and what it knows about building agents is what the model knows plus what you type. Helix is built for one job, building and improving AI agents, and it carries what practitioners have learned about that job over decades of machine learning and systems engineering: measure before you change anything, change one variable at a time, keep the judge away from the thing being judged, review the design before the expensive run, root-cause from evidence instead of guessing, and write the spec down so the next stage can check against it. In Helix those aren’t reminders. They’re the stages, with an artifact between each pair that the next one reads. The second difference is that Helix is tuned for this. The meta-evaluation in this post isn’t a one-off study. We run it constantly, on every agent-building problem that comes through, and what it finds goes back into the design of the loop. Claude Code’s harness is tuned too, by Anthropic, for coding in general, and nothing we learn about building agents changes it. Both tools can read a trace, write a test, or review a design when asked. The difference is in what happens without asking, and in what survives from one iteration to the next. | | Claude Code | Mutagent Helix | |---|---|---| | Unit of work | A session. What it produced is in the transcript. | A lifecycle stage. What it produced is an artifact with a schema: spec, build report, scorecard, findings. | | What decides the next step | The person, every turn. | The stage’s verdict. Proceed, steer, or abort routes to the next stage, and the person answers when asked. | | What carries over between iterations | The context window, then a summary of it. What survives is what you remembered to write to a file. | The artifacts. The spec, the interpretation ledger, and the last verdict are inputs to the next stage by contract. | | How a run gets scored | Whatever check you ask for, graded in the same session that wrote the code. | An eval suite derived from the spec: binary criteria, a separate judge, pinned model and temperature. | | How a failure gets explained | You paste the transcripts that fit and read the summary. | Tiered analysis over the whole trace batch, code-first, with a why-chain and evidence for each finding. | | How a fix lands | Edits land as you make them, often several at once. You re-run when you remember. | One remedy at a time, applied by a fix actor that is not the judge, re-evaluated before it counts. | | What the person does | Drives every step and holds the state in their head. | States the goal, drafts the first playbook, answers questions, approves verdicts. | | What gets better over time | The model, and a general coding harness, on the vendor’s schedule. | The loop itself, from the meta-evaluation across every problem it has run on. | Every row is something a careful person could reproduce by hand in a coding session, on a good day, with a checklist. The plot above is the price of the days that aren’t good and the steps that weren’t on the checklist. ## What we’re not claiming The plot is a reference, not a ledger. It’s built from the passes we’ve run, on benchmarks and on other problems, projected onto twenty public benchmarks using their published per-run costs and task counts. Read it as what to expect from a calibration pass, not as twenty completed ones. None of the three showcase agents is a leaderboard entry. The parametric CAD agent’s conventions were calibrated against the benchmark’s public reference geometry, and its submission said so, which keeps it off the gNucleus board by design. The in-car assistant was scored with our own simulated user, not the organizers’. The CAD-editing agent sits on the benchmark’s unvalidated tab pending methodology review. Each post says so. And the interaction ratio is the claim we’re most careful with. Five to fifty against three to ten times that is what we’ve seen. It’s a ratio measured by the people who built the tool. We’ll keep publishing the passes, including the ones where the ratio narrows. ## What we do with the answer The point of running the same calibration on twenty benchmarks is not to have twenty scores. It’s to know, when the next real problem arrives, which tool to pick and what it will cost. If the next problem is a month of traces from a support agent, we know what a calibration pass against that looks like because we’ve run its nearest benchmark neighbor both ways. Next on the list: the generation half of CADGenBench, which would put the CAD-editing agent on every task the leaderboard aggregates, and the in-car assistant re-scored under the organizers’ simulator. Both are one more row on the plot. Either the loop reads the traces, or you do. Everything on that plot is the price of that choice.