← All posts A lab of Claude bots ran 103 coding trials for $3.97 and found Haiku passed everything Sonnet and Opus did. A coder bot taught by that lab then passed 10 of 10 new tasks for 5% of what always-Opus cost.
Hi everyone, Chris here. Most agents today are a prompt and a hope. You tell the model the rules, you tell it the budget, and you trust it to remember both. Then you pay for the biggest model on every step, because you don't know which steps need it.
So I tried something else, and ran it on my Mac today. Here's what a Glyphh bot is, what a lab of them found out about Claude, and what it saved when a bot used what the lab learned.
10 / 10 new coding tasks passed by the coder botalways Sonnet: 9 / 10
0.05× what always-Opus cost for the same 10$0.050 against $1.057
103 trials the lab ran to learn it$3.97 in all
What a bot is #
A Glyphh bot is a Claude agent with a program around it. Four parts:
- The program runs the loop. What happens next, in what order, and when it's done is code, not a prompt.
- Hard gates are rules in that code: the budget, what the bot may do, what needs a person to approve. A step that breaks one is refused. Every time.
- Ada gates make the judgment calls between steps, like which model a task needs. Each one is an Ada model. It learns from real outcomes and from the answers a person gives.
- Claude is called only for the steps that need it.
And bots are made by a bot. The orchestrator is a bot like any other, with its own program and gates, and it learns from every bot it makes.
The lab #
I pointed the orchestrator at one question: how does Claude do coding work best? Each generation it proposed experiments. Haiku against Sonnet against Opus. A plain prompt against tests first, plan first, or edge cases first. Its hard rules refused any proposal outside those arms, and any that changed more than one thing at once. For each proposal left, it made an experiment bot.
claude -p in a fresh temporary folder on one of 12 small Python tasks: bugfixes, from-scratch builds and multi-file changes. A cron next-run, an expression evaluator, a bank ledger bug, CSV quoting, money rounding. Each was judged by hidden tests Claude never saw. Each experiment bot's Ada gate learned which cells, a task and an arm, were already known, and only sent trials where the outcome wasn't.
Here's what it found:
- Haiku passed every task Sonnet and Opus did , at about a tenth of Sonnet's cost and a 25th of Opus's, six trials an arm per experiment.
- Tests first and edge cases first cost more and gained nothing.
- Haiku was slower , two to three times on the hardest tasks.
promising gate rated two model-against-model proposals at P(decisive) 1.00 and 0.95, and a prompting proposal at 0.17. It ran them in that order. The orchestrator had learned which experiments pay off.
The whole lab was 103 trials and $3.97 of Claude usage. Per trial, on average: Haiku $0.0039, Sonnet $0.0395, Opus $0.1011.
The proof #
coder bot and handed it what the lab learned: its Ada route gate was seeded from the lab's trials, each task's cheapest model that passed every time. Then I gave it five tasks the lab never saw: a slugify bugfix, a token-bucket rate limiter bugfix, a topological sort that reports cycles, a small template language, and a multi-file job queue with retries and dead letters. Twice each, against always Opus and always Sonnet, which is Claude Code's default.
| Tasks | Passed | Total $ | vs always Opus | Total s | |
|---|---|---|---|---|---|
| Always Opus | 10 | 10 | 1.057 | 1.00× | 315 |
| Always Sonnet (Claude Code's default) | 10 | 9 | 0.487 | 0.46× | 280 |
| Routed by the coder bot | 10 | 10 | 0.050 | 0.05× | 351 |
The bot passed all ten, like always Opus, for a twentieth of the cost. It picked Haiku for every task at P 1.00, and never needed its escalation ladder, Haiku to Sonnet to Opus on a failure. Always Sonnet missed one and cost ten times as much.
What it leaves out #
Here's what this test doesn't show, so you can weigh the rest:
What's next #
The point isn't that Haiku is always enough. It won't be, on harder work. The point is that a bot doesn't have to guess. It can find out what a job needs, remember it, and pay for that and no more. When a task does need Opus, the gate learns that too.
the Claude bots use case. Build with AI. Run without it.
— Timme
Ada learns from your labelled cases, answers from its nearest precedents, and hands what it is unsure of to an LLM. Start free on Ada Cloud.