Open-weight models on owned hardware cost nothing per call. Claude Opus is billed per token and is better at almost everything. So the point where the cloud actually earns its fee was measured: six benchmarks, fifty real GitHub issues, and twenty ways of splitting the work between the two.
Open-weight models on owned hardware cost nothing per call. Claude Opus is billed per token and is better at almost everything. The obvious plan: let the local model do the work, and call Opus only when the local model is about to get something wrong. Cloud quality, mostly at local prices.
The whole plan hangs on one question: how do you know when the local model is wrong? Get that right and you pay for Opus only on the handful of tasks that need it. Get it wrong in one direction and you've paid for the cloud on work the free model had already done. Get it wrong in the other and you've shipped the free model's mistakes.
It was measured over a few weeks, on six benchmarks, climbing from multiple-choice trivia to real GitHub issues, with twenty different ways of splitting the work. Each round sharpened the question, and the last one settled it.
The setup #
Every setup in this post is built from the same three parts.
| Part | What it does | What it costs |
|---|---|---|
| The worker | An open-weight model does the task first. The smaller ones run on owned hardware, the larger ones on Cloudflare Workers AI | Negligible: a few cents across the whole study |
| The judge | Something reads the worker's result, or watches the worker as it goes, and says how likely it is to be wrong. Also open-weight | Negligible |
| The cloud | Claude Opus, which takes over a task when the judge's doubt is high enough | Billed per token |
The worker goes first on everything. The judge then decides, one task at a time, whether to escalate: hand that task to Opus and pay. That single decision is the whole subject of the post.
Here is one task going through it. A maths word problem comes in. The worker, a 7B Qwen model, answers 42. The judge, a 3.8B Phi model, is shown the problem and the answer and asked one question, "Is this answer correct?", and says yes with only 30% confidence. That is doubt enough, so the problem goes to Opus, which answers 48, the right answer, and bills about half a cent. Had Phi said 95%, the 42 would have stood, right or wrong, at no cost.
How a judge is scored. A judge is good if, when you sort the worker's answers by its doubt, the wrong ones come out on top. The score used runs from 0.5 to 1.0. Pick one wrong answer and one right answer at random: the score is the chance the judge is more doubtful about the wrong one. A coin flip scores 0.5. A judge that ranks every wrong answer above every right one scores 1.0.
How a setup is scored. Accuracy, or issues fixed, against what was spent on Opus. Spend is written as a share of what Opus-on-everything costs, so the worker alone sits at 0% and Opus alone at 100%. Sending a random share of tasks to Opus gets you the straight line between those two points. A setup is worth having only if it lands above that line.
The tasks climb from cheap to real:
| Benchmark | What the worker has to do | Why it's here | Worker | Opus on all of it |
|---|---|---|---|---|
| MMLU, 600 questions | Multiple-choice trivia across 57 subjects | The cheapest place to test the basic ideas | Qwen2.5-7B | $1.20 |
| GSM8K, 600 questions | Grade-school maths word problems, answer is a number | A different kind of mistake from trivia | Qwen2.5-7B | $2.71 |
| ΟΒ²-bench, 40 tasks | Act as a customer-service agent: look up orders, process exchanges, follow store policy, chat with a simulated customer | Multi-step work, where mistakes build up over a run | Qwen3-32B and Qwen3-8B | Not spent. This one only asks who can spot a failing run |
| HumanEval+ and MBPP+, 540 problems | Write a Python function, checked by a hidden test suite | Code is the real target | Qwen3.8-27B | $2.10 |
| SWE-bench Verified mini, 50 issues | Fix a real GitHub issue from Django or Sphinx by editing the repository, checked by the project's own tests | Where the money is | Qwen3.8-27B as a coding agent | $4.35 |
Qwen2.5-7B is a general chat model. Qwen3.8-27B is a popular local coding model, and on the last row it runs as a plain command-line agent: it reads files, edits them, runs tests and submits a patch when it thinks it is done, up to 100 steps.
The judges come in three kinds:
- Small ordinary models. Phi-4-mini (3.8B), Llama-3.2-3B and Llama-3.1-8B, asked "Is this correct?" and read for their yes/no probability.
- The worker itself , two ways: the confidence it had while answering, which is free because it's already there, or shown its own answer and asked the same question.
- A decision model. Clef is not a chat model. It is a 27B model built to take a situation plus a yes/no question and return a calibrated probability, nothing else. Cloudflare hosts it. It turns up in every section.
The cloud is Claude Opus 5.5 throughout, with Claude Haiku, its cheap sibling, tried once near the end. Every cost in the post is what they billed.
1. Who should decide "this answer is probably wrong"? #
The local model is decent and Opus is better: 71.5% against 93.0% on trivia, 91.5% against 98.5% on maths. Opus on all 1,200 questions costs about $4. Four free ways of spotting the local model's mistakes were tried:
| Method | How it works |
|---|---|
| The worker's own confidence | How sure was it when it answered? |
| Self-check | Show it its own answer and ask "Is this correct?" |
| A second model | Ask Llama-3.1-8B or Phi-4-mini (3.8B) the same question |
| A decision model | Ask Clef |
To compare them: what share of questions do you have to send to Opus before the combined system is as accurate as Opus alone? Lower is better.
| Judge (trivia, with Opus) | Questions you must send to Opus |
|---|---|
| Clef, the decision model | 44% |
| Phi-4-mini (3.8B) | 79% |
| Llama-3.1-8B | 84% |
| The worker's own confidence | 88% | | Self-check | 93% |
Among the ordinary models, the smallest was the best judge, and the worker judging itself was the worst. A model checking its own work seems to share the blind spot that caused the mistake. The decision model beat all of them by a mile. On maths it was 33% with Clef against 52% for the best ordinary judge.
Even 44% isn't as bad as it sounds. "As accurate as Opus" is only a yardstick: a judge that doubts answers that were actually right sends them to Opus for nothing, and has to send many of those before the last few wrong ones get caught. The yardstick is also beatable, because sometimes the local model is right and Opus is wrong (8 of 600 trivia questions). A perfect judge would send 21.5% and score 94.3%, above Opus. Clef gets some of that: 93.2% on trivia for 68% of the money, 98.7% on maths for 62%. The margin over Opus is tiny. The saving is not.
Then code was tried, and the ordinary-model result flipped. New worker, the 27B coding model (92.6% on HumanEval+ and 76.4% on MBPP+, against Opus's 95.1% and 89.4%):
| Judge | HumanEval+ | MBPP+ |
|---|---|---|
| The worker checking its own code | 0.85 | 0.79 |
| The worker's own confidence | 0.86 | 0.68 |
| Clef (27B decision model) | 0.69 | 0.76 |
| Llama-3.1-8B | 0.62 | 0.62 |
| Llama-3.2-3B | 0.53 | 0.51 |
The small judge was a coin flip, and worse than random once money was involved: on MBPP+ with half of Opus's money, the worker's self-check reached 86.7%, random picks 82.9%, the 3B judge 81.4%. On trivia the judge was half the worker's size. Here it's a ninth, and judging code means reading code, which a 27B coding model does far better than a 3B generalist. Even the decision model, so strong on trivia, is a little behind the worker here.
So "small and different" was never the rule. The judge has to be capable enough to understand the work, and built to give honest probabilities. A decision model the size of the worker is the best default found here. For code, add the worker's self-check. And it's cheap: all three Clef models together judged about 8,000 answers for well under a dollar.
2. Don't set a cutoff. Set a budget and pace it #
Once you can rank answers by doubt, the question becomes what's the best accuracy for a given amount of money? The simplest rule: sort every local answer from most doubtful to least, send them to Opus in that order until the money runs out, keep the rest. With Phi-4-mini judging:
| Budget | Trivia | Maths |
|---|---|---|
| $0 (local only) | 71.5% | 91.5% |
| Half of what Opus-for-everything costs | 88.7% | 98.3% | | Opus for everything | 93.0% | 98.5% |
On maths, half the money gets you within 0.2 points of Opus.
That table cheats, though. It sorts all 600 questions first, which you can only do if you have them all up front. Real work arrives one task at a time, and you decide right then whether to pay. So 1,050 questions were replayed as a stream, trivia and maths mixed, with a hard budget of 25% of Opus-for-everything, and two ways to decide were compared:
| Approach | How it decides | Think of it as |
|---|---|---|
| Fixed threshold | Pick a cutoff once from 150 past questions, then never change it | A thermostat set once in January |
| Pacer | Keep adjusting the cutoff from your own spending: escalate more when behind budget, less when ahead | A thermostat that watches the temperature |
| Approach | Accuracy | Share of budget used |
|---|---|---|
| Local only | 83.1% | 0% |
| Fixed threshold | 87.3% | 29% |
| Pacer | 92.6% | 100% |
| Perfect hindsight (sort the whole stream first) | 92.8% | 100% |
The fixed threshold left 71% of its money unspent. Its cutoff was tuned on trivia, and the judge is far more confident about maths, so on the maths half of the stream it almost never fired. The money was there, and the rule had no way of noticing. The pacer spent the whole budget and landed within a point of perfect hindsight, and it held up in four different stream orders, including one that switches from all trivia to all maths halfway through.
On code the fixed threshold failed the opposite way. Tuned on HumanEval+, where the worker is confident, it escalated too eagerly on MBPP+ and ran out of money two-thirds of the way through the stream, every time. The pacer scored 85.8β86.0% in every order, against 86.1% for hindsight. It can't hoard and it can't over-spend early, because all it watches is the bill.
Two refinements that didn't help: preferring cheap questions (most doubt per dollar) was 0.3β0.8 points worse than the plain pacer, and a classic online-knapsack rule under-spent like the threshold. Keep it simple.
3. Watching an agent work #
Multiple choice is a toy. The work worth splitting is multi-step: browsing, coding, using tools. So the next stop was the customer-service agent of ΟΒ²-bench. Qwen3-32B passed 15 of 40 tasks, Qwen3-8B passed 7. Plenty of room for help.
Before any Opus was paid for, it was checked whether free warning signs predict a failing run. "Free" means read off the run itself, without asking any model: did a tool call return an error, did the agent repeat the exact same call. Then each recorded run was stopped halfway and three-quarters through, and two small models were asked, "Is the agent off track?"
| Signal (all 76 runs) | Halfway through | Three-quarters through |
|---|---|---|
| Free warning signs, counted | 0.64 | 0.70 |
| Phi-4-mini reading the run | 0.61 | 0.73 |
| Qwen3-8B reading the run | 0.71 | 0.73 |
- Repeated calls catch the weak model (0.82 for the 8B agent): when it's lost, it loops.The strong model fails quietly. It rarely loops, and tool errors show up in its good runs too.
- Readers help, but modestly. Nothing got much past 0.7. Later is easier: failing runs look more like failure the longer they go.
- The raw numbers lie, the ranking doesn't. Phi said "off track" with 45β65% confidence on average, Qwen3-8B with 6β13%, for the same runs. Neither number means anything on its own, but both rank failing runs above passing ones, which is all a pacer needs.
(The simulated customer here is a local 32B, not the stronger model the official leaderboard uses, so these pass rates can't be compared with published ones.)
The overseer, live on real bugs
Now the real test: a live agent fixing the 50 real GitHub issues of SWE-bench Verified mini, each checked by the project's own tests.
The overseer is everything above, applied mid-run:
- The local model works as usual.
- From its third step on, before every step, a gate looks at the run. Two free checks first: an exact repeated command, or three failed commands in a row. Then a judge reads the issue plus the last ten messages and answers "Is the agent on track?" as a probability. One version uses Llama-3.2-3B for this, the other Clef.
- A pacer turns that into a decision. It ranks the score against every score seen so far in the batch, works out what share of the remaining checks it can still afford to escalate, and escalates only when this one falls in that worst share.
- On escalation, Opus takes over the same conversation for 3 steps , then hands back.
- The budget for the batch was 25% of what Opus-only cost on the same issues.
| Setup (50 real issues) | Issues fixed | Opus cost |
|---|---|---|
| Opus only | 48 | $4.35 |
| Overseer, Clef gate | 38 | $0.89 |
| Overseer, 3B gate | 34 | $1.18 |
| Local only | 29 | $0 |
That looks like a clean win: nine more issues than local for a fifth of Opus's cost. Then it was checked which of the Clef overseer's 11 wins over local actually involved Opus doing anything:
| Clef overseer: its 11 issues fixed that local-only missed | Count |
|---|---|
| Opus stepped in on that issue | 3 |
| Opus never acted. The same local model just succeeded this time | 8 |
Most of the gain was luck. The local model, run again on the same issue, changed its outcome on roughly one issue in four, even at temperature 0: hosted inference isn't perfectly repeatable, and a 50-step agent amplifies tiny differences. Three local-only runs fixed 29, 29 and 34. Both overseers got 3β4 genuine wins from Opus.
Two things follow from that. A single run per setup can't rank setups on 50 tasks: differences of two or three issues are noise, and every table below should be read with that in mind. And count wins by cause: "fixed because Opus stepped in" and "fixed anyway" have to be logged separately, or the overseer gets credit for luck.
4. Don't watch the work. Check the result #
Judging a run in progress is hard because a run that's going badly often looks like one that's going fine, until it ends. So the order was flipped. Let the local model finish every task, then ask one question about the finished patch, "Is this a correct and complete fix for the issue?", and let Opus redo only the doubtful ones.
| Who checks the finished patch | Spots a bad patch |
|---|---|
| Clef, the decision model | 0.89 |
| The local model checking its own patch | 0.84 |
| "Did the agent submit, or run out of steps?" | 0.64 |
| Llama-3.2-3B | 0.60 | That's 0.89, against 0.6β0.7 for watching mid-run. Judging finished work is much easier than judging work in progress. Then the money question, with Opus redoing the most doubtful tasks until the budget runs out:
| Opus budget (share of Opus-for-everything) | 20% | 40% | 50% |
|---|---|---|---|
| Check with Clef | 40 | 43 | 45 | | Check with the local model's self-check | 38 | 43 | 44 | | Redo random tasks | 33 | 37 | 38 |
45 of 50 for half of Opus's cost, against 48 for Opus alone and 29 for local alone. It's a replay (real local runs and real Opus redos, combined after the fact), but it was the first setup that clearly beat the straight line between local and Opus.
5. Who should go first? #
Both setups so far let the local model start and brought Opus in later. There's a mirror image: let Opus go first. It can go first in two very different ways, and they came out on opposite ends of the table.
Scout first: local explores, Opus finishes. The local model takes 10 steps (find the files, reproduce the bug), then Opus continues in the same conversation. It fixed 48, as many as Opus alone, and every win involved Opus. But it cost 110% of Opus alone: the local model finished inside its 10 steps on only 6 issues, and picking up a conversation that already holds 10 steps of someone else's work costs more than starting fresh. Handing over a short brief instead of the transcript (the files found, how to reproduce the bug, what was tried) brought it to 46 for 98%. A brief clearly beats forwarding a transcript, but here it only broke even, because the cloud agent was already lean at about 9Β’ an issue. There wasn't much searching left to save.
Plan first: Opus thinks, local builds. The reverse:
- Opus reads the code, without changing anything, and writes a short plan: which file, which function, the exact change, and how to test it.
- A fresh local model gets the issue plus the plan, nothing else, and does all the editing and testing.
- Opus never touches the code.
A plan for a Django bug about two-digit years in HTTP dates looked like this:
File: django/utils/http.py, function parse_http_date. Change: replace the hard-coded rule ("under 70 means 20xx, otherwise 19xx") with this code [about ten lines, given in full], which reads a two-digit year as the past if it would otherwise land more than 50 years in the future. Check: parse two sample dates and expect 1994 and 2070, then run tests/runtests.py utils_tests.test_http.
| Opus alone | Opus plans, local builds | |
|---|---|---|
| Who reads the code | Opus | Opus, 2β4 steps |
| Who edits and tests | Opus | The local model, about 17 steps |
| Opus cost per issue (typical) | 6.2Β’ | 3.3Β’ |
| Issues fixed (of 50) | 48 | 44 and 47 (two runs) |
| Issues local alone fixed that this lost | 0 |
Why it works: thinking is the cheap part of an agent. The expensive part is the long tail of editing, running tests, reading errors and trying again. Here Opus does only the thinking, in a few short steps, and the free model does the long tail. It was run with Opus capped at 8 reading steps and with no cap. Opus never used more than 4 either way, so the gap between 44 and 47 is run-to-run noise. A cousin, where Opus splits the issue into up to three subtasks and reviews the result once, fixed 44 for 40%. Three live runs of "Opus thinks, local types", three results between 44 and 47, all at 40β44% of the cost.
One surprise: checking doesn't stack on planning. The check from section 4 was expected to stack on top. But Clef couldn't tell good plan-built patches from bad ones (0.53, a coin flip, against 0.89 on patches the local model wrote alone). With three failures in 50 there was little to find, and the failures looked as convincing as the successes.
6. Everything else that was tried #
A local model and Opus are a team with a big skill gap, and people have been splitting work across gaps like that forever: surgeons and assistants, help desks with tiers, two radiologists reading every scan. The obvious arrangements were turned into setups too. Everything on the same 50 issues, sorted by issues fixed. Cost is the share of what Opus-for-everything cost ($4.35):
| Setup | Issues fixed | Cloud cost |
|---|---|---|
| Opus only | 48 | 100% |
| Scout: local for 10 steps, then Opus | 48 | 110% |
| Opus plans, local builds (best of two runs) | 47 | 43% |
| Scout with a short brief | 46 | 98% |
| The surgical team: Opus works, local summarizes long outputs | 46 | 84% |
| Two readers check the patch, Opus breaks ties | 46 | 59% |
| Check the finished patch, Opus redoes doubtful ones | 45 | 50% |
| A tiered help desk: local, then Haiku, then Opus (replay, cutoffs tuned after the fact) | 45 | 49% |
| Opus plans, local builds (other run) | 44 | 44% |
| Opus splits into subtasks, local builds | 44 | 40% |
| Pair programming: Opus advises every 5 steps | 43 | 206% |
| Route by the issue text: hard-looking ones straight to Opus | 42 | 50% |
| Trust earned per project | 42 | 50% |
| Overseer, Clef gate | 38 | 20% |
| Overseer, 3B gate | 34 | 27% |
| Local may ask Opus for help | 34 | 1% |
| Claude Haiku alone | 33 | 3% |
| Try three times, keep the best (no Opus) | 33 | 0% |
| Local only (three runs) | 29, 29, 34 | 0% |
A few of them deserve a line each.
- Pair programming was the most expensive setup of all. Opus looked over the local model's shoulder every 5 steps, 298 reviews in total. Twice the cost of Opus doing everything, and fewer fixed. Watching costs more than working.
- The junior never asked. Allowed to ask Opus up to three questions per issue, the local model asked twice in 50 issues. It doesn't know when it's stuck, so you get the local model alone.
- Three tries barely beat one. Clef picking the best of three local patches: 33, against 30.7 for picking one at random. Even a perfect picker would reach only 36. Most issues the local model fails, it fails every time.
- A middle tier helps a little, at best. Claude Haiku alone fixed 33 for 3% of Opus's price. Putting it between local and Opus, with Clef scoring the finished patch at each tier and escalating below a cutoff, did 1β4 issues better than local-then-Opus at the same price. But both cutoffs were chosen after the fact on these same 50 issues, so the real gain is probably smaller.
- A second reader added nothing. Clef and the local model disagreed on only 8 of 50 patches, so the Opus tie-breaker had little to do. Clef alone, given the same money, also fixed 46.
- Seeing the work beats predicting it. Guessing from the issue text alone was decent (0.81), but fixed 42 against 45 for checking the finished patch at the same price.
- The surgical team had nothing to do. Only 7 of 50 issues produced a long output for the local model to summarize. It saved 16%.
Judgment, not labour
Sort that table by what Opus was actually paid to do, and one pattern covers all of it:
| What Opus was paid for | Setups | Issues fixed | Cloud cost |
|---|---|---|---|
| Think briefly up front, then leave | Write a plan, split into subtasks | 44β47 | 40β44% |
| Read the finished work, redo only what's doubtful | Check the patch, two readers, the tiered desk | 45β46 | 49β59% |
| Stay involved throughout | Pair programming, the surgical team, scout then take over | 43β48 | 84β206% |
| Step in mid-run, when a gate or the local model calls | The overseers, ask a senior | 34β38 | 1β27% |
| Do everything | Opus only | 48 | 100% |
The cloud model is worth paying for its judgment, not its labour. Where Opus thought for a few steps and left, the free model did the rest nearly as well as Opus does it, for under half the price. Where Opus stayed to work or to watch, the bill climbed back to Opus's own, or past it, because the long tail of an agent's work (edit, run, read the error, retry) is the expensive part no matter who does the thinking. And where the decision to call Opus was left to the run itself, mid-flight, it barely got called at all: a struggling run looks fine from the inside.
What should be built #
For coding:
| Step | What happens | Cost | Evidence |
|---|---|---|---|
| 1. The cloud model writes a short plan | Read-only, a few steps, then it leaves | A few cents | 44β47 of 50 at 40β44% of Opus-only |
| 2. The local model does the work, following the plan | Always | Free | |
| 3. Stop a run that's clearly stuck | Repeated identical calls, error streaks | Free | Catches a weak model's loops |
| 4. Judge the result by its tests | After the local model finishes | Free | A check model can't sort plan-built patches |
For work without a plan, quick answers or tasks where planning isn't worth it:
| Step | What happens | Cost | Evidence |
|---|---|---|---|
| 1. The local model does the work | Always | Free | |
| 2. Check the finished result | A decision model the worker's size asks "is this correct?" For code, the worker checks itself too | Cents | 0.89 on finished patches, against 0.6β0.7 for watching mid-run |
| 3. The cloud model redoes what the check doubts | Only the doubtful ones | Paid | 45 of 50 at half the cost |
| 4. A pacer sets how strict step 3 is | Always | Caps the bill | Beat a fixed cutoff whether the cutoff under-spent or over-spent |
Leave out: a cloud model watching every few steps, letting the local model decide when to ask for help, and running the local model several times. If the cloud model ever has to take over mid-task, hand it a short brief, never the transcript.
The scorecard #
Every question asked along the way, and what the data said:
| Question | Answer |
|---|---|
| Who spots a wrong local answer best? | A model built for judging : a 27B decision model beat every ordinary model on trivia and maths, by a wide margin. On code, the worker checking its own code. A judge too small to understand the work is a coin flip |
| How should you limit cloud spend? | Set a budget and pace it. A fixed confidence cutoff under-spent on one task and ran out of money early on another. Pacing landed within a point of perfect hindsight |
| Can you tell, mid-run, that an agent is going wrong? | Barely. Watchers reading a run halfway scored 0.6β0.7, on a scale where 0.5 is a coin flip. On real bug fixes, mid-run rescues by Opus won 3β4 issues of 50 |
| Can you tell from the finished work? | Yes. A decision model reading the finished patch scored 0.89, and letting Opus redo what it doubted fixed 45 of 50 for half of Opus's cost |
| What was the best deal on real bug fixes? | Opus writes a short plan, the local model builds it: 44β47 of 50 fixed for 40β44% of what Opus alone costs. Opus alone fixed 48 |
| What didn't work? | Opus advising every few steps (twice Opus's cost), the local model scouting and then handing over (more than Opus's cost), letting the local model ask for help (it almost never asks), and running it three times |
The short version #
A handful of models and six benchmarks don't make a law. But for anyone splitting work between a model on their desk and one in a datacenter, four defaults are the place to start. Pay the cloud model for judgment, not labour: a short plan from the expensive model makes the free one far more capable, and keeping it around afterwards mostly adds cost. Let the local model finish, then check the result, rather than trying to spot trouble along the way. Let whoever best understands the work do the checking: a decision model for simple answers, the worker itself for code. And let a paced budget decide how much doubt you can afford.
A plug #
Does your local model live on a Mac at home while you work from somewhere else? Do you want to watch the local agent build, see the moment Claude is called, and nudge either of them from your phone or any browser? Every run in this post was driven that way, from a Mac in another room. Try reminal.