Claude didn’t get smarter. We changed everything around it. Somehow, 6.5 points appeared between them.
We beat Claude Code with Claude.
Which is a slightly ridiculous sentence, but it is also a useful one.
We ran Backboard CLI on Terminal-Bench 2.1 using Claude Opus 4.8 through Amazon Bedrock. Claude Code, using the same model, had a published score of 78.9%.
Our submitted result was 85.4% ± 0.8%.
Same underlying model.
Different system around it.
A 6.5-point difference.
And that is much more interesting to me than saying we found a smarter model.
Because we didn’t.
For the last few years, AI performance has mostly been discussed as a model problem. Want better results? Use the better model. And obviously, models matter. A lot. But once models become sufficiently capable, another question starts becoming more important:
How much of that capability can your system actually get out of them?
A model working inside an agent does not operate in a vacuum.
It receives context.
It sees tools.
It decides what to do next.
It fails.
It recovers.
It maintains state.
It spends tokens while doing all of this.
The model may be the engine, but there is quite a lot of car wrapped around it.
And occasionally someone remembers that transmissions matter too.
It is tempting to describe everything around the model as plumbing.
Context management. Tool interfaces. Planning. State. Recovery. Verification.
Necessary, certainly.
Exciting enough for a keynote slide?
Apparently less so.
But those decisions compound.
Give a capable model too much irrelevant context, and you make its job harder.
Give it tools it does not need, and you create more ways to make the wrong choice.
Handle failure poorly and one bad step can ruin an otherwise good trajectory.
Burn too many tokens getting there, and you may technically solve the task while also solving your CFO’s problem of having too much money.
The model did not suddenly become more intelligent because Backboard called it.
The system around the model got more of the existing intelligence into the result.
That distinction is the reason I care about the benchmark.
Performance was only half of it.
Our full submitted Terminal-Bench run cost $280.72. At the time of the original article, the current verified leader had scored 83.8% at a reported cost of $552.67.
So our submitted result came in higher, while the reported run cost was roughly 49% lower. That changes the question.
It is easy to ask:
Which model are you using?
It is harder, and much more useful, to ask:
What can you get that model to accomplish reliably, and what does it cost to get there?
Those are different optimization problems.
One produces model comparison charts. The other produces systems you can afford to run.
I have mixed feelings about AI leaderboards.
They are useful and extremely effective at turning otherwise sensible adults into people refreshing a webpage to see whether a decimal moved.
For us, the useful part is not the trophy. It is the failure data. The submitted evaluation covered 89 tasks, with five attempts per task, for 445 trials under the same agent configuration. We included the failures and errors. The aggregate result was 85.4%.
That was the number we were comfortable submitting.
Every failed task tells us something.
Sometimes the model made a bad decision. Sometimes context was poor.
Sometimes the harness allowed an unproductive trajectory to continue too long.
Sometimes an implementation change that seemed clever turned out to be very clever at making the benchmark worse.
Useful information, all of it.
A benchmark is valuable when it tells your engineers where the system breaks. The leaderboard is what happens afterward.
Comparing AI systems gets messy fast. Change the model, provider, prompt, tools, or context strategy and you’ve changed the experiment.
That’s why I find the Claude comparison interesting.
Same Claude Opus 4.8 on both sides. 6.5 points apart.
I don’t think there’s one clever trick behind that gap. It’s probably a lot of fairly boring engineering choices adding up.
Which, unfortunately, is usually how software works.
We care about this for a bigger reason too.
I don’t think companies should have to rebuild their AI stack every time a new model becomes the favorite.
Claude might be the right fit today. Tomorrow it could be something else. Six months from now, it could be a model nobody has heard of yet.
The models will keep changing. That’s normal.
The infrastructure around them shouldn’t have to.
Your context, memory, evaluation, application logic, and the lessons you’ve learned about turning model capability into useful work should be able to stick around.
That’s where I think a lot of the durable value in AI infrastructure will live.
This benchmark does not prove that models do not matter.
That would be a strange conclusion from an experiment powered by Claude Opus 4.8.
It proves something narrower.
Choosing a strong model is not the end of the engineering problem.
There is still a substantial gap between having access to intelligence and building a system that uses it well.
The surrounding architecture determines what context reaches the model, what actions it can take, how failures are handled, and how much the whole thing costs when it finally succeeds.
Those decisions are easy to dismiss individually.
Collectively, they can move a benchmark by 6.5 points.
Foundation models will keep getting better. They’ll keep changing too.
If several models can solve the same class of problems, simply having access to a good one becomes less of a differentiator. The harder questions move up the stack:
Can you make it reliable?
Can you use context well?
Can you recover when things go wrong?
Can you tell whether an optimization actually helped?
Can you do all of that without treating tokens like confetti?
That’s the engineering layer we’re betting on at Backboard.
Our Terminal-Bench result is one piece of evidence, not proof of everything. It’s simply a useful controlled comparison:
The model stayed the same. The system changed. The outcome changed with it.
That’s the part worth paying attention to.
This is a conversational remix of an article I published on Backboard’s blog. Read the original deep dive here.