{"slug": "same-claude-different-harness-very-different-result", "title": "Same Claude. Different Harness. Very Different Result.", "summary": "A developer's Backboard CLI harness running Claude Opus 4.8 through Amazon Bedrock scored 85.4% ± 0.8% on Terminal-Bench 2.1, beating Claude Code's published 78.9% on the same model. The submitted run cost $280.72 versus a reported $552.67 for the then-verified leader at 83.8%, suggesting the surrounding agent system — context management, tools, state, and recovery — accounts for much of the performance gap. The evaluation covered 89 tasks with five attempts each, for 445 trials under a single agent configuration.", "body_md": "*Claude didn’t get smarter. We changed everything around it. Somehow, 6.5 points appeared between them.*\n\nWe beat Claude Code with Claude.\n\nWhich is a slightly ridiculous sentence, but it is also a useful one.\n\nWe ran Backboard CLI on Terminal-Bench 2.1 using Claude Opus 4.8 through Amazon Bedrock. Claude Code, using the same model, had a published score of 78.9%.\n\nOur submitted result was 85.4% ± 0.8%.\n\nSame underlying model.\n\nDifferent system around it.\n\nA 6.5-point difference.\n\nAnd that is much more interesting to me than saying we found a smarter model.\n\nBecause we didn’t.\n\nFor the last few years, AI performance has mostly been discussed as a model problem.\n\nWant better results? Use the better model. And obviously, models matter. A lot. But once models become sufficiently capable, another question starts becoming more important:\n\nHow much of that capability can your system actually get out of them?\n\nA model working inside an agent does not operate in a vacuum.\n\nIt receives context.\n\nIt sees tools.\n\nIt decides what to do next.\n\nIt fails.\n\nIt recovers.\n\nIt maintains state.\n\nIt spends tokens while doing all of this.\n\nThe model may be the engine, but there is quite a lot of car wrapped around it.\n\nAnd occasionally someone remembers that transmissions matter too.\n\nIt is tempting to describe everything around the model as plumbing.\n\nContext management. Tool interfaces. Planning. State. Recovery. Verification.\n\nNecessary, certainly.\n\nExciting enough for a keynote slide?\n\nApparently less so.\n\nBut those decisions compound.\n\nGive a capable model too much irrelevant context, and you make its job harder.\n\nGive it tools it does not need, and you create more ways to make the wrong choice.\n\nHandle failure poorly and one bad step can ruin an otherwise good trajectory.\n\nBurn too many tokens getting there, and you may technically solve the task while also solving your CFO’s problem of having too much money.\n\nThe model did not suddenly become more intelligent because Backboard called it.\n\nThe system around the model got more of the existing intelligence into the result.\n\nThat distinction is the reason I care about the benchmark.\n\nPerformance was only half of it.\n\nOur full submitted Terminal-Bench run cost $280.72. At the time of the original article, the current verified leader had scored 83.8% at a reported cost of $552.67.\n\nSo our submitted result came in higher, while the reported run cost was roughly 49% lower. That changes the question.\n\nIt is easy to ask:\n\nWhich model are you using?\n\nIt is harder, and much more useful, to ask:\n\nWhat can you get that model to accomplish reliably, and what does it cost to get there?\n\nThose are different optimization problems.\n\nOne produces model comparison charts. The other produces systems you can afford to run.\n\nI have mixed feelings about AI leaderboards.\n\nThey are useful and extremely effective at turning otherwise sensible adults into people refreshing a webpage to see whether a decimal moved.\n\nFor us, the useful part is not the trophy. It is the failure data.\n\nThe submitted evaluation covered 89 tasks, with five attempts per task, for 445 trials under the same agent configuration. We included the failures and errors. The aggregate result was 85.4%.\n\nThat was the number we were comfortable submitting.\n\nEvery failed task tells us something.\n\nSometimes the model made a bad decision. Sometimes context was poor.\n\nSometimes the harness allowed an unproductive trajectory to continue too long.\n\nSometimes an implementation change that seemed clever turned out to be very clever at making the benchmark worse.\n\nUseful information, all of it.\n\nA benchmark is valuable when it tells your engineers where the system breaks. The leaderboard is what happens afterward.\n\nComparing AI systems gets messy fast. Change the model, provider, prompt, tools, or context strategy and you’ve changed the experiment.\n\nThat’s why I find the Claude comparison interesting.\n\nSame Claude Opus 4.8 on both sides. 6.5 points apart.\n\nI don’t think there’s one clever trick behind that gap. It’s probably a lot of fairly boring engineering choices adding up.\n\nWhich, unfortunately, is usually how software works.\n\nWe care about this for a bigger reason too.\n\nI don’t think companies should have to rebuild their AI stack every time a new model becomes the favorite.\n\nClaude might be the right fit today. Tomorrow it could be something else. Six months from now, it could be a model nobody has heard of yet.\n\nThe models will keep changing. That’s normal.\n\nThe infrastructure around them shouldn’t have to.\n\nYour context, memory, evaluation, application logic, and the lessons you’ve learned about turning model capability into useful work should be able to stick around.\n\nThat’s where I think a lot of the durable value in AI infrastructure will live.\n\nThis benchmark does not prove that models do not matter.\n\nThat would be a strange conclusion from an experiment powered by Claude Opus 4.8.\n\nIt proves something narrower.\n\nChoosing a strong model is not the end of the engineering problem.\n\nThere is still a substantial gap between having access to intelligence and building a system that uses it well.\n\nThe surrounding architecture determines what context reaches the model, what actions it can take, how failures are handled, and how much the whole thing costs when it finally succeeds.\n\nThose decisions are easy to dismiss individually.\n\nCollectively, they can move a benchmark by 6.5 points.\n\nFoundation models will keep getting better. They’ll keep changing too.\n\nIf several models can solve the same class of problems, simply having access to a good one becomes less of a differentiator.\n\nThe harder questions move up the stack:\n\nCan you make it reliable?\n\nCan you use context well?\n\nCan you recover when things go wrong?\n\nCan you tell whether an optimization actually helped?\n\nCan you do all of that without treating tokens like confetti?\n\nThat’s the engineering layer we’re betting on at Backboard.\n\nOur Terminal-Bench result is one piece of evidence, not proof of everything. It’s simply a useful controlled comparison:\n\n*The model stayed the same. The system changed. The outcome changed with it.*\n\nThat’s the part worth paying attention to.\n\n*This is a conversational remix of an article I published on [Backboard](http://backboard.io)’s blog. Read the original deep dive [here](https://backboard.io/blog/backboard-cli-beats-claude-code-terminal-bench).*", "url": "https://wpnews.pro/news/same-claude-different-harness-very-different-result", "canonical_source": "https://dev.to/robimbeault/same-claude-different-harness-very-different-result-1k58", "published_at": "2026-09-18 15:20:23+00:00", "updated_at": "2026-09-18 15:22:49.767453+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Claude Opus 4.8", "Claude Code", "Backboard CLI", "Terminal-Bench 2.1", "Amazon Bedrock", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/same-claude-different-harness-very-different-result", "markdown": "https://wpnews.pro/news/same-claude-different-harness-very-different-result.md", "text": "https://wpnews.pro/news/same-claude-different-harness-very-different-result.txt", "jsonld": "https://wpnews.pro/news/same-claude-different-harness-very-different-result.jsonld"}}