{"slug": "benchmarks-don-t-build-great-products-engineers-do", "title": "Benchmarks Don't Build Great Products. Engineers Do.", "summary": "Backboard, an AI startup, emphasizes that benchmarks should serve as tools for improving products rather than trophies for marketing. The company published an R-CLI result of 84.3% on Terminal Bench 2.1, surpassing Codex CLI and Claude Code, and advocates for transparent, reproducible evaluations. Backboard's approach prioritizes customer-centric benchmarking over score-chasing, aiming to keep engineers honest and foster a common language in the AI industry.", "body_md": "The AI industry has a benchmark problem. Not because we have too many benchmarks. Because too many companies treat them like trophies instead of tools.\n\nA benchmark's primary job is to make your product better. Publishing the score is secondary. The test I use: **a benchmark should challenge your engineers before it impresses your marketing team.**\n\nIf it isn't making your product better, it probably isn't serving its most important purpose.\n\nIf you've followed AI over the past year, you've seen an endless stream of benchmark announcements. Every week another model reaches the top of another leaderboard. Every release claims a new state of the art. Every company seems to have a chart proving they're the best.\n\nIt's easy to become cynical.\n\nThe problem isn't that benchmarks exist. Benchmarks are one of the most valuable engineering practices we have.\n\nThe problem is that we've started confusing the measurement with the mission.\n\nAt Backboard, we don't benchmark because we believe benchmarks are perfect. We benchmark because imperfect, transparent measurements are better than subjective claims.\n\nThat idea shapes how we think about engineering.\n\n**Benchmarks are feedback loops.** That's the whole point.\n\nA benchmark tells us where we're strong, where we're weak, and whether the changes we made actually improved something meaningful. Sometimes an optimization delivers exactly what we hoped for. Other times it exposes a regression we never expected.\n\nWithout objective evaluation, it's remarkably easy to convince yourself your product is getting better simply because you've spent weeks working on it.\n\nBenchmarks have a way of keeping engineers honest.\n\nThey also create a common language. Our customers shouldn't have to rely solely on our opinion of our own products. Public evaluations give everyone a shared point of reference. No benchmark captures every aspect of an AI system, but transparent measurements let different products be compared using the same criteria.\n\nThat's healthier than a world where every company simply declares itself the best.\n\nLike any measurement, they can be gamed. Four ways, all common:\n\nThey improve the marketing.\n\nThis is the distinction that matters, and the two can produce similar-looking leaderboard results while being fundamentally different engineering philosophies.\n\n| Building for benchmarks | Benchmarking what you built | |\n|---|---|---|\nStarts with |\nThe test | The customer |\nGoal |\nMaximize a score | Validate real improvement |\nOptimizes |\nThe evaluation set | The capability |\nWhen the score improves |\nShip the announcement | Check whether customers feel it |\nWhen the score drops |\nFind a better configuration | Find the regression |\nFailure mode |\nA number that never translates to value | A slower news cycle |\n\nBuilding for benchmarks starts with the test. The goal becomes maximizing a score, even if that improvement never translates into real world value.\n\nBenchmarking what you've built starts with the customer. You solve real problems first, then use independent evaluations to validate you're moving in the right direction.\n\nBuild first. Benchmark second.\n\nTransparency matters as much as performance. Whenever possible we publish our methodology, open source our evaluation frameworks, and share the logs, configurations, and results so others can reproduce our findings.\n\nIf someone discovers we've made a mistake, that's not a failure of the process. It's evidence the process is working.\n\nScience advances because results can be challenged. Engineering improves because assumptions are tested.\n\nHere's what that means concretely.\n\nIn July 2026 we published an R-CLI result of **84.3% (75 of 89 tasks) on Terminal Bench 2.1**, running Claude Opus 4.8 via Bedrock. That's above every published result we're aware of, including Codex CLI at 83.4% and Claude Code at 83.1%. On the same underlying model, R-CLI scored 5.4 points higher than Claude Code.\n\nWe also published **72% with GLM 5.2**, an open source model. That number is less flattering and we published it anyway, because a company that only shows you its best configuration is showing you marketing.\n\nWe did not claim a ranking. Terminal Bench isn't currently accepting submissions, so \"highest published result\" is accurate and \"number one on the leaderboard\" would not be. That distinction is small, and getting it right is the entire job.\n\nThe per-task verifier logs are public: [github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results](https://github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results)\n\nOur memory results, which lead LoCoMo and LongMemEval, are published the same way: [github.com/backboard-io](https://github.com/backboard-io)\n\nWe want people to inspect the work, reproduce it, challenge it, and improve on it. That creates far more confidence than posting a screenshot of a leaderboard ever could.\n\nNo benchmark will ever tell the entire story.\n\nThey don't measure customer trust. They don't measure usability. They don't capture every workflow or every edge case that matters to an enterprise.\n\nPublic benchmarks should always be complemented by real customer evaluations, production deployments, and continuous feedback.\n\nBenchmarks are one input. Not the only input.\n\nNo, and I'm happy to acknowledge it.\n\nBenchmarks create reach. Strong benchmark performance helps people discover what we're building. It starts conversations with engineers, customers, investors, and partners who otherwise might never have found us.\n\nThere's nothing wrong with that.\n\nWhat's important is the sequence:\n\n**What is the purpose of an AI benchmark?**\n\nTo create a feedback loop that tells engineers whether a change actually improved the product, and to give customers a shared point of reference instead of a vendor's opinion of itself.\n\n**How can you tell if a company is gaming a benchmark?**\n\nLook for published methodology, open evaluation frameworks, raw logs, and per-task results. Look for whether they publish their weaker configurations too. A single screenshot of a score is not evidence.\n\n**What's the difference between building for benchmarks and benchmarking what you built?**\n\nBuilding for benchmarks starts with the test and optimizes the score. Benchmarking what you built starts with the customer and uses evaluations to validate that real improvements happened.\n\n**Are benchmark scores enough to evaluate an AI system?**\n\nNo. Benchmarks don't measure trust, usability, or enterprise edge cases. They should be one input alongside customer evaluations and production feedback.\n\n**Why publish evaluation logs and not just scores?**\n\nBecause a score you can't reproduce is a claim, not a result. Logs let other people verify the work, and being corrected is how the process is supposed to function.\n\n**What has Backboard published?**\n\nR-CLI at 84.3% on Terminal Bench 2.1 with Claude Opus 4.8, and 72% with the open source GLM 5.2, both with per-task verifier logs on GitHub. Backboard's memory system leads LoCoMo and LongMemEval, published the same way.\n\nWe hope our products perform well on public evaluations. Of course we do.\n\nBut that's never been the goal.\n\nThe goal is to build software that genuinely helps people solve difficult problems. If we ever stop learning from benchmarks and start treating them as trophies, we'll have missed the point entirely.\n\nBecause in the end, benchmarks don't build great products.\n\nEngineers do.", "url": "https://wpnews.pro/news/benchmarks-don-t-build-great-products-engineers-do", "canonical_source": "https://dev.to/jon_at_backboardio/benchmarks-dont-build-great-products-engineers-do-49m7", "published_at": "2026-08-21 14:26:26+00:00", "updated_at": "2026-08-21 14:46:48.489284+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-research", "developer-tools"], "entities": ["Backboard", "R-CLI", "Terminal Bench 2.1", "Claude Opus 4.8", "Bedrock", "Codex CLI", "Claude Code", "GLM 5.2"], "alternates": {"html": "https://wpnews.pro/news/benchmarks-don-t-build-great-products-engineers-do", "markdown": "https://wpnews.pro/news/benchmarks-don-t-build-great-products-engineers-do.md", "text": "https://wpnews.pro/news/benchmarks-don-t-build-great-products-engineers-do.txt", "jsonld": "https://wpnews.pro/news/benchmarks-don-t-build-great-products-engineers-do.jsonld"}}