The AI industry has a benchmark problem. Not because we have too many benchmarks. Because too many companies treat them like trophies instead of tools.
A benchmark's primary job is to make your product better. Publishing the score is secondary. The test I use: a benchmark should challenge your engineers before it impresses your marketing team.
If it isn't making your product better, it probably isn't serving its most important purpose.
If you've followed AI over the past year, you've seen an endless stream of benchmark announcements. Every week another model reaches the top of another leaderboard. Every release claims a new state of the art. Every company seems to have a chart proving they're the best.
It's easy to become cynical.
The problem isn't that benchmarks exist. Benchmarks are one of the most valuable engineering practices we have.
The problem is that we've started confusing the measurement with the mission.
At Backboard, we don't benchmark because we believe benchmarks are perfect. We benchmark because imperfect, transparent measurements are better than subjective claims.
That idea shapes how we think about engineering.
Benchmarks are feedback loops. That's the whole point.
A benchmark tells us where we're strong, where we're weak, and whether the changes we made actually improved something meaningful. Sometimes an optimization delivers exactly what we hoped for. Other times it exposes a regression we never expected.
Without objective evaluation, it's remarkably easy to convince yourself your product is getting better simply because you've spent weeks working on it.
Benchmarks have a way of keeping engineers honest.
They also create a common language. Our customers shouldn't have to rely solely on our opinion of our own products. Public evaluations give everyone a shared point of reference. No benchmark captures every aspect of an AI system, but transparent measurements let different products be compared using the same criteria.
That's healthier than a world where every company simply declares itself the best.
Like any measurement, they can be gamed. Four ways, all common:
They improve the marketing.
This is the distinction that matters, and the two can produce similar-looking leaderboard results while being fundamentally different engineering philosophies.
| Building for benchmarks | Benchmarking what you built | |
|---|---|---|
| Starts with | ||
| The test | The customer | |
| Goal | ||
| Maximize a score | Validate real improvement | |
| Optimizes | ||
| The evaluation set | The capability | |
| When the score improves | ||
| Ship the announcement | Check whether customers feel it | |
| When the score drops | ||
| Find a better configuration | Find the regression | |
| Failure mode | ||
| A number that never translates to value | A slower news cycle |
Building for benchmarks starts with the test. The goal becomes maximizing a score, even if that improvement never translates into real world value.
Benchmarking what you've built starts with the customer. You solve real problems first, then use independent evaluations to validate you're moving in the right direction.
Build first. Benchmark second.
Transparency matters as much as performance. Whenever possible we publish our methodology, open source our evaluation frameworks, and share the logs, configurations, and results so others can reproduce our findings.
If someone discovers we've made a mistake, that's not a failure of the process. It's evidence the process is working. Science advances because results can be challenged. Engineering improves because assumptions are tested.
Here's what that means concretely.
In July 2026 we published an R-CLI result of 84.3% (75 of 89 tasks) on Terminal Bench 2.1, running Claude Opus 4.8 via Bedrock. That's above every published result we're aware of, including Codex CLI at 83.4% and Claude Code at 83.1%. On the same underlying model, R-CLI scored 5.4 points higher than Claude Code.
We also published 72% with GLM 5.2, an open source model. That number is less flattering and we published it anyway, because a company that only shows you its best configuration is showing you marketing.
We did not claim a ranking. Terminal Bench isn't currently accepting submissions, so "highest published result" is accurate and "number one on the leaderboard" would not be. That distinction is small, and getting it right is the entire job.
The per-task verifier logs are public: [github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results](https://github.com/Backboard-io/Backboard-R-CLI-Terminal-Bench-2.1-Results)
Our memory results, which lead LoCoMo and LongMemEval, are published the same way: [github.com/backboard-io](https://github.com/backboard-io)
We want people to inspect the work, reproduce it, challenge it, and improve on it. That creates far more confidence than posting a screenshot of a leaderboard ever could.
No benchmark will ever tell the entire story.
They don't measure customer trust. They don't measure usability. They don't capture every workflow or every edge case that matters to an enterprise.
Public benchmarks should always be complemented by real customer evaluations, production deployments, and continuous feedback. Benchmarks are one input. Not the only input.
No, and I'm happy to acknowledge it.
Benchmarks create reach. Strong benchmark performance helps people discover what we're building. It starts conversations with engineers, customers, investors, and partners who otherwise might never have found us.
There's nothing wrong with that.
What's important is the sequence:
What is the purpose of an AI benchmark?
To create a feedback loop that tells engineers whether a change actually improved the product, and to give customers a shared point of reference instead of a vendor's opinion of itself.
How can you tell if a company is gaming a benchmark?
Look for published methodology, open evaluation frameworks, raw logs, and per-task results. Look for whether they publish their weaker configurations too. A single screenshot of a score is not evidence.
What's the difference between building for benchmarks and benchmarking what you built?
Building for benchmarks starts with the test and optimizes the score. Benchmarking what you built starts with the customer and uses evaluations to validate that real improvements happened.
Are benchmark scores enough to evaluate an AI system?
No. Benchmarks don't measure trust, usability, or enterprise edge cases. They should be one input alongside customer evaluations and production feedback.
Why publish evaluation logs and not just scores?
Because a score you can't reproduce is a claim, not a result. Logs let other people verify the work, and being corrected is how the process is supposed to function.
What has Backboard published?
R-CLI at 84.3% on Terminal Bench 2.1 with Claude Opus 4.8, and 72% with the open source GLM 5.2, both with per-task verifier logs on GitHub. Backboard's memory system leads LoCoMo and LongMemEval, published the same way.
We hope our products perform well on public evaluations. Of course we do.
But that's never been the goal.
The goal is to build software that genuinely helps people solve difficult problems. If we ever stop learning from benchmarks and start treating them as trophies, we'll have missed the point entirely.
Because in the end, benchmarks don't build great products.
Engineers do.