Every LLM launch announcement comes with a similar slide: a wall of benchmark scores, each a point or two higher than last month’s. Those numbers matter but they are also about someone else’s problem.
Plus they make you pay for capabilities you probably don’t need.
You write your code, you own the unit tests. You leverage an LLM, you should own the benchmark too. Same ownership.
Writing the first cases is easy. The hard part is structuring them so the benchmark keeps evolving when you have a hundred cases, when a new model ships every month, and when someone asks later why a model failed a particular case.
This post is about creating your own benchmark and running your own harness.
First, a quick tour of the most common ones. Good ideas are worth borrowing.
Hall of Fame of LLM Benchmarks #
These are some of the most common ones:
- MMLU-Pro : About 12,000 multiple-choice questions across 14 subjects, from law to engineering, with ten options per question instead of the original MMLU’s four.
- SWE-bench : Real GitHub issues from real open-source projects. The model has to write a fix, and the project’s own tests decide whether it worked. Its Verified subset was checked by human engineers to make sure every task is actually solvable.
- Humanity’s Last Exam : Around 2,500 questions written by subject-matter experts around the world, kept only if the best models of the time got them wrong. It covers mathematics, science and the humanities at the edge of what specialists know.
- ARC-AGI : Small visual grid puzzles that most people solve in seconds and models find surprisingly hard. It tests whether a system can adapt to a task it has never seen.
These are all great benchmarks but have one thing in common: they don’t evaluate your specific problem.
So you want to design your own benchmark #
1. Determinism triumphs.
Probably the most important rule to keep in mind: the tests should be as deterministic as possible. Everything that can be validated programmatically should live as a standard test (think of a traditional assert).
Using an LLM as a judge is tempting. It also adds noise, cost and bias, and gives you a second model whose accuracy you’d need to benchmark too. For most real tasks you can avoid it by asking for the right kind of answer. A case passes only on an exact match: every expected item, nothing extra, on specific data items.
Require a structured answer in which every finding cites something specific: a record, a file, a line, a cell. The scorer compares those citations with the expected list. A case passes only on an exact match: every expected item, each from the right source. Some examples:
- Deterministic checks: the LLM output has a certain JSON structure, the test run within X number of steps or the LLM found a certain ID in the data.
- Questions that require an LLM: the summary of the findings is aligned with the evidence, the LLM followed a certain reasoning chain.
2. Give every test the same shape: the harness.
Reusability starts with one separation: the harness is code and the tests are content. The harness (the agent, its tools, the runner, the results summary and a results dashboard) is written once and knows nothing about any particular test. Just the results.
Tests and cases are just folders of data that follow a fixed shape. Adding a test should never mean touching the harness.
The harness runs all the test families and aggregates results into a dashboard: Anthropic’s prompt chaining pattern.
Each test family maintains its own scorer, cases and instruction set. The task itself and specific case input is provided at case level. The goal is to encapsulate tests as much as possible.
This is a sample agent input file. It defines the core of the task. Notice that it’s restricted to the particular case resources: in this case verifying a certain result value using files available in the evidence folder.
Also, put a canary string in every case: a unique marker that asks AI labs to keep the data out of training and lets you check later whether a model has seen it. The model itself should never see it.
A canary ID lets you check whether a model has used off-limits data as part of its reasoning process.
3. Define a case card with metadata information
Maintain a metadata file for each case defining the case itself. Things like case difficulty, tags, and steps to reproduce the test itself.
You should be able to manually validate the test checks step by step if needed.
When creating the benchmark, make sure you maintain the metadata accurate, and that it defines the list of instructions to manually run through the checks.
Some benchmarks refer to such file as meta.json, and you should have one per case. The model under evaluation does not see this file. The harness can use it to classify results by tags and difficulty. It’s also a good place to provide the list of steps to reproduce the tests manually.
4. Structure the tests to run independently and save budget.
Most of the tests will require a reasoning LLM which means you should be able to reuse results as much as possible. A new model, test or scenario added to your benchmark shouldn’t require you to rerun everything again.
This is an example of how to structure the tests, used in some benchmarks. A central benchmark folder containing the entire benchmark, with specific folders for each test class and cases.
Such structure allows the harness to run tests independently, by family and case and consolidate results with existing tests to avoid executing the entire benchmark.
npm run my-benchmark —test=txt-matching —case=case-03
5. Design to add new models at any time
The LLM landscape changes every week so at some point you’ll consider testing with a new model.
Include this point into your design so you can run the tests with the new model without rerunning everything. A valid option is to run based on a certain model ID.
This example runs a benchmark subset using only [DeepSeek V4.1 Flash model](https://www.deepseek.com/en/news/deepseek-v4-1-flash/) (an excellent model btw):
*npm run my-benchmark —test=txt-matching —case=case-03 —model=ds-flash*
You can add flexibility to this. Define model Ids by provider and reasoning capability.
The following input runs the benchmark using [OpenAI Sol Medium](https://openai.com/index/introducing-gpt-6-1-sol/) (one of my favourites too):
*npm run my-benchmark —test=txt-matching —case=case-03 —model=oa-sol-medium*
6. Make the Output a Webpage
Enough of plain, raw JSON output. There is no reason to not deliver a nice dashboard.
A table of averages in a console gets read once and scrolled away. An HTML page can be bookmarked, shared, argued about and acted on, and it shows patterns that a column of numbers hides.
It costs almost nothing. One static HTML file that reads one summary file needs no server, no build step and no account.
The same models you’re testing will happily draft the charts for you.
7. Deliver your own Pareto frontier
Every answer costs money and time, and a model that’s two points better at ten times the price is rarely the right choice.
The Pareto frontier captures that trade-off: the set of models that no other model beats on both axes at once. Every model off the frontier is dominated: some other option is at least as accurate and cheaper. Providing the frontier line is now almost an industry standard.
8. Enable caching on your model requests
Last but not least, this is the single most important factor towards reducing costs in your tests.
Enable caching to reuse the input prompt in your tests, which is reused for each test case over and over.
This changes by model (e.g DeepSeek, OpenAI) provider but typically involves two steps:
- Enable caching in the API requests.
- Configure the prompt in a certain way.
Any generic LLM will check the caching API documents and generate the right code for you.
Conclusion
This is the end of the tips on creating your own benchmark and how to run it efficiently.
In the next article I’ll cover a concrete example. You may have noticed the sample benchmark I used on the screenshots, I’ll review it in detail on the next article.
Own your benchmark!