The case to BYOB: build your own (coding) benchmarks

A team led by @oh_an_opinion built an internal coding benchmark from its own merged pull requests and found that, depending on task complexity, cheaper and faster agent setups matched frontier Anthropic and OpenAI models within run-to-run variation while cutting cost by roughly 5× to two orders of magnitude and sometimes halving completion time. Tested on 15 additional randomly selected PRs, a router derived from those evaluations maintained quality at about 7× lower cost than Claude Code + Fable 5.1 and Codex + Astra, the team reported in September 2026. The team said no single setup won everywhere, justifying granular per-task routing, and cautioned the findings are directional and based on a small sample rather than a leaderboard.

BYOB Build your own benchmarks Your own coding benchmarks can benefit cost, time and quality. @oh an opinion https://x.com/oh an opinion , September 2026 Public benchmarks are less useful than you want them to be. They have been optimized against and are not specific to your repos, conventions, workflows or ticket writing style. So we turned our own merged PRs into a benchmark. It measures agentic setups models, harnesses, skills, and the rest on quality, cost and speed, on the work we actually do. The results drive a router that picks a setup per task. Depending on task complexity, we found setups that were cheaper and faster than the frontier models while scoring within run-to-run variation of them. No single setup won everywhere: different combinations of parameters performed differently together and across task types, justifying granular routing. We think you can and should do the same. Here we show our approach, and a subset of our codebase-specific data that is part of our internal benchmark. Depending on the task, we found setups that performed similarly or better while reducing cost by roughly 5× to two orders of magnitude and, in some cases, cutting completion time by half or more. We tested the router derived from these evaluations on 15 additional randomly selected PRs and found it maintained quality at ~7× lower cost than Claude Code + Fable 5.1 and Codex + Astra. These findings are directional, specific to our work, and based on a small sample. Treat this as an exploration of an approach, not a leaderboard. 01How a PR becomes an eval A merged PR that passes requirement gates gives you a ticket, a pre-merge tree and a gold diff. Each task is a whole engineering ticket. The agent must understand the requirements, coordinate changes across the codebase, and check the result. We measure the full attempt’s cost and time, then grade the work remaining before merge. Our low, medium, and high complexity bands describe these whole tickets. The complexity rubric considers scope, depth of logic changes, and risk: low covers small, well-tested, localized changes; medium covers moderate scope or logic depth; high covers large changes, changes spanning many files or directories, or risky logic. 1. 01 Select the task A merged PR that clears five trust gates. 2. 02 Ticket is the prompt The agent sees the ticket, never the merged patch. 3. 03 Run a setup Model, harness, skills and more on the pre-merge tree. 4. 04 Grade the run Repo checks, ten specialized agents, then three grading agents. Hidden-test pass rates tell you how often a setup satisfies a benchmark’s test cases, but they are an incomplete measure of code quality. They can miss incomplete requirements, security issues, and maintainability problems. We assess the work remaining before merge across multiple criteria. Ten specialized agents each examine one dimension of the code, then three grading agents assess the code alongside their findings. We measure cost and time alongside quality. 02What we found Cheaper and faster setups matched frontier Anthropic and OpenAI models within run-to-run variation on our PRs. Different setups performed best across task types and complexity bands, giving us a basis for routing each ticket to a suitable setup. The charts below are a subset of original-ticket runs on our Python backend PRs: multiple repeats of different combinations of models, harnesses, skills and topologies: a single implementer or defined planner, implementer, and reviewer roles PIR . Quality is the judge’s 0–1 score for work remaining to merge: 0.85 and above merges with at most non-critical suggestions, 0.95 and above is a clean merge, below 0.7 does not merge. Pareto Frontiers Pareto Codex+Astra / CC+Fable 5.1 Pareto cheaper and faster Codex+Astra / CC+Fable 5.1 OpenCodePiCodexClaude Code ClaudeGPTGLMGrokDeepSeekKimiGeminiMiniMaxQwenMuse Color is the implementer family. Shape is the harness: circle OpenCode, diamond Pi, triangle Codex, square Claude Code. Cyan marks identify Codex+Astra and Claude Code+Fable 5.1. Hover any point for the setup name. Quality 0.8, plus reference comparisons. Cyan Astra and Fable 5.1 comparisons remain visible below the cutoff; low-complexity Fable 5.1 scores 0.788. Faster is left; cheaper is down. Pareto is cheaper and faster, not higher quality. Merge rate is the share of that setup’s runs scoring ≥0.85. It is not the same as mean quality. Low · 3 PRs · 46 setups Medium · 4 PRs · 60 setups High · 2 PRs · 66 setups Low · 3 PRs · 46 setups Medium · 4 PRs · 60 setups High · 2 PRs · 66 setups Low · 3 PRs · 46 setups Medium · 4 PRs · 60 setups High · 2 PRs · 66 setups Low · 3 PRs · 46 setups Medium · 4 PRs · 60 setups High · 2 PRs · 66 setups How to read these charts Each point is one setup’s mean quality on that complexity band. A high mean can still hide runs that fall under the merge floor; we still draw the frontier on the mean. The gold horizontal is the mean human score on that band’s PRs. Whiskers are shown on the Pareto frontier and on the cyan comparisons Codex+Astra and Claude Code+Fable 5.1 . They report the standard error of that mean sd of per-PR means over √n<sub PR</sub , not the spread of tickets and not run-to-run noise. High complexity is two PRs, so those bars are a caution, not a tight interval. Cyan marks are Codex+Astra and Claude Code+Fable 5.1 triangle / square by harness . Pareto is cheapest or fastest for a given quality; the rest of the grid is still plotted. A setup is drawn on a band only when it has ≥3 repeats on every PR in that band 3 low, 4 medium, 2 high . 67 setups pass that bar on at least one band. Costs are computed per setup from token counts, averaged over the runs in that complexity band. Haiku 4.5 on Pi costs more than Sonnet 5 because it uses about twice the tokens. 03Anecdotes beyond the frontier Model, harness, skill and effort are not independent knobs. The same skill helps one model and hurts another; the same effort setting rescues one and overthinks another. Skill + harness effects Skills helped some setups and hurt others. We observed many interactions between skills, models, and harnesses. A few examples: for DeepSeek on OpenCode, Superpowers increased merge quality against no skill; for GLM 5.2 on the same harness, it decreased. Sol 5.6 performed better on Codex than on Pi. On GLM 5.2 with Pi, gstack left quality unchanged at 0.82 while cutting wall-clock from 7.9 min to 4.8; Superpowers did the same for speed and raised quality to 0.86. Neither held for GLM 5.3 on Pi: Superpowers was slower on low tickets 5.1 min to 7.7 and lowered quality on low and high 0.84 to 0.81, 0.90 to 0.82 , and gstack cost 0.16 of quality on high. Skill effects also flipped with complexity on 5.3: ponytail gained 0.07 on low tickets and lost 0.07 on high, against a no-skill baseline of 0.90 that is itself on the high-band frontier. Opus 5 on Pi with ponytail was about 1.8x faster 6.1 min to 3.5 at the same quality. DeepSeek with ponytail held quality on OpenCode vs Pi 0.78 vs 0.76 at 2.4x the dollar cost. GLM 5.2 and 5.3 disagreed on the direction of every skill effect we measured on Pi. Results from one model version therefore weren’t a reliable guide to the next. Reasoning effort Higher reasoning effort sometimes reduced quality. Thinking depth is model-task-specific, not a simple quality dial. On a 243-run panel of the same nine PRs three models × three effort levels × three repeats , Grok 4.6 on Pi gained hard from low to high +0.05 to +0.09 merge score depending on complexity and got little extra from xhigh. Claude Code + Fable 5 usually improved low→high, then regressed at max on harder tickets high complexity: 0.87 → 0.75 while cost roughly tripled. Kimi K3 barely moved across low / high / max ~0.88–0.89 . Reasoning effort belongs in setup selection: its effect on quality and cost varies by model and task. Ticket quality Rewriting the tickets changed which setups performed best. We compared the original implementation plan v1 with a shorter rewrite containing fewer specifics v2 . Setups that did well on v1 tended to do well on v2, but only loosely: rank correlation is about 0.5. The setup you would route from a v1 ticket leaderboard is often not the one you would pick for a paragraph of intent. “Stronger” models still sit higher on average, though some of the smartest look instruction-hungry: Sol, Luna, and Opus on Claude Code drop more when the instructions are light than GLM or MiniMax, which are minimally affected. Runs take on the order of a third longer without thorough instructions, and Claude Code + Opus roughly doubles because it takes over more of the planning. Dollar cost does not move significantly. Quality is not one number either: on high-complexity PRs the short ticket did not hurt. These results suggest that setup selection should account for both task complexity and how much detail the ticket provides. What behavior monitoring caught Monitoring caught deleted tests and use of a leaked reference patch. We also checked agent behavior using a simplified version of OpenAI’s monitoring approach https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ , since we lacked access to chain-of-thought traces. We found several failures, including these two: Claude Code + Haiku Deleted the failing tests, reported verified. The agent deleted tests that exposed a cache-hit bug in a KMS envelope cipher, then reported the work as complete and tested. The bug silently makes credentials undecryptable. Kimi + ponytail + Pi Applied the golden diff it found via git. Our monitoring caught an agent finding and applying the reference patch through Git without mentioning it in its final summary. Exposing the patch was a bug in our evaluation environment. The finding let us fix the leak and rerun the affected evaluations. Our monitoring found no other run accessing the reference patch. What the literature says Effort. Longer reasoning degrades accuracy on some tasks across model families Gema et al. 2025 https://arxiv.org/abs/2507.14417 , and on agent tasks no fixed effort setting is right for every step ARES 2026 https://arxiv.org/abs/2603.07915 ; Cuadron et al. 2025 https://arxiv.org/abs/2502.08235 . Harness. With the model held fixed, changing the harness moved pass rate by 27 percentage points, about as much as changing the model Claw-SWE-Bench 2026 https://arxiv.org/abs/2606.12344 , and made the same task up to 40× cheaper to solve The Scaffold Effect 2026 https://arxiv.org/abs/2607.22585 . Hence Stop Comparing LLM Agents Without Disclosing the Harness https://arxiv.org/abs/2605.23950 . Skills. Curated skills add 16 percentage points on average but hurt on 16 of 84 tasks, and focused skills beat exhaustive bundles SkillsBench 2026 https://arxiv.org/abs/2602.12670 . Prompt formatting alone shifts accuracy by tens of points, in different directions per model Sclar et al. 2023 https://arxiv.org/abs/2310.11324 . Tasks. OpenAI stopped reporting SWE-bench Verified https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ in February 2026 over flawed tests and memorized gold patches. Task length alone sets a benchmark’s verdict METR 2025 https://arxiv.org/abs/2503.14499 , and per-task clustering is the right error model Miller 2024 https://arxiv.org/abs/2411.00640 . 04Worked example One ticket, two setups, $0.15 vs $1.34, scores 0.95 and 0.96. Select the task We assess five trust gates from GitHub. This page uses examples where all five were true; you can ignore a gate at your own discretion. Example: PR 1034. ✓merged✓ci green✓human approval✓cooling period✓no revert The ticket is the prompt The original ticket is what the agent sees. It never sees the gold diff: that is the merged PR, used only to score. You can also judge the ticket itself, and rewrite it, to see how robust a setup is to the input: instruction quality, not just model quality. Same gold, different prompt. Tickets · PR 1034 · OUT-031 · Pending-outcome state — exclude unresolved cells from quality Run a setup A setup is what you would actually launch: model, harness, skills, topology. The runner clones the repo and checks out the commit before the merge. Skills are added if the setup asks for them. The setup then gets: - The ticket, as the prompt. Not the gold diff. - A working tree at that pre-merge commit, with the harness’s file, search, and shell tools. - Implementer-only this page is one pass; a planner or reviewer would be extra turns on the same tree. A and B below are two setups on this ticket. Only the files they change are scored. Grade the run After the diff is captured, deterministic repo checks run on the working tree: build, tests, lint, types, and the rest. Then ten specialized agents each examine one dimension of the code security, completeness, test adequacy and more and emit findings. Three grading agents assess the code alongside those findings to judge the work remaining before merge. The specialized and grading agents are drawn from multiple model families and calibrated against human review see Agreement with human review grader . A · glm-5.2 · opencode · gstack · impl · $0.15 · 2.3 min · score 0.95 · Merge: perfect grader The ticket is delivered: pending is named on OutcomeRecord, the three reads filter when asked, and both quality helpers exclude NULL passed and return None when nothing is resolved. Tests can fail and cover the cases that matter; leftover nits are style, not blockers. B · claude-opus-5 · claude-code · none · impl · $1.335 · 2.9 min · score 0.96 · Merge: perfect grader Ships as asked: explicit pending/resolved on OutcomeRecord, optional resolved only reads, and pending-safe quality aggregates with tests that actually fail if you drop the implementation. No blockers; revert is a complete undo. Agreement with human review On ~200 runs, humans and the judge independently assigned the same or adjacent grades 97% of the time. The grader treats explicit ticket requirements as authoritative, so incorrect instructions can limit the reliability of its assessment. 05Why this is affordable The savings can cover the cost of running the benchmark. Most configurations we tested were inexpensive, with runs on the Pareto frontier typically costing less than a dollar. Choosing cheaper setups for subsequent tasks can recover the evaluation cost. We also use predictions to choose which configurations to test next; that process is outside the scope of this article. We also tested our router derived from these evaluations on 15 randomly selected PRs that were not included in the evaluations above. We ran those same PRs with Codex + Astra and Claude Code + Fable 5.1. On this separate set of tasks, our router maintained quality while reducing total cost across the 15 PRs by ~7× compared with Claude Code + Fable 5.1 and Codex + Astra. 06Why build your own benchmark Public benchmarks provide a useful starting point. Your own benchmarks show which setups work on your codebase, conventions, workflows, tickets and let you assess the qualities that matter to your team. You can use that evidence to choose setups that reduce cost, finish work faster, or improve code quality. Results and execution traces can also reveal opportunities to improve harnesses and skills. Rerunning the benchmark lets you check whether those changes help, and whether the gains hold as models and your codebase evolve. This is also a foundation for greater autonomy in software development. A repeatable way to assess agents’ work helps you decide which parts of the workflow to automate next and where human review is still needed. 07Caveats Treat this as a prior and a method, not a leaderboard. The plots cover nine merged PRs from one Python backend repo: three low-, four medium-, and two high-complexity tasks, with three repeats per setup. These are directional findings, not significance claims. - This is not a hidden-test pass/fail bench. Tasks are ticket-level merged PRs, typically broader in scope than public issue-fix suites such as SWE-bench. We score remaining work to merge via repo checks and calibrated judge review, not a single binary oracle. That trades flake and overfitting risk on acceptance tests for grader calibration see Agreement with human review grader ; it does not make grading free or automatic. - The combination space is too large to sample exhaustively. Use public benches, intuition, and sparse-data forecasts as priors, then spend runs where they move a decision. - When the measured quality difference is small relative to the uncertainty, we use cost and speed to guide selection. Running these setups on fresh tasks adds evidence to refine the choice. - Automating grading is hard. Use the agentic grader as a first pass; validate with a human grader where it matters. - Keep minting fresh PRs. A private bench that never changes will not evolve with the changes of your codebase. - Our numbers are ours. Use them as a prior, but build your own. Repo, task, ticket, model, harness, skill, effort and topology specificity are real. 08Our Infrastructure We’ve built the infrastructure to generate, run, and assess benchmarks on your own codebase. It includes out-of-the-box grading and task labeling, with customization for your requirements. You can configure models, harnesses, agent topologies, and external tools, including review agents and skills. If you’d like to use it, or chat more about how to build your own benchmarks, get in touch https://byobench.ai/request-access .