Frontier-Bench

The team behind Terminal-Bench and Harbor released Frontier-Bench v0.1, a continuous benchmark with 74 tasks across 7 domains that measures agent abilities at the frontier, with the best models achieving approximately 34% accuracy. Frontier-Bench introduces CI/CD, semantic versioning, and result migrations to address saturation and representation issues in existing benchmarks.

Frontier-Bench Frontier-Bench measures agent abilities at the frontier. Built by the team behind Terminal-Bench https://www.tbench.ai/ and Harbor https://www.harborframework.com/ , Frontier-Bench which started out as Terminal-Bench 3.0 is an on-going large-scale community effort to source the most diverse, difficult and high-quality tasks. Our first release contains 74 tasks across 7 domains https://github.com/harbor-framework/frontier-bench task-distribution . The best models achieve ~34% on Frontier-Bench v0.1. Frontier-Bench is a continuous benchmark. The frontier is wide and dynamic, Frontier-Bench evolves along with it. We are already developing https://github.com/harbor-framework/frontier-bench/issues/1422 subsequent releases, with new and improved tasks. Overview overview Frontier-Bench expands on Terminal-Bench with more challenging tasks across a wider distribution of valuable work. Frontier-Bench also introduces a new way to continuously improve benchmarks with CI/CD, semantic versioning, and result migrations. We need a harder benchmark. Many Terminal-Bench tasks have become saturated and leaderboard entries are condensed into a narrow range that may not fully represent capability gaps across models. We need a more representative benchmark. Terminal-Bench was built on the thesis that code and the CLI were powerful tools for agents to accomplish tasks. Frontier-Bench expands on that idea by introducing more task diversity to measure performance across more types of coding and non-coding work. "Work" has expanded to longer-running tasks across a greater diversity of domains and "computer" has expanded to richer environments, multi-container networks, and GPUs. We anticipate this definition continuing to broaden in future Frontier-Bench versions. We're fixing the way benchmarks are built. Many benchmarks have known problems, many of these problems go unfixed. It erodes the usefulness of the benchmark and also dilutes the signal. Frontier-Bench is a continuously improving benchmark and open community project https://github.com/harbor-framework/frontier-bench . We will regularly release updates to fix and add tasks, making the benchmark more valuable over time. Tasks tasks Frontier-Bench was built using our learnings from building Terminal-Bench. Each task goes through a rigorous review process with dense human and agent feedback at each step. Frontier agent, oracle agent, and cheating agent runs ensure that the task is well defined. Domain experts review tasks to probe instruction/verifier alignment, task realism, and manage under/over specification risks. Frontier-Bench is an open internet benchmark. Capabilities are best demonstrated when agents have access to their full set of tools. We instruct agents not to look for task-specific solutions or hints online, which is surprisingly effective. We also tell the agent how long it has to complete the task. We separate the agent container and verifier container. Artifacts from the agent container are downloaded at the end of the trial, logged, and uploaded to the verifier container. This prevents many reward hacking vectors allowing for re-grading when task verifiers are updated. The diversity of artifact filetypes reflects the expansion into new domains: - Database snapshots .sql , pg.dump , redis.rdb , Kafka .tgz - ML checkpoints .pt , .safetensors - RTL designs .v - Formal proofs Lean .lean , Coq .v - Source code and patches Python, C, CUDA, TypeScript, etc. - VM images .qcow2 , .fd - Compiled binaries .jar , ELF executables - Music scores .mxl , .musicxml - CAD files .step , .FCStd - Scientific analyses .json , .csv , .tsv - Quantitative computations .json , .csv , .xlsx - Reverse-engineered specs .json - Business-process decisions .json , .jsonl Results results Frontier-Bench discriminates better between frontier and sub-frontier models than Terminal-Bench 2.1. For example, Fable 5 in Claude Code and Opus 4.8 in Claude Code are separated by only 4.9 points on Terminal-Bench v2.1, but 12.7 points on Frontier-Bench v0.1. While frontier models Fable 5 and GPT-5.6 Sol achieve similar performance on Frontier-Bench v0.1, cost and token usage varies significantly. GPT-5.6 Sol is ~40% cheaper and uses ~50% fewer tokens than Fable 5. All of the GPT-5.6 models Sol, Terra, Luna appear on the cost vs pass rate Pareto front for the models we ran. 1 user-content-fn-1 Agents approach problem solving differently, even at similar pass rates. Fable 5 33.8% and Opus 4.8 21.1% use more tokens and take fewer actions. Meanwhile, GPT-5.6 Sol 34.4% and Terra 20.8% use less tokens and take more actions. Roadmap roadmap The Frontier-Bench team is already planning upcoming releases, which we are documenting in our public roadmap https://github.com/harbor-framework/frontier-bench/issues/1422 . In our next major release, we plan to add new tasks that measure new capabilities, add greater task diversity, and tune timeouts. Tasks are versioned so corresponding trials can be re-used, re-graded, or re-run to minimize the cost and complexity of updating leaderboard results. All of these features are simple commands in Harbor https://www.harborframework.com/ . Conclusion conclusion Frontier-Bench is a community effort from 100+ task contributors and reviewers. It is the hardest and highest quality benchmark we could make together in the open, but it is just the start. Let's find issues, fix them, and add new tasks together. Let's make this benchmark measure the most valuable frontier capabilities in the most accurate way possible. Join the effort on Discord https://discord.com/invite/ZvcWupVXjz and GitHub https://github.com/harbor-framework/frontier-bench . Let's continuously and rigorously define the frontier together. Acknowledgements acknowledgements Thank you to all the task contributors and reviewers /contributors . Special thanks to our senior reviewers Ivan Bercovich, Benedikt Droste, Tommaso Cerruti, Steven Dillmann, Ruiyang Wang, Dariush Wahdany, Allen Hart, and Karl Krauth and advisors Mike Merrill and Nicholas Carlini. We appreciate our sponsors who made this project possible including Modal https://modal.com/ , Anthropic https://www.anthropic.com/ , OpenAI https://openai.com/ , Google https://ai.google/ , Scale AI https://github.com/scaleapi , Snorkel Open Benchmarks https://benchmarks.snorkel.ai/ , Turing https://github.com/TuringEnterprises , gNucleus AI https://github.com/gNucleus-AI , Boolean AI https://www.boolean.ai/ , and Handshake AI https://github.com/Handshake-AI-Research . Frontier-Bench is hosted by Harbor https://www.harborframework.com/ and Laude Institute https://www.laude.org/ . Written by: Ryan Marten, Alex Shaw, Andy Konwinski Footnotes footnote-label - Grok 4.5 was removed from this chart due to a cost reporting issue with the Cursor CLI harness. ↩ user-content-fnref-1