FrontierCode LeaderboardEvaluating models against production code standardsView Now
Recent progress in AI has made software engineers dramatically more productive. However, our experience building and deploying Devin suggests that agentic software engineering remains far from solved. Models struggle to infer user intent, respect unstated constraints, and anticipate long-term consequences of their changes, forcing human intervention.
Most existing coding benchmarks fail to capture these gaps and tend to be saturated quickly. We are excited to introduce FrontierCode 2.0, a new generation of our coding benchmark series and our best attempt yet to evaluate user experience.
In FrontierCode 1.0, elite open-source maintainers crafted rigorous criteria to measure code mergeability. FrontierCode 2.0 builds on this methodology with several key improvements:
- Broader coverage : the tasks are drawn from more repositories and span more categories, including security patches and performance optimizations, to better reflect the real distribution of coding work.
- Higher mergeability standards : grading criteria impose more extensive correctness and design requirements, with added emphasis on thoroughness.
- Larger task scope : compared to FrontierCode 1.1, reference solutions modify 67% more files and 89% more lines of code on average, in codebases that are 25% larger.
Results #
The Pareto frontier includes Opus 5.5 on the high end along with Luna, Haiku, and Sol on the low-cost end, with GPT-6.1 Sol offering particularly strong performance relative to its cost. Our results also highlight significant room for improvement for open source models, even more so for western OSS models.
FrontierCode 1.1 vs. FrontierCode 2.0
| FrontierCode 1.1 vs. FrontierCode 2.0 | |||
|---|---|---|---|
| Category | FrontierCode 1.1 | FrontierCode 2.0 | Change |
| --- | --- | --- | --- |
| Claude Opus 5.5 | 54.6% | 45.5% | |
| GPT-6 Astra | 53.3% | 42.8% | |
| Grok 4.7 | 47.6% | 28.6% | |
| Gemini 4 Argon | 47.5% | 26.3% | |
| Muse Wonder 1.0 | 47.0% | 24.3% | |
| SWE-2 | 50.0% | 23.9% | |
| DeepSeek V4.1 Flash | 43.9% | 19.3% | |
| Kimi K3 | 44.2% | 17.7% |
Compared with benchmarks like DeepSWE v1.1 and FrontierCode 1.1, FrontierCode 2.0 separates models more clearly across tiers of capability. The benchmark is far from saturated: the top-performing model, Opus 5.5, scores only 45%.
What FrontierCode measures #
Models have become very good at following extensive, careful problem specs. Indeed, this is exactly what most coding benchmarks test for. As an example, consider the following excerpt from vf2-speedup-networkx in Terminal-Bench 4.0.
The task fully specifies the approach and style that the agent should use, rather than requiring any inference from the environment.
While Terminal-Bench 4.0 is a great benchmark for isolating pure model intelligence, we also try to capture other components of model capability. As models have become stronger, users increasingly want them to work autonomously. Ideally, a coding agent should take an informal spec, find a course of action that satisfies both the spec and the constraints and preferences it implies, account for the user’s blind spots, and execute without repeated corrections or clarifications. What unstated constraints and preferences arise in a typical engineering context? Recurring examples include reusing existing abstractions, following repository conventions, and writing nontrivial tests that actually catch bugs. These are important to ensure that code is maintainable in the long term and understandable to the user.
With these and other considerations in mind, FrontierCode attempts to measure the following:
- Functional correctness
- Interpreting and expanding on a user’s initial prompt
- Making reasonable design and scoping choices for implementation and smooth integration with an existing codebase
How we built FrontierCode 2.0
FrontierCode 2.0 largely retains the methodology of FrontierCode 1.0. More than 20 world-class open-source developers each spent over 40 hours per task building realistic coding evals from the repositories they maintain. Our goal is to measure how well models would serve as real independent contributors, using carefully calibrated grading criteria that test whether agent output is actually mergeable. For more on our grading methods and quality control, see our FrontierCode 1.0 blog post.
As discussed above, FrontierCode 2.0 mainly improves upon FrontierCode 1.0 through increased task diversity, grading criteria, and task scope. In more detail:
- Our tasks are drawn from 55% more repositories in order to better capture the distribution of real coding work.
- Tasks include 43% more blocking criteria, which capture correctness and design requirements with greater specificity.
- Tasks tend to require larger changes to the codebase.
We also strengthened our quality control by auditing task prompts, reference solutions, and grading criteria more rigorously. This investment in both human and automated review frequently surfaced subtle edge cases in reference solutions taken from upstream repositories. In 80% of FrontierCode 2.0 tasks, the reference solution differs from the upstream PR, up from 60% in FrontierCode 1.0. This extra care matters more as FrontierCode expands into security-related tasks, where exploits often hinge on small details. In one of our automated audit pipelines, an agent discovered a security vulnerability in the upstream implementation; after human review, we reported it to the repository maintainers.
Evaluating thoroughness
When classifying our grading criteria, we identified a “thoroughness” cluster. Thoroughness criteria enforce correctness conditions that an agent can be expected to infer from the instructions and the existing codebase, even when they aren’t stated explicitly. They are central to FrontierCode 2.0’s ability to evaluate agents under incomplete prompting, because they catch solutions that superficially address the prompt but leave or introduce significant functional flaws.
Examples of thoroughness criteria include:
- Fixing every instance of a bug. The prompt identifies a bug in one code path, but the same bug exists in a sibling path. A thorough agent finds and fixes both.
- Matching existing support. The prompt asks for new functionality and mentions one input type. A thorough agent notices that similar code handles other input types and supports them too.
Blockers by category
| Blockers by category | |||
|---|---|---|---|
| Category | FrontierCode 1.1 | FrontierCode 2.0 | Change |
| --- | --- | --- | --- |
| Thoroughness | 168 | 501 | |
| Core behavior | 399 | 353 | |
| Avoiding regressions | 174 | 259 | |
| Effective test writing | 202 | 213 | |
| Lint / format / typecheck | 112 | 183 | |
| Scope | 56 | 123 | |
| Backwards compatibility | 85 | 76 | |
| Compilation | 86 | 69 | |
| Test integrity | 48 | 48 | |
| Docs / metadata | 6 | 17 | |
| Implementation detail | 9 | 11 |
Distribution of grading criterion type for FrontierCode 1.1 and 2.0.
Across all blocking criteria in FrontierCode 2.0, thoroughness makes up a significantly larger share, with its total count growing by 3x compared to FrontierCode 1.1.
Blocker failure rate by category
| Blocker failure rate by category | |||
|---|---|---|---|
| Category | FrontierCode 1.1 | FrontierCode 2.0 | Change |
| --- | --- | --- | --- |
| Thoroughness | 11.7% | 19.2% | |
| Core behavior | 9.5% | 14.9% | |
| Backwards compatibility | 2.9% | 5.9% | |
| Scope | 9.0% | 3.8% | |
| Avoiding regressions | 1.3% | 3.3% | |
| Test integrity | 3.4% | 2.5% | |
| Effective test writing | 2.8% | 1.6% | |
| Compilation | 0.7% | 1.2% | |
| Lint / format / typecheck | 1.7% | 1.0% |
How reasoning effort affects performance
FrontierCode is one of the few coding benchmarks where writing additional (correct) code doesn’t necessarily raise a model’s score. This leads to an interesting phenomenon: models can score lower at higher reasoning effort. Because this non-monotonicity often matches user sentiment but is hard to see in other evals, we consider it a useful feature of the benchmark.
However, this scoring methodology led to somewhat hard-to-interpret zig-zaggy model performances on FrontierCode 1.1 with increasing effort level:
As we have previously discussed, a primary factor causing this is the presence of scope blockers. Pass rates on most criteria increase with model effort level, but pass rates on scope criteria decrease:
Rubric criterion pass rate by reasoning effort
Implementation criteria
Scope criteria
| Rubric criterion pass rate by reasoning effort | |||
|---|---|---|---|
| Category | Low effort | Max effort | Change |
| --- | --- | --- | --- |
| Implementation criteria: Claude Opus 5 | 79.3% | 83.0% | |
| Implementation criteria: Claude Opus 4.8 | 75.5% | 79.2% | |
| Implementation criteria: Claude Fable 5 | 79.2% | 81.8% | |
| Implementation criteria: GPT-5.6 Sol | 68.6% | 73.0% | |
| Scope criteria: Claude Opus 5 | 69.2% | 51.1% | |
| Scope criteria: Claude Opus 4.8 | 81.9% | 70.9% | |
| Scope criteria: Claude Fable 5 | 79.1% | 63.4% | |
| Scope criteria: GPT-5.6 Sol | 79.7% | 68.3% |
In other words, models tend to write more code and touch more files with increased effort, even when the changes may not be preferable. Stacking the effects of these curves can produce an oddly shaped score curve. Smaller contributing factors include more frequent cheating at higher effort (models tend to “accidentally” pull the solution commit) and outright overthinking, where a model talks itself into implementing a misreading of the requested change.
The use of scope blockers in FrontierCode 1.0 was somewhat controversial. Another interpretation of changes that violate scope criteria is that the model may be going beyond the prompt in ways that improve the code overall. On investigation, we indeed saw some surprisingly reasonable solutions that failed scope blockers.
In FrontierCode 2.0, we loosened our scope criteria to accommodate surprising but acceptable behavior. Models are now less likely to fail any given scope criterion: averaged across effort levels, the failure rate is 3.8% in FrontierCode 2.0, compared with 9.0% in FrontierCode 1.1. In conjunction with our improved auditing, we are confident that our effort-performance curves measure real regressions at higher reasoning effort.
Blocker failure rate by category and effort
Failed blocker checks as a percentage of all blocker checks in the category.
| Blocker failure rate by category and effort | |||
|---|---|---|---|
| Category | Low effort | Max effort | Change |
| --- | --- | --- | --- |
| Thoroughness | 26.0% | 15.3% | |
| Core behavior | 20.9% | 12.3% | |
| Backwards compatibility | 9.7% | 5.2% | |
| Scope | 3.8% | 4.9% | |
| Avoiding regressions | 4.4% | 2.2% | |
| Test integrity | 3.0% | 1.9% | |
| Effective test writing | 3.9% | 1.3% | |
| Compilation | 2.3% | 0.7% | |
| Lint / format / typecheck | 1.3% | 0.5% |
Blocker failure rate by category and reasoning effort on FrontierCode 2.0. Each line shows the share of blocker checks in a category that failed. Thoroughness has the highest failure rate at every effort level, above core behavior. Results pool four models (Claude Opus 5.5, Claude Fable 5.1, GPT-6.1 Sol, and GPT-6 Astra).
Task Examples
To illustrate FrontierCode 2.0’s task design and grading criteria, we walk through a few example tasks below.
This task is from the open-source Mock Service Worker repo, a widely used library for mocking network requests in JavaScript. It asks the model to fix a bug reported by a user: resetHandlers() doesn’t restart the generator.
On this task, Claude Opus 5.5 discards the reference to the old generator on reset, so the next request starts a new generator from the beginning. This fixes the bug as reported. But simply dropping the reference means the old generator is never closed, so any cleanup code the user wrote in a finally block never runs.
Skipping the user’s cleanup code can cause resource leaks or leave the application in an inconsistent state. The task doesn’t explicitly ask the model to handle cleanup, but experienced JavaScript developers expect a library that discards a generator to close it so its finally block runs, and Opus 5.5’s solution fails that implicit correctness expectation. Tasks like this in FrontierCode 2.0 show that models often fail to infer user intent beyond what is directly specified, which leads them to introduce bugs that violate unstated but clearly inferable constraints.
This task comes from PostHog’s open-source Android SDK, which apps use to record analytics events. When the device is offline, the SDK writes events to disk and sends them the next time the app starts. The task is to fix a startup bug in which two modules send data at the same time: the live queue sends new events while a legacy integration, PostHogSendCachedEventsIntegration.kt, sends the cached ones. The two race, and events reach the server out of order.
At low effort, GPT-6 Astra moves cache into PostHogQueue.kt, removes the sending logic from the legacy integration, and leaves the SDK’s public API untouched, which produces a passing solution.
At max effort, the model often deletes PostHogSendCachedEventsIntegration.kt outright and spreads its responsibilities across other legacy files. This also fixes the race and passes most of the hidden behavior tests. But the deleted class is part of the SDK’s public posthog.api surface, so removing it introduces an unnecessary breaking change for every app that references it. Scope blockers, which deterministically check that the model hasn’t modified certain files, catch this change, and the agent fails the task.
Acknowledgments #
- Thanks to
- Claudio Costa*, Martin McKeaveney*, Tomer Nosrati*, Merlijn Vos*, Cristiam Da Silva*, Daniel Fonseca, Fawzi Essam, Ibrahima Gaye Coulibaly, Juan Patricio Marroquin Gavelan, Kai (Tam Nhu) Tran, Manuel Kießling, Mateusz Łopaciński, Thorbjorn Lindeijer
- pod lead