Cognition, the company behind Devin, has published FrontierCode 2.0, a tougher version of its "would a maintainer merge this pull request?" coding benchmark. In the official results, Claude Opus 5.5 leads at 45.5%, with GPT-6 Astra second at 42.8%. No model reaches 50%. Hours later, a post on X claimed a hidden agent called "Obsidian 3" scored 46.8% in the same data. We could not confirm that row anywhere official, so this post lays out what Cognition actually published, what is only claimed, and how to read a benchmark like this.
If you followed our earlier coverage of FrontierCode 1.1 and the 40% failure rate, 2.0 is the sequel: harder tasks, stricter grading, and lower numbers across the board.
TL;DR: what is verified and what is not #
| Question | Answer | Status |
|---|---|---|
| Who published it? | Cognition, on its own blog and X account | Verified |
| How many tasks? | 132 private tasks (per the BenchLM mirror; Cognition's post does not state a count) | Third-party mirror only |
| Who is first? | Claude Opus 5.5, 45.5% | Verified (official table) |
| Is anything above 50%? | No | Verified |
| Is there a hidden "Obsidian 3" at 46.8%? | One X account says so | Unverified |
| Is "Obsidian 3" an unreleased Anthropic model? | Speculation only | Unverified |
| Is it comparable to 1.1 scores? | No, different tasks and grading | Verified (Cognition and the mirror say so) |
What FrontierCode 2.0 measures #
Most coding evals ask whether code passes a spec. Cognition's FrontierCode 2.0 post says it asks a different question: would a maintainer actually merge the pull request? Over 20 open-source maintainers each spent more than 40 hours per task building them from their own repositories. Scores are weighted aggregates of rubric items covering functional correctness, how well the agent interprets and expands on an under-specified prompt, and whether its design and scoping choices are sound. If the solution fails any blocking criterion, the score is zero.
That last rule is why the numbers are low. A pull request that works but misses one instance of a bug, or touches the wrong scope, is a zero, not a partial credit.
A branching pull request merging into the main line, illustrating the FrontierCode 2.0 maintainer-merge test
What changed from 1.x
According to Cognition's post, compared with the earlier version:
- Tasks come from 55% more repositories.
- Tasks carry 43% more blocking criteria.
- Reference solutions modify 67% more files and 89% more lines, in codebases 25% larger.
- Reference solutions differ from the upstream pull request in 80% of tasks, up from 60%.
- "Thoroughness" blockers roughly tripled, and thoroughness is now the most frequently failed blocker category. Fixing every instance of a problem matters more than fixing one.
- Scope criteria were loosened: the average scope failure rate fell from 9.0% to 3.8%.
- Auditing of prompts, reference solutions and grading criteria was tightened.
The thoroughness shift is the part practitioners should notice. Agents that stop after the first fix, a habit familiar from how we read AI benchmarks, now get punished.
The official results #
Cognition's launch table lists each model at its best reasoning effort. The first eight rows from the post, with the FrontierCode 1.1 score alongside:
| Model | FrontierCode 2.0 | FrontierCode 1.1 |
|---|---|---|
| Claude Opus 5.5 | 45.5% | 54.6% |
| GPT-6 Astra | 42.8% | 53.3% |
| Grok 4.7 | 28.6% | 47.6% |
| Gemini 4 Argon | 26.3% | 47.5% |
| Muse Wonder 1.0 | 24.3% | 47.0% |
| SWE-2 | 23.9% | 50.0% |
| DeepSeek V4.1 Flash | 19.3% | 43.9% |
| Kimi K3 | 17.7% | 44.2% |
Source: Cognition, FrontierCode 2.0. A third-party mirror, BenchLM's FrontierCode 2.0 page, copies 21 model-and-harness rows from the launch post as of October 27 and adds the rest of the ranking: Claude Sonnet 5.5 third at 41.0%, GPT-6.1 Sol at 39.0%, GPT-6 Sol at 34.3%, Claude Fable 5.1 at 32.4%, Claude Haiku 5.5 at 27.7%, and Claude Opus 4.8 at 14.7%. BenchLM notes it mirrors the table without re-running anything and does not use the scores in its own rankings. Cognition's post also says the cost-quality Pareto frontier is Opus 5.5 at the high end and Luna, Haiku and Sol at the low-cost end.
Scoreboard with bars being redrawn, showing how FrontierCode 2.0 scores fell against 1.1
Reading the drops
Every model fell sharply from 1.1 to 2.0. Opus 5.5 lost about nine points. Grok 4.7 dropped roughly 19 points, Gemini 4 Argon about 21, and Cognition's own SWE-2 about 26. The gap between the top two and the rest widened. On 1.1 the field was bunched between 43.9% and 54.6%. On 2.0 it spreads from 17.7% to 45.5%. That is the point of a harder benchmark: it separates models the older one could not.
Compare that with the 1.1 story in our SWE-2 coverage, where Cognition's own model scored 50.0%. On 2.0 the same model is at 23.9%. A score only means something against the test it came from.
The "Obsidian 3" claim: what we found #
Shortly after the launch, an X account (@tensorson) posted that Cognition's FrontierCode 2.0 data contains a hidden model called "Obsidian 3" at 46.8%, above Opus 5.5's 45.5%. Aggregator summaries added two further details: that the row uses a claude-code harness, and that it costs about 3.6 times what Opus 5.5 costs at the same effort. Some commenters concluded it must be an unreleased Anthropic model, possibly a higher Fable-class tier.
Here is what we could and could not check:
- Cognition's blog post. The table we fetched lists the eight rows above and does not mention Obsidian 3 or any hidden model.
- The third-party mirror. BenchLM's 21-row snapshot, taken from the launch post, has no Obsidian 3 row.
- Anthropic. We found no Anthropic announcement, model card or API listing using that name. Our earlier piece onreports of new Anthropic and Google models did not mention it either.
- The source of the 46.8% number. It traces to one X post pointing to data we could not independently locate. It may come from a results file Cognition also published (the mirror links to an official result data file), but we could not open that file, so we cannot confirm or rule it out.
That is the whole evidence. A claimed row is not a verified row, and "claude-code harness, so it must be Anthropic" is an inference: harness names can mean different things, and a row labelled this way could be an internal, partner or pre-release configuration, a labelling artifact, or a mistake. The cost multiple is also second-hand.
Why the speculation is premature
Three reasons to hold back:
- A hidden row would be unusual. Cognition's public post presents a table of named, shipped systems. If a pre-release model was evaluated, the lab would normally be named or the row omitted.
- The margin is small. 46.8% versus 45.5% is 1.3 points on 132 tasks, each graded all-or-nothing on blocking criteria. Run-to-run noise at that size can swap ranks, so a 1.3-point lead is not decisive.
- No second source. Until Cognition, Anthropic or an independent grader confirms the row, repeating "Anthropic has an unreleased model that beats Opus 5.5" would be reporting a guess.
We have covered unconfirmed model-name rumors before, such as the Opus 5.2 stealth-routing rumor, and the safe approach is the same: wait for the lab or the benchmark owner to name it.
How to check it yourself #
- Open Cognition's FrontierCode 2.0 post and compare its table against any screenshot you see. The official table is the reference.
- Look for the linked results data. If a row exists there but not in the post, ask Cognition directly through its official X account.
- Check whether Anthropic, OpenAI or any lab claims the name in a model card or changelog.
- Ignore cost multiples unless they come with the harness, effort setting and token counts that produced them.
What this means for people who build with coding agents #
- Thoroughness is now a scored skill. If your own agent workflows end after the first fix, add a verification pass: search the repo for every instance of the pattern before declaring done.
- Spec-following is not merge-readiness. The FrontierCode premise is that high pass rates on spec-based coding tests can coexist with pull requests a reviewer would reject. For the model at the top of this table, see ourOpus 5.5 launch post .
- Harness matters as much as model. Agent results depend on the scaffolding around the model, which can move scores, so do not read the table as a pure model ranking.
- Price the gap. The 2.7-point lead of Opus 5.5 over GPT-6 Astra is real on this test, but ourcost-per-task look at GPT-6.1 Sol and Astra shows why the cheapest acceptable model is often the right default.
- Private tasks cannot be re-run. The tasks are not public, so outsiders can only trust aggregates. That is a trade-off for avoiding contamination, and it is also why a surprise row is hard to audit.
Who is Cognition, and why does it run this benchmark? #
Cognition makes Devin and trains its own SWE models, so it is both a benchmark author and a competitor, with SWE-2 on its own table at 23.9%. The company deserves credit for publishing a table where its own model sits well below frontier models and for describing the methodology openly. It is still wise to read any vendor-run benchmark as one input. For earlier chapters, see the SWE-1.7 and FrontierCode post, where the first FrontierCode numbers appeared.
Bottom line #
FrontierCode 2.0 is a useful, harder benchmark: Opus 5.5 at 45.5%, GPT-6 Astra at 42.8%, nothing above 50%, and a field that spreads out more than 1.1 did. The "Obsidian 3" row at 46.8% is, for now, a single-source claim that does not appear in the published table or the mirror we checked. We will update this post if Cognition, Anthropic or an independent source confirms or kills it. If you run agents in production, the practical lesson is the thoroughness result, not the rank order.
Related reading #
Scores and claims are accurate as of October 28, 2026 and may change as Cognition updates its results.