{"slug": "artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points", "title": "Artificial Analysis' FrontierCode 2.0 Drops Coding Agent Scores by 16 Points", "summary": "Artificial Analysis released version 2.1 of its Coding Agent Index, swapping in Cognition's harder FrontierCode 2.0 benchmark and Terminal-Bench 4.1, and all 23 configurations evaluated on both FrontierCode versions scored lower, with mean FrontierCode scores falling 16 percentage points from 45% to 29% and overall Index scores dropping about five points on average. Claude Code with Claude Fable 5.2 (high and xhigh effort), Claude Code with Claude Opus 5.5 at max effort, and Devin Fusion CLI with Claude Opus 5.5 Max and GPT-6 Luna High tie for the top score of 58, while cost per task among top configurations varies 30x, from $1.51 to $44.22. Cognition's redesign added tasks from 55% more repositories, 43% more blocking criteria per task, and reference solutions touching 67% more files and 89% more lines, and Artificial Analysis notes that higher reasoning effort can lower scores by expanding patches beyond the requested scope.", "body_md": "# Artificial Analysis' FrontierCode 2.0 Drops Coding Agent Scores by 16 Points\n\nArtificial Analysis refreshed its Coding Agent Index with a harder FrontierCode 2.0 and tightened Terminal-Bench 4.1, reshuffling the leaderboard and dropping average scores by five points.\n\n- Artificial Analysis [Coding Agent Index v2.1](https://artificialanalysis.ai/agents/coding-agents?utm_source=alphasignal) upgrades to FrontierCode 2.0 and Terminal-Bench 4.1.\n- Claude Fable 5.2 and Claude Opus 5.5 tie for the lead at 58 points.\n- Scores dropped 5 points on average, driven by harder [FrontierCode 2.0](https://cognition.com/blog/frontier-code-2.0?utm_source=alphasignal) grading.\n- Cost per task varies 30x among top configs: $1.51 to $44.22.\n- Kimi K3.1 Pro leads [Terminal-Bench 4.1](https://www.tbench.ai/news/terminal-bench-4-1?utm_source=alphasignal) at 68%, ahead of Claude Sonnet 5.5 at 66%.\n- Max reasoning effort can lower scores by violating scope and blowing time limits.\n\nArtificial Analysis, an independent AI evaluation firm, has released version 2.1 of its [Coding Agent Index](https://artificialanalysis.ai/agents/coding-agents?utm_source=alphasignal). The update replaces the previous FrontierCode component with Cognition’s harder FrontierCode 2.0 benchmark and refreshes Terminal-Bench to version 4.1. Most tested configurations lost points, and the new benchmark mix breaks direct score comparisons with earlier Index versions.\n\nThe top score of 58 belongs to four configurations:\n\n- Claude Code with Claude Fable 5.2 at high reasoning effort\n- Claude Code with Claude Fable 5.2 at extra-high, or xhigh, effort\n- Claude Code with Claude Opus 5.5 at max effort\n- Devin Fusion CLI with Claude Opus 5.5 Max and GPT-6 Luna High\n\nClaude Code with Claude Sonnet 5.5 Max follows at 56. Gemini 4 Carbon at high effort scores 55 in Antigravity CLI.\n\n## A stricter merge bar drives the drop\n\nThe Index averages results from three tests and reports the result on a 100-point scale:\n\n- **FrontierCode-AA 2.0** evaluates repository patches against criteria that could block a merge.\n- **Terminal-Bench 4.1** measures an agent’s ability to complete tasks in a command-line environment.\n- **SWE-Atlas-QnA-AA** tests software-engineering question answering.\n\nFrontierCode 2.0 caused most of the movement. All 23 configurations evaluated on both versions scored lower on the new test. Their mean FrontierCode score fell 16 percentage points, from 45% to 29%, while overall Index scores declined by about five points on average.\n\nCognition redesigned [FrontierCode 2.0](https://cognition.com/blog/frontier-code-2.0?utm_source=alphasignal) to judge whether a repository maintainer would accept an agent’s patch. The update includes:\n\n- Tasks from 55% more repositories, including security fixes and performance optimizations\n- 43% more blocking criteria per task\n- Reference solutions that modify 67% more files and 89% more lines in codebases that are 25% larger\n- Three times the previous share of thoroughness criteria\n\nThoroughness criteria penalize patches that fix the reported case while missing sibling code paths, related defects, or implied edge cases. Across the four frontier models Cognition analyzed, thoroughness produced the highest blocker failure rate at every reasoning level, exceeding failures on core behavior.\n\n## More reasoning can over-expand a patch\n\nHigher reasoning settings sometimes reduce FrontierCode scores because agents write more code, touch more files, and create more opportunities to exceed the requested scope. Cognition documented one GPT-6 Astra task involving a race condition in the PostHog Android SDK. The low-effort run produced a minimal patch, while the max-effort run deleted a public API class and broke backward compatibility for downstream applications.\n\nClaude Fable 5.2 follows a similar pattern in the new Index. Its max setting scores 56, two points below high and xhigh. At max effort, 16% of its FrontierCode attempts and 13% of its SWE-Atlas-QnA-AA attempts reached their time limits. The corresponding rates at xhigh were 1% and 0%.\n\nFable’s max setting also generates about three times as many output tokens per task as xhigh, increasing cost without improving the aggregate score.\n\n## Equal scores come with a 30-fold cost gap\n\nAmong configurations scoring at least 48, estimated pay-per-token API cost varies by roughly 30-fold. The following selected results show how reasoning effort changes the performance and cost of the same model:\n\n| Selected Coding Agent Index v2.1 results |  |  | \n|---|---|---|\n| Configuration | Index score | Cost per task | \n|---|---|---|\n| Claude Code + Claude Fable 5.2 Max | 56 | $44.22 | \n| Claude Code + Claude Fable 5.2 Xhigh | 58 | $15.47 | \n| Claude Code + Claude Fable 5.2 High | 58 | $7.51 | \n| Claude Code + Claude Fable 5.2 Medium | 53 | $2.89 | \n| Codex + GPT-6.1 Sol Max | 52 | $1.51 | \n\nFable at high effort matches xhigh’s score for less than half the cost and exceeds max by two points at about one-sixth the price. Codex with GPT-6.1 Sol gives up six points against the leaders but costs $1.51 per task. Codex with GPT-6.1 Terra Max was the only tested configuration to remain within one point of its previous Index result.\n\n## Terminal-Bench barely moves\n\nTerminal-Bench 4.0 includes 66 tasks for testing agentic terminal use. The revisions aim to improve grading reliability and reduce cases where an agent earns credit by exploiting the checker. Across the 23 configurations tested on both versions, every score moved by less than three points.\n\nKimi Code CLI with Kimi K3.1 Pro and Devin Fusion CLI each score 68% on the terminal benchmark. Claude Code with Claude Sonnet 5.5 Max reaches 66%, while Claude Opus 5.5 Max and Gemini 4 Carbon in Antigravity CLI each score 65%.\n\n## Fallback routing changes the comparison\n\nClaude Code and Devin Fusion CLI automatically route an attempt to a secondary model after some safety refusals. The refusal remains recorded, while the fallback can still complete the task and preserve the benchmark result.\n\nFallback models completed 9% of attempts for Devin Fusion CLI and 9% for Claude Code with Claude Opus 5.5 Max. The rates were 6% for Claude Fable 5.2 Xhigh and 5% for Claude Sonnet 5.5 Max. Codex blocked about 1% of attempts with GPT-6.1 Sol Max and GPT-6 Astra Max.\n\nCross-harness scores therefore measure both model performance and orchestration policy, including refusal handling, fallback routing, and timeout behavior.\n\n## The tie conceals different strengths\n\nClaude Fable 5.2 Xhigh scores 49% on FrontierCode-AA 2.0, five points above Claude Opus 5.5 Max. Opus leads Fable by four points on SWE-Atlas-QnA-AA, 66% to 62%, while their Terminal-Bench results are within one point.\n\nDevin Fusion CLI reaches the same 58-point Index score through a different profile: 44% on FrontierCode, 68% on Terminal-Bench, and 63% on SWE-Atlas-QnA-AA.\n\n## A workload-based shortlist\n\n- **General repository work:** Claude Fable 5.2 at high effort combines an Index-leading score of 58 with a reported cost of $7.51 per task.\n- **Terminal-heavy workflows:** Kimi K3.1 Pro in Kimi Code CLI and Devin Fusion CLI share the highest Terminal-Bench score at 68%.\n- **Lower-cost deployments:** Codex with GPT-6.1 Sol Max scores 52 at $1.51 per task.\n- **Reasoning-effort tuning:** Claude Fable 5.2 Xhigh outperforms max while using fewer tokens and encountering fewer timeouts.\n\nThe cost figures estimate token charges at pay-per-token API rates. Subscription plans, private discounts, latency requirements, tool permissions, and repository-specific tasks can change the production trade-offs, so teams still need evaluations based on their own code and workflows.\n\nVersion 2.1 shifts more weight toward maintainer acceptance, including complete fixes, compatibility, and disciplined scope. The 16-point average decline on FrontierCode 2.0 shows that these requirements continue to separate leading coding-agent configurations.", "url": "https://wpnews.pro/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points", "canonical_source": "https://alphasignal.ai/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points", "published_at": "2026-11-14 00:04:35+00:00", "updated_at": "2026-11-14 01:22:38.489295+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "ai-tools", "artificial-intelligence"], "entities": ["Artificial Analysis", "Cognition", "FrontierCode 2.0", "Terminal-Bench 4.1", "Claude Fable 5.2", "Claude Opus 5.5", "Kimi K3.1 Pro", "Devin Fusion CLI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points", "markdown": "https://wpnews.pro/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points.md", "text": "https://wpnews.pro/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points.txt", "jsonld": "https://wpnews.pro/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points.jsonld"}}