{"slug": "coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per", "title": "Coding Agent Index v2.1: scores fell 5 points, 4 configs tie at 58, and cost per task spans 30x", "summary": "Artificial Analysis released version 2.1 of its Coding Agent Index on November 14, scoring coding agents as a combined harness, model and effort setting across 322 tasks. Index scores fell about 5 points on average after the swap to FrontierCode-AA 2.0, with all 23 configurations evaluated on both versions scoring lower and the mean FrontierCode score dropping 16 points, from 45% to 29%. Four configurations now tie for the lead at 58, while the cost to run a single task at the top of the board spans roughly 30x.", "body_md": "On November 14 Artificial Analysis shipped version 2.1 of its [Coding Agent Index](https://artificialanalysis.ai/agents/coding-agents), the leaderboard that scores a coding agent as one unit of harness plus model plus effort setting. The short version: nearly everyone lost points, the lead is a four-way tie at 58, and the cost to run one task at the top of the board spans roughly 30x.\n\nWhen the index reset in October I wrote about [what the v2.0 reset measures](https://dev.to/piekwerk/20-of-21-agent-configs-just-scored-lower-what-the-coding-agent-index-v20-reset-measures-70j). The v2.1 change is narrower but sharper, and the cost column is the part worth your time if you pick models for a coding agent.\n\nThe index is an equal-weight average of three benchmarks, 322 tasks total, three attempts per task:\n\n| Component | Tasks | What it measures | How it scores | \n|---|---|---|---|\n| FrontierCode-AA 2.0 | 132 | changes to real open-source repositories | maintainer-merge rubric, 0 or 70-100% | \n| Terminal-Bench 4.1 | 66 | multi-step terminal work | test suite, pass/fail | \n| SWE-Atlas-QnA-AA | 124 | repository question answering | binary pass/fail | \n\nFrontierCode-AA 2.0 is Artificial Analysis' implementation of [Cognition's FrontierCode 2.0](https://cognition.com/blog/frontier-code-2.0), released October 27. An attempt that fails any blocking check scores zero. An attempt that passes them all scores between 70% and 100% depending on non-blocking checks. A trajectory scanner zeroes any attempt that went looking for the upstream solution or its merge commit.\n\nTerminal-Bench 4.1 keeps 4.0's 66 tasks and environments and replaces the verifiers. Code that runs during grading now runs as an untrusted user in a separate process, and verifiers that rejected valid solutions were corrected. That is reward-hacking hardening, not new tasks.\n\nThe third component and the weighting did not change. The largest component got harder, and that moved the whole board. The full [methodology page](https://artificialanalysis.ai/methodology/coding-agents-benchmarking) is worth a read.\n\nPer the [v2.1 announcement](https://x.com/ArtificialAnlys/status/2121393821955334328), index scores came down about 5 points on average, driven by the FrontierCode swap. [AlphaSignal's read](https://alphasignal.ai/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points) of the underlying data is more specific: all 23 configurations evaluated on both versions scored lower on the new FrontierCode, and the mean FrontierCode score fell 16 points, from 45% to 29%.\n\nThat is not the models getting worse. FrontierCode 2.0 pulls from 55% more repositories, carries 43% more blocking criteria per task, and its reference solutions modify 67% more files and 89% more lines in codebases that are 25% larger. The share of thoroughness criteria roughly tripled. Thoroughness criteria catch the patch that fixes the reported case and misses the sibling code path, and Cognition reports thoroughness produced the highest blocker failure rate at every reasoning level they analyzed.\n\nOne configuration barely moved: Codex with GPT-6.1 Terra at max effort stayed within a point of its previous index score. Everyone else ate the reset.\n\nThis is the second consecutive version change that lowered scores across the board. That is the design. A suite that refreshes before vendors tune against it keeps some discriminating power. A score you saw quoted last month is a different exam.\n\nThe top score belongs to four configurations, not one product: Claude Code with Fable 5.2 at high effort, Claude Code with Fable 5.2 at xhigh, Claude Code with Opus 5.5 at max, and Devin Fusion CLI with Opus 5.5 Max paired with GPT-6 Luna High.\n\nTwo details matter more than the rank. First, Fable 5.2 at max scores 56, below its own high-effort result, because more of its attempts hit task time limits. Effort is not a simple dial where more is better. Second, Gemini 4 Carbon at high effort scores 55 in Antigravity CLI. The index deliberately scores the harness and the model together, which is why I trust it more than raw model benchmarks for agent work. We have seen this before from the other side: [Hugging Face measured the same model scoring 33% or 62% depending on the harness](https://dev.to/piekwerk/hugging-face-measured-it-the-same-model-scores-33-or-62-depending-on-the-harness-1mkk).\n\nv2.1 also changed how cost is computed. Artificial Analysis aligned it with their Intelligence Index method, combining provider-reported token counts with measured typical cache hit rates. So v2.0-era cost figures you see quoted in older posts are not comparable either.\n\nAmong configurations scoring 48 or above, cost per task runs from $1.51 for Codex with GPT-6.1 Sol at max (index 52) to $44.22 for Claude Code with Fable 5.2 at max (index 56). That is roughly 30x.\n\nThe cheap end of that spread is not a fringe pick. GPT-6.1 Sol is the configuration [Codex flipped to by default in 0.161](https://dev.to/piekwerk/codex-0161-flipped-the-default-model-to-gpt-61-sol-if-you-never-set-model-you-already-switched-577b). For a mid-board reference, Artificial Analysis' own [Codex page](https://artificialanalysis.ai/agents/coding-agents/codex) pegs Codex with GPT-6 Astra at max at 53 index, $7.63 per task on average, and 26.0 minutes per task against a 40.5 minute median.\n\nSo the top of the board asks roughly 29x the per-task cost for 4 index points. Whether that trade is worth it depends entirely on what a task is worth to you, which brings me to the part I actually use.\n\nThe arithmetic is blunt:\n\n```\n# per-task API cost at list pricing, AA v2.1 figures\n# Codex + GPT-6.1 Sol (max), index 52\n40 tasks * $1.51  = $60.40\n\n# Claude Code + Fable 5.2 (max), index 56\n40 tasks * $44.22 = $1,768.80\n```\n\nThree caveats before you take that to a budget meeting. These are benchmark tasks, not your tasks. If you are on a subscription plan, the per-task cost is hidden inside an allowance rather than billed per call. And a 4-point index gap can be a large gap or a rounding error depending on which benchmark components moved, which is exactly why the tie deserves a closer read.\n\nThe index is an equal-weight average of three quite different formats: rubric-graded repository patches, pass/fail terminal work, and pass/fail question answering. Two configurations a couple of points apart can have different strengths, and Artificial Analysis themselves tell you to read the per-benchmark breakdowns rather than the headline.\n\nA tie at the top means the gap between the leaders is smaller than the evaluation's variance. Treat 58 versus 56 as the same class of thing, not a ranking.\n\nCross-version comparisons are category errors. v2.0 to v2.1 changed a component and the cost method, so a vendor graph showing a jump across that boundary is showing the suite change, not the model.\n\nFinally, safety refusals score zero on every benchmark, so the published blocked rate is the ceiling of what refusals can cost a configuration, and reward-hacking rates are reported per benchmark after integrity review. Those two columns tell you things the headline hides.\n\nI pin the harness, model, and effort together as one unit in config, because that is what the index scores and [alias names drift](https://dev.to/piekwerk/model-haiku-stopped-meaning-one-thing-pin-the-exact-model-id-7de). For bulk work, many small edits, test runs, dependency bumps, I run the cheap configuration first and escalate only the tasks it fails, and I label that as my own workflow choice rather than a measured result. And when a launch post cites a benchmark jump, the first thing I check is which index version it was scored on. The cheaper question is always your own repo: cost per merged pull request, measured by you, beats any leaderboard.\n\nSources: [Artificial Analysis announcement](https://x.com/ArtificialAnlys/status/2121393821955334328), [AA methodology](https://artificialanalysis.ai/methodology/coding-agents-benchmarking), [Cognition FrontierCode 2.0](https://cognition.com/blog/frontier-code-2.0), [AlphaSignal analysis](https://alphasignal.ai/news/artificial-analysis-frontiercode-2-0-drops-coding-agent-scores-by-16-points).", "url": "https://wpnews.pro/news/coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per", "canonical_source": "https://dev.to/piekwerk/coding-agent-index-v21-scores-fell-5-points-4-configs-tie-at-58-and-cost-per-task-spans-30x-1c1c", "published_at": "2026-11-15 19:14:26+00:00", "updated_at": "2026-11-15 19:20:46.086947+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools", "ai-research"], "entities": ["Artificial Analysis", "Cognition", "FrontierCode", "Terminal-Bench", "Claude Code", "Devin", "Gemini", "AlphaSignal"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per", "markdown": "https://wpnews.pro/news/coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per.md", "text": "https://wpnews.pro/news/coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per.txt", "jsonld": "https://wpnews.pro/news/coding-agent-index-v2-1-scores-fell-5-points-4-configs-tie-at-58-and-cost-per.jsonld"}}