Frontier Coding Models Are Two Points Apart. Costs Aren't

SWE-rebench's latest leaderboard shows frontier coding models within 2.2 points of each other on resolved rate, with Fable 5 (Anthropic) at 64.5%, Grok 4.5 (xAI) at 63.8%, Opus 5 (Anthropic) at 63.4%, GLM-5.2 (Z.ai) at 62.9%, and GPT-5.6 Sol (OpenAI) at 62.3%, but cost per solved problem diverges fivefold, from $6.80 for Fable 5 to $1.36 for GPT-5.6 Sol. The benchmark, from Nebius, evaluates 13 models and 4 coding agents on 111 decontaminated issues across Go, Java, Python, Rust, and TypeScript, with a rolling window of tasks from May 15 to July 1, 2026.

AI https://sourcefeed.dev/c/ai Article Frontier Coding Models Are Two Points Apart. Costs Aren't SWE-rebench's decontaminated five-language leaderboard shows accuracy converging while price per solved issue diverges fivefold. Mariana Souza https://sourcefeed.dev/u/mariana souza Benchmark leaderboards for coding models have two chronic diseases: contamination the model trained on the answers and Python monoculture the tasks look nothing like your Go services or your Rust crates . SWE-rebench https://swe-rebench.com , the continuously refreshed benchmark from Nebius https://nebius.com , was built to treat the first. Its latest update — 13 models and 4 coding agents evaluated on real issues in Go, Java, Python, Rust, and TypeScript — takes a serious swing at the second. And the results tell developers something more interesting than who's on top: at the frontier, nobody is meaningfully on top anymore. Why this leaderboard is worth trusting more than most The original SWE-bench https://www.swebench.com changed how we evaluate coding models, but its static task set has been public since 2023, which means every frontier lab has had years to train on or near it. SWE-rebench's answer is a rolling window: tasks are continuously mined from live open-source repositories, and the current board covers 111 problems from 65 repos filed between May 15 and July 1, 2026. Issues that predate a model's release get an explicit contamination flag rather than a quiet pass. The evaluation setup is deliberately boring, which is a compliment. Every bare model runs in the same minimal ReAct-style scaffold with identical prompts, a 128K context cap, and no model-specific tuning, five full runs each, reporting resolved rate, pass@5, cost per problem, and token burn. That last pair matters more than it used to — more on that in a moment. The multilingual push isn't happening in a vacuum. ByteDance's Multi-SWE-bench https://github.com/multi-swe-bench/multi-swe-bench hand-curated 2,132 instances across eight languages, and SWE-bench itself grew a multilingual split. But those are static sets with the same shelf-life problem the original had. SWE-rebench's V2 corpus — 32,079 tasks across 20 languages and 3,600+ repositories, built by an automated pipeline where a setup agent synthesizes per-repo install and test procedures and an ensemble of LLM judges filters flaky instances — is the first attempt to make multilingual evaluation renewable . That's the actual innovation here. Fresh tasks in five languages, forever, is a different product than 300 curated tasks, once. The frontier is two points wide Here's the top of the current board, plus the notable value entries: | System | Resolved | Pass@5 | Cost/problem | |---|---|---|---| | Fable 5 Anthropic, high effort | 64.5% | 78.4% | $4.40 | | Grok 4.5 xAI | 63.8% | 77.5% | $1.47 | | Opus 5 Anthropic | 63.4% | 74.8% | $3.47 | | GLM-5.2 Z.ai | 62.9% | 81.1% | $1.40 | | GPT-5.6 Sol OpenAI, medium effort | 62.3% | 79.3% | $0.85 | | Junie JetBrains, agent | 61.8% | 73.9% | $0.81 | | Claude Code Anthropic, agent | 60.4% | 75.7% | $3.39 | | GPT-5.6 Luna OpenAI | 43.6% | 59.5% | $0.11 | | DeepSeek-V4 Pro | 40.2% | 64.0% | $0.15 | Read that top cluster again: five systems from five different labs within 2.2 points of each other, on decontaminated tasks the models couldn't have memorized. On resolved rate, the frontier has effectively converged. What hasn't converged is everything else. Fable 5's two-point edge over GPT-5.6 Sol costs five times as much per problem — and Sol did it in 605K tokens per task against Fable's 2.5M. Normalize to cost per solved problem and the spread is brutal: roughly $6.80 for Fable 5, $2.30 for Grok 4.5, $2.20 for GLM-5.2, $1.36 for Sol. If you're wiring a model into CI to auto-triage issues at scale, that's not a rounding error; it's the whole decision. The agent rows carry a second lesson. Junie — which per the leaderboard's own notes runs Claude under the hood — hits 61.8% at $0.81, but Claude Code, Codex 58.0% , and Cursor 51.7% all land below the best bare models running in SWE-rebench's dumb 128K ReAct loop. For contained issue-fixing tasks, the elaborate harness is close to commoditized. The scaffold you're paying a product margin for is worth less than the model inside it, and Junie's showing suggests the remaining harness value is in token discipline, not cleverness. What to actually do with this If you're choosing a model for automated code fixing today: default to the cheap frontier tier Sol-class, Grok, GLM and escalate to an expensive model only on retry, using pass@5 as your guide — GLM-5.2's 81.1% pass@5 says multiple cheap attempts beat one expensive one. The sub-$0.20 tier DeepSeek-V4 Pro, GPT-5.6 Luna at ~40–46% is now genuinely usable for high-volume, low-stakes work like lint-grade fixes where a 40% hit rate on nearly-free attempts pencils out. Now the caveats, because this board has real ones. It's 111 problems — small enough that a couple of points of separation is noise, which reinforces rather than undermines the convergence reading. The per-language story is weaker than the headline implies: each language contributes different problems, so you can't cleanly conclude "model X is better at Rust than model Y" — only that the aggregate now includes non-Python reality. And commenters on Hacker News flagged a fairness wrinkle worth taking seriously: Fable 5 was run at high reasoning effort while GPT-5.6 Sol ran at medium, so the one benchmark where the labs' flagships finally meet on decontaminated ground still isn't quite comparing like with like. If Sol at high effort closes a 2.2-point gap — plausible, given what effort scaling typically buys — the "leader" changes. My read: this is the most decision-useful public coding leaderboard running today, precisely because it reports the columns vendors would rather you skip. The resolved-rate race is over for now; the frontier labs are trading fractions of a point. The race that matters — for anyone actually deploying this stuff — is tokens and dollars per solved issue, and on that axis the market is wide open and the incumbents are losing. Sources & further reading - SWE-rebench Leaderboard https://swe-rebench.com — swe-rebench.com - 13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS https://news.ycombinator.com/item?id=49124336 — news.ycombinator.com - SWE-rebench V2: A 20-Language Benchmark for AI Coding Agents https://toknow.ai/posts/swe-rebench-v2-language-agnostic-coding-benchmark-nebius/ — toknow.ai - SWE-rebench-leaderboard dataset https://huggingface.co/datasets/nebius/SWE-rebench-leaderboard — huggingface.co Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.