# Frontier Coding Models Are Two Points Apart. Costs Aren't > Source: > Published: 2026-07-31 17:09:17+00:00 [AI](https://sourcefeed.dev/c/ai)Article # Frontier Coding Models Are Two Points Apart. Costs Aren't SWE-rebench's decontaminated five-language leaderboard shows accuracy converging while price per solved issue diverges fivefold. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) Benchmark leaderboards for coding models have two chronic diseases: contamination (the model trained on the answers) and Python monoculture (the tasks look nothing like your Go services or your Rust crates). [SWE-rebench](https://swe-rebench.com), the continuously refreshed benchmark from [Nebius](https://nebius.com), was built to treat the first. Its latest update — 13 models and 4 coding agents evaluated on real issues in Go, Java, Python, Rust, and TypeScript — takes a serious swing at the second. And the results tell developers something more interesting than who's on top: at the frontier, *nobody* is meaningfully on top anymore. ## Why this leaderboard is worth trusting more than most The original [SWE-bench](https://www.swebench.com) changed how we evaluate coding models, but its static task set has been public since 2023, which means every frontier lab has had years to train on or near it. SWE-rebench's answer is a rolling window: tasks are continuously mined from live open-source repositories, and the current board covers 111 problems from 65 repos filed between May 15 and July 1, 2026. Issues that predate a model's release get an explicit contamination flag rather than a quiet pass. The evaluation setup is deliberately boring, which is a compliment. Every bare model runs in the same minimal ReAct-style scaffold with identical prompts, a 128K context cap, and no model-specific tuning, five full runs each, reporting resolved rate, pass@5, cost per problem, and token burn. That last pair matters more than it used to — more on that in a moment. The multilingual push isn't happening in a vacuum. ByteDance's [Multi-SWE-bench](https://github.com/multi-swe-bench/multi-swe-bench) hand-curated 2,132 instances across eight languages, and SWE-bench itself grew a multilingual split. But those are static sets with the same shelf-life problem the original had. SWE-rebench's V2 corpus — 32,079 tasks across 20 languages and 3,600+ repositories, built by an automated pipeline where a setup agent synthesizes per-repo install and test procedures and an ensemble of LLM judges filters flaky instances — is the first attempt to make multilingual evaluation *renewable*. That's the actual innovation here. Fresh tasks in five languages, forever, is a different product than 300 curated tasks, once. ## The frontier is two points wide Here's the top of the current board, plus the notable value entries: | System | Resolved | Pass@5 | Cost/problem | |---|---|---|---| | Fable 5 (Anthropic, high effort) | 64.5% | 78.4% | $4.40 | | Grok 4.5 (xAI) | 63.8% | 77.5% | $1.47 | | Opus 5 (Anthropic) | 63.4% | 74.8% | $3.47 | | GLM-5.2 (Z.ai) | 62.9% | 81.1% | $1.40 | | GPT-5.6 Sol (OpenAI, medium effort) | 62.3% | 79.3% | $0.85 | | Junie (JetBrains, agent) | 61.8% | 73.9% | $0.81 | | Claude Code (Anthropic, agent) | 60.4% | 75.7% | $3.39 | | GPT-5.6 Luna (OpenAI) | 43.6% | 59.5% | $0.11 | | DeepSeek-V4 Pro | 40.2% | 64.0% | $0.15 | Read that top cluster again: five systems from five different labs within 2.2 points of each other, on decontaminated tasks the models couldn't have memorized. On resolved rate, the frontier has effectively converged. What hasn't converged is everything else. Fable 5's two-point edge over GPT-5.6 Sol costs five times as much per problem — and Sol did it in 605K tokens per task against Fable's 2.5M. Normalize to cost per *solved* problem and the spread is brutal: roughly $6.80 for Fable 5, $2.30 for Grok 4.5, $2.20 for GLM-5.2, $1.36 for Sol. If you're wiring a model into CI to auto-triage issues at scale, that's not a rounding error; it's the whole decision. The agent rows carry a second lesson. Junie — which per the leaderboard's own notes runs Claude under the hood — hits 61.8% at $0.81, but Claude Code, Codex (58.0%), and Cursor (51.7%) all land *below* the best bare models running in SWE-rebench's dumb 128K ReAct loop. For contained issue-fixing tasks, the elaborate harness is close to commoditized. The scaffold you're paying a product margin for is worth less than the model inside it, and Junie's showing suggests the remaining harness value is in token discipline, not cleverness. ## What to actually do with this If you're choosing a model for automated code fixing today: default to the cheap frontier tier (Sol-class, Grok, GLM) and escalate to an expensive model only on retry, using pass@5 as your guide — GLM-5.2's 81.1% pass@5 says multiple cheap attempts beat one expensive one. The sub-$0.20 tier (DeepSeek-V4 Pro, GPT-5.6 Luna at ~40–46%) is now genuinely usable for high-volume, low-stakes work like lint-grade fixes where a 40% hit rate on nearly-free attempts pencils out. Now the caveats, because this board has real ones. It's 111 problems — small enough that a couple of points of separation is noise, which reinforces rather than undermines the convergence reading. The per-language story is weaker than the headline implies: each language contributes *different* problems, so you can't cleanly conclude "model X is better at Rust than model Y" — only that the aggregate now includes non-Python reality. And commenters on Hacker News flagged a fairness wrinkle worth taking seriously: Fable 5 was run at high reasoning effort while GPT-5.6 Sol ran at medium, so the one benchmark where the labs' flagships finally meet on decontaminated ground still isn't quite comparing like with like. If Sol at high effort closes a 2.2-point gap — plausible, given what effort scaling typically buys — the "leader" changes. My read: this is the most decision-useful public coding leaderboard running today, precisely because it reports the columns vendors would rather you skip. The resolved-rate race is over for now; the frontier labs are trading fractions of a point. The race that matters — for anyone actually deploying this stuff — is tokens and dollars per solved issue, and on that axis the market is wide open and the incumbents are losing. ## Sources & further reading - [SWE-rebench Leaderboard](https://swe-rebench.com)— swe-rebench.com - [13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS](https://news.ycombinator.com/item?id=49124336)— news.ycombinator.com - [SWE-rebench V2: A 20-Language Benchmark for AI Coding Agents](https://toknow.ai/posts/swe-rebench-v2-language-agnostic-coding-benchmark-nebius/)— toknow.ai - [SWE-rebench-leaderboard dataset](https://huggingface.co/datasets/nebius/SWE-rebench-leaderboard)— huggingface.co [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.