JetBrains released the Kotlin Benchmark, its official benchmark for grading AI coding agents on real Kotlin engineering work. Claude Code with Opus 4.7 xhigh leads the first leaderboard at 85.7 percent, with JetBrains Junie and OpenAI's Codex right behind at 81.9 percent. But the resolution rate is the least interesting column on the page.
The number that deserves your attention is tokens per solved task. It ranges from about 66,000 to 777,000 across the top twenty setups. That is a 12x spread in cost for setups whose resolution rates differ by roughly 20 percentage points, and some of the expensive entries solve fewer tasks than setups spending a tenth of what they burn. That column, not the ranking, is what decides which setup makes sense for your team.
The Kotlin Benchmark follows the SWE-bench methodology, applied to repository-level Kotlin work. The dataset has 105 engineering tasks taken from active open-source Kotlin repositories. Each agent gets a real issue description, has to navigate the project, and produce a working patch. Solutions are verified in containerized environments, and a task only counts as resolved when the patch passes the repo's own tests. No self-reporting, no vibes.
JetBrains built it on the open-source Multi-SWE-bench infrastructure and published the datasets, test harnesses, and leaderboard. Alongside it, they released the methodology page and opened the GitHub repo so teams can inspect the tasks themselves.
Why Kotlin matters here: Kotlin is now the default language for Android, a first-class JVM language for server-side work, and the ecosystem has been poorly served by generic benchmarks. JetBrains already maintained model-focused evals like Kotlin_HumanEval, but those test whether a model knows syntax. This benchmark tests the layer above it: can an agent complete a validated engineering task inside an existing project, the way your team actually works.
Here are the results, resorted by my own math into tokens per resolved task. Full disclosure: I have not run these evaluations myself. Everything comes from the official leaderboard, the JetBrains announcement post, and the methodology page. The token math below is mine.
Read that list twice and the ranking stops mattering. Here is what jumps out.
Reasoning effort has terrible marginal returns on the margin you would pay. Claude Code with Opus 4.7 medium spent half the tokens of the xhigh setup and solved 80 of the 90 tasks xhigh solved. The extra ten tasks cost 5.28M extra tokens, which works out to about 528K tokens per additional task, roughly 8 times the average cost per task of the medium setup. If your tasks cluster near the easy end, xhigh reasoning is mostly paying premium prices for tasks medium already solved cheaply.
The same model in a different loop differs by 3.7x. Codex with GPT 5.5 xhigh tied Junie with Opus 4.7 max at 86 tasks. Codex did it with 38.78M tokens, Junie with 19.98M. The agent harness around the model matters as much as the model itself. This is the strongest public evidence yet for something practitioners keep saying: you are not picking a model, you are picking a loop.
Flash models are not cheap. Two Gemini 3 Flash entries sit at the bottom of the token-efficiency table, at 628K and 777K tokens per solved task. The obvious theory: weaker models fail more, and every failed attempt burns a full context of tokens before the verification step says no. Cheap-per-token is not cheap-per-solved-task when the failure rate is high. The real unit of cost is not the token, it is the solved task.
Wall-clock time is a separate budget. Junie with Opus 4.7 max needed nearly 20 hours to finish the suite. Claude Code medium finished in under 5. If your workflow is a developer waiting on an agent between code reviews, a 4x slower loop is a real cost even at identical token counts.
One number from a different benchmark puts this whole leaderboard in perspective. Yesterday I wrote about Real-SWE, a benchmark that ran 8 frontier models on 10 tasks from private enterprise codebases. The best model there scored 38.8 percent.
The Kotlin Benchmark's top score is 85.7 percent. Same year, same models, same general category of work. The gap is not a mystery, and it is not mostly about language.
JetBrains itself says the quiet part in the announcement: the scores are "intended as a signal, not a guarantee for your codebase," and real results depend on your architecture, internal APIs, and validation process. Neither benchmark is lying. They are measuring different things, and the difference between 85.7 and 38.8 is roughly the difference between your public repo work and your private repo work.
If you run Kotlin or Java server-side work and you are evaluating agent setups, this benchmark is the closest public proxy you have. Here is the approach it suggests. JetBrains did the ecosystem a favor here, and not just Kotlin developers. The interesting output of this release is not that Claude Code leads, or that Junie and Codex are close. It is that publishing tokens and latency next to resolution rates makes the quality ranking look almost like a distraction. Once you sort by cost per solved task, most teams' optimal setup moves down the leaderboard, and the assumption that more reasoning and cheaper models always help dies on the same page.
The benchmark is open, the tasks are inspectable on GitHub, and JetBrains plans to expand coverage to Android and Kotlin Multiplatform, add cost and maintainability metrics, and evaluate more open-weight models. If they add a Java edition with private-repo task sourcing, it might be the most useful benchmark in the JVM ecosystem. Until then, use this one the way it is meant to be used: as a signal, then run your own scoreboard on your own tickets.
I write about AI agents, JVM ecosystems, and backend engineering every week. If that is your world, subscribing is free and every post is grounded in linked sources.
Have you run coding agents against your own codebase as a benchmark? What did your tokens-per-merged-task number look like compared to the public leaderboards?