{"slug": "jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real", "title": "JetBrains Ranked AI Agents on Real Kotlin Projects. The Token Column Is the Real Story.", "summary": "JetBrains released the Kotlin Benchmark, an official SWE-bench-style evaluation that grades AI coding agents on 105 real repository-level Kotlin engineering tasks, with Claude Code running Opus 4.7 xhigh leading the first leaderboard at 85.7 percent. Analysis of the published results shows tokens per solved task ranging from roughly 66,000 to 777,000 across the top twenty setups, a 12x cost spread that the writeup argues matters more than the resolution ranking. The same model in different agent harnesses differed by up to 3.7x in token use, and the author notes the token math is their own, based on the official leaderboard and methodology page.", "body_md": "JetBrains released the [Kotlin Benchmark](https://kotlinlang.org/benchmark/), its official benchmark for grading AI coding agents on real Kotlin engineering work. Claude Code with Opus 4.7 xhigh leads the first leaderboard at **85.7 percent**, with JetBrains Junie and OpenAI's Codex right behind at 81.9 percent. But the resolution rate is the least interesting column on the page.\n\nThe number that deserves your attention is tokens per solved task. It ranges from about **66,000 to 777,000** across the top twenty setups. That is a 12x spread in cost for setups whose resolution rates differ by roughly 20 percentage points, and some of the expensive entries solve *fewer* tasks than setups spending a tenth of what they burn. That column, not the ranking, is what decides which setup makes sense for your team.\n\nThe Kotlin Benchmark follows the SWE-bench methodology, applied to repository-level Kotlin work. The dataset has **105 engineering tasks** taken from active open-source Kotlin repositories. Each agent gets a real issue description, has to navigate the project, and produce a working patch. Solutions are verified in containerized environments, and a task only counts as resolved when the patch passes the repo's own tests. No self-reporting, no vibes.\n\nJetBrains built it on the open-source Multi-SWE-bench infrastructure and published the datasets, test harnesses, and leaderboard. Alongside it, they released the [methodology page](https://kotlinlang.org/benchmark/methodology/) and opened the GitHub repo so teams can inspect the tasks themselves.\n\nWhy Kotlin matters here: Kotlin is now the default language for Android, a first-class JVM language for server-side work, and the ecosystem has been poorly served by generic benchmarks. JetBrains already maintained model-focused evals like [Kotlin_HumanEval](https://huggingface.co/datasets/JetBrains/Kotlin_HumanEval), but those test whether a model knows syntax. This benchmark tests the layer above it: can an agent complete a validated engineering task inside an existing project, the way your team actually works.\n\nHere are the results, resorted by my own math into tokens per resolved task. Full disclosure: I have not run these evaluations myself. Everything comes from the [official leaderboard](https://kotlinlang.org/benchmark/), the JetBrains announcement post, and the methodology page. The token math below is mine.\n\nRead that list twice and the ranking stops mattering. Here is what jumps out.\n\n**Reasoning effort has terrible marginal returns on the margin you would pay.** Claude Code with Opus 4.7 medium spent half the tokens of the xhigh setup and solved 80 of the 90 tasks xhigh solved. The extra ten tasks cost 5.28M extra tokens, which works out to about 528K tokens per additional task, roughly **8 times the average cost per task** of the medium setup. If your tasks cluster near the easy end, xhigh reasoning is mostly paying premium prices for tasks medium already solved cheaply.\n\n**The same model in a different loop differs by 3.7x.** Codex with GPT 5.5 xhigh tied Junie with Opus 4.7 max at 86 tasks. Codex did it with 38.78M tokens, Junie with 19.98M. The agent harness around the model matters as much as the model itself. This is the strongest public evidence yet for something practitioners keep saying: you are not picking a model, you are picking a loop.\n\n**Flash models are not cheap.** Two Gemini 3 Flash entries sit at the bottom of the token-efficiency table, at 628K and 777K tokens per solved task. The obvious theory: weaker models fail more, and every failed attempt burns a full context of tokens before the verification step says no. Cheap-per-token is not cheap-per-solved-task when the failure rate is high. The real unit of cost is not the token, it is the solved task.\n\n**Wall-clock time is a separate budget.** Junie with Opus 4.7 max needed nearly 20 hours to finish the suite. Claude Code medium finished in under 5. If your workflow is a developer waiting on an agent between code reviews, a 4x slower loop is a real cost even at identical token counts.\n\nOne number from a different benchmark puts this whole leaderboard in perspective. Yesterday I wrote about [Real-SWE](https://dev.to/jamilxt/8-frontier-models-10-real-enterprise-tickets-the-benchmark-that-humbled-the-leaderboards-1b0h), a benchmark that ran 8 frontier models on 10 tasks from private enterprise codebases. The best model there scored **38.8 percent**.\n\nThe Kotlin Benchmark's top score is 85.7 percent. Same year, same models, same general category of work. The gap is not a mystery, and it is not mostly about language.\n\nJetBrains itself says the quiet part in the announcement: the scores are \"intended as a signal, not a guarantee for your codebase,\" and real results depend on your architecture, internal APIs, and validation process. Neither benchmark is lying. They are measuring different things, and the difference between 85.7 and 38.8 is roughly the difference between your public repo work and your private repo work.\n\nIf you run Kotlin or Java server-side work and you are evaluating agent setups, this benchmark is the closest public proxy you have. Here is the approach it suggests.\n\nJetBrains did the ecosystem a favor here, and not just Kotlin developers. The interesting output of this release is not that Claude Code leads, or that Junie and Codex are close. It is that publishing tokens and latency next to resolution rates makes the quality ranking look almost like a distraction. Once you sort by cost per solved task, most teams' optimal setup moves down the leaderboard, and the assumption that more reasoning and cheaper models always help dies on the same page.\n\nThe benchmark is open, the tasks are inspectable on GitHub, and JetBrains plans to expand coverage to Android and Kotlin Multiplatform, add cost and maintainability metrics, and evaluate more open-weight models. If they add a Java edition with private-repo task sourcing, it might be the most useful benchmark in the JVM ecosystem. Until then, use this one the way it is meant to be used: as a signal, then run your own scoreboard on your own tickets.\n\nI write about AI agents, JVM ecosystems, and backend engineering every week. If that is your world, subscribing is free and every post is grounded in linked sources.\n\nHave you run coding agents against your own codebase as a benchmark? What did your tokens-per-merged-task number look like compared to the public leaderboards?", "url": "https://wpnews.pro/news/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real", "canonical_source": "https://dev.to/jamilxt/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real-story-22li", "published_at": "2026-09-15 16:06:48+00:00", "updated_at": "2026-09-15 16:20:28.414993+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "developer-tools", "large-language-models", "ai-tools"], "entities": ["JetBrains", "Kotlin Benchmark", "Claude Code", "Opus 4.7", "JetBrains Junie", "OpenAI", "Codex", "GPT 5.5"], "alternates": {"html": "https://wpnews.pro/news/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real", "markdown": "https://wpnews.pro/news/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real.md", "text": "https://wpnews.pro/news/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real.txt", "jsonld": "https://wpnews.pro/news/jetbrains-ranked-ai-agents-on-real-kotlin-projects-the-token-column-is-the-real.jsonld"}}