{"slug": "databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how", "title": "Databricks Benchmarked Coding Agents on Its Own Codebase. The Results Should Change How You Buy", "summary": "Databricks CTO Matei Zaharia's team built an internal benchmark on their own polyglot codebase to evaluate coding agents, finding that multiple models are now competitive at the top tier, including open-source GLM-5.2, and that a minimal harness called Pi achieved the same success rate as vendor harnesses at half the cost. The results challenge common assumptions about model choice and cost optimization, showing that cheaper per-token models can be more expensive per-task due to higher token usage.", "body_md": "Public coding benchmarks have a dirty secret: they all look suspiciously alike. SWE-bench and Terminal-Bench are valuable, but they’re dominated by Python-ish repos with fixed tests — and your codebase probably isn’t. So when Databricks CTO Matei Zaharia’s team wanted to know which coding agents actually work, they did the thing more companies should do: **built an internal benchmark on a sample of their own codebase** — a polyglot beast of Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet, and more — and published what they found.\n\nZaharia was careful about the epistemics up front: these are results on *their* sample of *their* codebase, not meant to be comprehensive — but many companies can run a similar internal benchmark. The findings are still the most useful coding-agent data of the summer, because two of them cut directly against how most teams choose their stack. Let’s take the four in turn.\n\nThe first result is the least surprising but sets the stage: **many models are now competitive at the top tier, including open source.** The era when one closed model was unambiguously ahead on real engineering work is over; on Databricks’ internal tasks, multiple models cluster at the top.\n\nThat has a practical consequence: if several models are near-equivalent on *your* tasks, then model choice stops being the decision that matters most — and the decisions that do matter shift downstream, to the harness and the economics. Which is exactly where the next three findings go.\n\nThe sharper version of finding one: **GLM-5.2 in particular was a major step forward in open-source coding-agent performance** — *even on a codebase that looks nothing like SWE-Bench or TerminalBench.*\n\nThat italicized clause is the whole point. The standing objection to open-model benchmark wins has always been distribution overlap: maybe the model is great at the kind of Python-repo task benchmarks measure, and falls apart on your Scala services and Protobuf schemas. Databricks’ codebase is close to a worst-case test of that objection — heavy in Scala, Go, Rust, Java, TypeScript, Protobuf, Jsonnet — and GLM-5.2 delivered anyway. When we covered GLM-5.2’s launch, the open question was whether the FrontierSWE numbers would survive contact with messy, unfamiliar reality. Here’s an independent, adversarially-different data point saying: largely, yes.\n\nThis is the finding that deserves to be framed and hung on the wall: **harnesses make a huge difference in cost-performance.** Specifically — the very simple **Pi harness** (Mario Zechner’s minimal, four-tool agent) achieved **the same success rate as the LLM vendors’ own harnesses running Opus and GPT-5.5 — at 2x less cost.** The mechanism, per Zaharia: mainly **smaller inputs to the LLM.**\n\nSit with what that means. Claude Code and Codex are harnesses their vendors co-train models against — the presumed home-field advantage. And a deliberately minimal harness, with the shortest system prompt in the business and four tools, matched them on task success while halving spend, simply by putting less in the context window. Every token of scaffolding a harness injects — tool descriptions, boilerplate instructions, machinery — is a cost you pay on *every model call of every step*. Pi’s minimalism, which read as philosophy when Armin Ronacher championed it, turns out to be **economics**.\n\nFor anyone running agents at scale, this reframes the optimization target: before you switch models, audit your harness’s input footprint. Half your bill might be scaffolding.\n\nThe subtlest finding is the one most likely to be costing you money right now: **cheaper per-token does not imply cheaper per-task.** In Databricks’ runs, Sonnet 5 costs less per token than Opus 4.8 — but *used more tokens*, resulting in **higher total cost and lower quality.**\n\nThe cheaper model took more steps, more retries, more verbose reasoning to do the same work — and lost on both axes. This isn’t a one-off quirk: Zaharia points to research from his former student Lingjiao Chen formalizing it as the [ “Price Reversal Phenomenon”](https://arxiv.org/abs/2506.02523v2) — the first systematic study showing listed API prices routinely misrepresent actual inference costs for reasoning models, across many tasks.\n\nThe buying implication is blunt: **the per-token rate card is nearly useless for comparing agentic workloads.** The only number that matters is cost-per-completed-task on your distribution — which you can only get by measuring, which loops back to why Databricks built this benchmark in the first place.\n\nThe findings explain two of the company’s recent bets. **Omnigent** — the open-source (Apache 2.0) “meta-harness,” a harness of harnesses in Zaharia’s phrase — exists because finding 3 implies you want to *switch and compose* agents and harnesses per task rather than marry one. And **Unity AI Gateway** exists because findings 3 and 4 imply someone needs to analyze and gate LLM usage centrally, since costs hide in places rate cards don’t show.\n\nBut the most transferable takeaway isn’t a product — it’s the method. Public benchmarks measure a distribution that probably isn’t yours; the top tier is crowded enough that rankings within it don’t transfer; and the real cost structure only reveals itself per-task. The move is Databricks’ move: **sample your own codebase, run the candidates through it, and measure success rate and cost-per-task yourself.** It’s less work than it sounds, and it’s the only benchmark whose distribution is guaranteed to match production.\n\nThe honest caveats cut both ways, and Zaharia named them himself: it’s one company’s sample, not a leaderboard, and internal benchmarks carry their own biases (task selection, harness familiarity). But that’s precisely the point — the value isn’t that Databricks’ numbers generalize to you. It’s that their *method* does.\n\n*The full write-up on how the benchmark was built is on the **Databricks blog**, with the Price Reversal paper on **arXiv** and Omnigent open-sourced under Apache 2.0. If you take one action from this: pull twenty representative tasks from your own repo, run your current agent stack against two alternatives, and compare cost-per-completed-task. Finding 3 and 4 suggest the result will surprise you in at least one direction.*\n\n[Databricks Benchmarked Coding Agents on Its Own Codebase. The Results Should Change How You Buy](https://pub.towardsai.net/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-change-how-you-buy-ed4a0b543def) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how", "canonical_source": "https://pub.towardsai.net/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-change-how-you-buy-ed4a0b543def?source=rss----98111c9905da---4", "published_at": "2026-07-11 15:01:02+00:00", "updated_at": "2026-07-11 15:39:42.565281+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Databricks", "Matei Zaharia", "GLM-5.2", "Pi", "Mario Zechner", "Claude Code", "Codex", "Armin Ronacher"], "alternates": {"html": "https://wpnews.pro/news/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how", "markdown": "https://wpnews.pro/news/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how.md", "text": "https://wpnews.pro/news/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how.txt", "jsonld": "https://wpnews.pro/news/databricks-benchmarked-coding-agents-on-its-own-codebase-the-results-should-how.jsonld"}}