AIArticle
An internal evaluation reveals why token costs lie, harness design matters, and open-source models are ready for production.
Public LLM leaderboards are losing their utility. Between data contamination and hyper-optimized prompts, generic scores rarely correlate with how an AI agent performs on a proprietary, multi-language codebase. To find out what actually works, Databricks built an internal benchmark using its own multi-million-line codebase.
The results, shared by co-founder Matei Zaharia, offer a masterclass in how to evaluate agentic workflows. They also bust several industry myths about pricing, model capabilities, and the role of the developer harness.
The Token Cost Illusion #
Developers often look at input and output token prices to estimate the cost of running an agent. Databricks' data shows this is a mistake. When comparing Anthropic's Claude 3.5 Sonnet (referred to as Sonnet 5 in the evaluation) to Claude 3 Opus (referred to as Opus 4.8), Sonnet 5 is roughly 1.7 times cheaper per token. Yet, on actual engineering tasks, Sonnet 5 ended up costing $2.09 per task compared to $1.94 for Opus 4.8, while scoring six percentage points lower on task completion (81% versus 87%).
This discrepancy comes down to reasoning efficiency. Sonnet 5 worked longer, read more context, and ultimately consumed 1.9 times more tokens to complete the same tasks. A model that seems expensive on a per-token basis can be far more token-efficient in practice because it reaches the correct solution with fewer iterations. When budgeting for agentic workflows, $/task is the only metric that matters.
The Harness is the Real Control Point #
The model is only half the equation. The agent harness, which is the wrapper that manages state, prompts, and tool calls, is just as critical. Databricks ran the same models through different harnesses, comparing native environments like Claude Code or Codex against a simpler harness called Pi.
The choice of harness altered the cost per task by more than 2x, with zero drop in quality. Pi achieved this by sending about three times less context per turn. By maintaining a tighter, more disciplined working set of context, Pi finished tasks in fewer runs. For instance, running Opus 4.8 through Pi was 2.08 times cheaper than running it through native environments.
Similarly, when running GPT 5.5, the Codex harness consumed 1,235,000 tokens compared to just 665,000 tokens for Pi. To manage this complexity, Databricks built Omnigent, a meta-harness that allows developers to swap and compose agents dynamically, alongside Unity AI Gateway to monitor and gate LLM usage.
Open Source Claims the Top Tier #
For a long time, open-source models were viewed as lightweight alternatives suited only for simple autocomplete. That era is over. The benchmark placed models into three distinct capability tiers.
The top tier (82% to 90% pass rate) included proprietary heavyweights like GPT 5.5 and Opus 4.8, but it also featured GLM 5.2, an open-source model. GLM 5.2 statistically tied with Opus 4.8 on quality but cost just $1.28 per task compared to Opus's $1.94.
Databricks has already begun deploying GLM 5.2 as a daily driver for its engineering team. They are not alone. Other companies like Coinbase and Snowflake are shifting workloads to highly efficient open models, driving up traffic for these alternatives on platforms like OpenRouter.
How to Build a Real-World Benchmark #
If you want to replicate this approach for your own team, you cannot rely on generic evaluations. Here is how Databricks structured their pipeline, and how you can build yours:
Use Real Pull Requests: Extract recent, human-written PRs from your own repositories. Databricks' codebase spans over ten languages, including Scala, Go, Rust, Java, Python, and Bazel. Your benchmark must reflect your actual stack.Enforce Strict Test-Based Scoring: Never use an LLM as a judge to evaluate code quality. LLM judges are notoriously biased toward eloquent explanations rather than working code. Instead, score tasks purely on whether they pass your automated test suites.Prevent Git Cheating: During early runs, Databricks noticed models "cheating" by searching the Git history to find the original human solution. To fix this, you must truncate the entire Git history before handing the repository to the agent.Route by Complexity: Analyze your engineering tasks. Databricks used Unity AI Gateway to categorize their tasks, revealing that the vast majority do not require the most expensive models.
xychart-beta
title "Distribution of Coding Tasks by Complexity"
x-axis [Low, Medium, High]
y-axis "Percentage of Tasks" 0 --> 70
bar [19, 61, 12]
Do not route simple config changes to expensive frontier models. Push low and medium tasks to cheaper, highly efficient models like GPT 5.4 Mini or Haiku 4.5, and reserve the top tier for deep design explorations.
The Databricks benchmark proves that the future of agentic engineering is not about finding one master model. It is about building a flexible infrastructure, using tools like SWE-bench for inspiration but relying on internal pipelines, to route the right task to the right model-harness combination. By focusing on task-level costs and context management rather than raw token prices, engineering teams can dramatically lower their AI spend while shipping better code.
Sources & further reading #
Benchmarking coding agents on Databricks' multi-million line codebase— databricks.com - Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase — brickster.ai— brickster.ai - Digg— digg.com - Databricks makes Chinese open-source model GLM 5.2 its default coding engine after it matched Opus at lower cost— the-decoder.com - AI Coding on Databricks: Tools, Setup, and Best Practices— hiflylabs.com
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.