{"slug": "lessons-from-databricks-multi-million-line-agent-benchmark", "title": "Lessons From Databricks' Multi-Million Line Agent Benchmark", "summary": "Databricks' internal benchmark of AI agents on its multi-million-line codebase reveals that token costs are misleading, agent harness design significantly impacts performance, and open-source models like GLM 5.2 rival proprietary ones. The evaluation found Claude 3 Opus outperformed Claude 3.5 Sonnet in cost and quality, while a simpler harness reduced costs by over 2x without quality loss. Databricks has deployed GLM 5.2 as a daily driver for its engineering team.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# Lessons From Databricks' Multi-Million Line Agent Benchmark\n\nAn internal evaluation reveals why token costs lie, harness design matters, and open-source models are ready for production.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\nPublic LLM leaderboards are losing their utility. Between data contamination and hyper-optimized prompts, generic scores rarely correlate with how an AI agent performs on a proprietary, multi-language codebase. To find out what actually works, [Databricks](https://www.databricks.com) built an internal benchmark using its own multi-million-line codebase.\n\nThe results, shared by co-founder Matei Zaharia, offer a masterclass in how to evaluate agentic workflows. They also bust several industry myths about pricing, model capabilities, and the role of the developer harness.\n\n## The Token Cost Illusion\n\nDevelopers often look at input and output token prices to estimate the cost of running an agent. Databricks' data shows this is a mistake. When comparing Anthropic's Claude 3.5 Sonnet (referred to as Sonnet 5 in the evaluation) to Claude 3 Opus (referred to as Opus 4.8), Sonnet 5 is roughly 1.7 times cheaper per token. Yet, on actual engineering tasks, Sonnet 5 ended up costing $2.09 per task compared to $1.94 for Opus 4.8, while scoring six percentage points lower on task completion (81% versus 87%).\n\nThis discrepancy comes down to reasoning efficiency. Sonnet 5 worked longer, read more context, and ultimately consumed 1.9 times more tokens to complete the same tasks. A model that seems expensive on a per-token basis can be far more token-efficient in practice because it reaches the correct solution with fewer iterations. When budgeting for agentic workflows, $/task is the only metric that matters.\n\n## The Harness is the Real Control Point\n\nThe model is only half the equation. The agent harness, which is the wrapper that manages state, prompts, and tool calls, is just as critical. Databricks ran the same models through different harnesses, comparing native environments like Claude Code or Codex against a simpler harness called Pi.\n\nThe choice of harness altered the cost per task by more than 2x, with zero drop in quality. Pi achieved this by sending about three times less context per turn. By maintaining a tighter, more disciplined working set of context, Pi finished tasks in fewer runs. For instance, running Opus 4.8 through Pi was 2.08 times cheaper than running it through native environments.\n\nSimilarly, when running GPT 5.5, the Codex harness consumed 1,235,000 tokens compared to just 665,000 tokens for Pi. To manage this complexity, Databricks built Omnigent, a meta-harness that allows developers to swap and compose agents dynamically, alongside Unity AI Gateway to monitor and gate LLM usage.\n\n## Open Source Claims the Top Tier\n\nFor a long time, open-source models were viewed as lightweight alternatives suited only for simple autocomplete. That era is over. The benchmark placed models into three distinct capability tiers.\n\nThe top tier (82% to 90% pass rate) included proprietary heavyweights like GPT 5.5 and Opus 4.8, but it also featured GLM 5.2, an open-source model. GLM 5.2 statistically tied with Opus 4.8 on quality but cost just $1.28 per task compared to Opus's $1.94.\n\nDatabricks has already begun deploying GLM 5.2 as a daily driver for its engineering team. They are not alone. Other companies like Coinbase and Snowflake are shifting workloads to highly efficient open models, driving up traffic for these alternatives on platforms like [OpenRouter](https://openrouter.ai).\n\n## How to Build a Real-World Benchmark\n\nIf you want to replicate this approach for your own team, you cannot rely on generic evaluations. Here is how Databricks structured their pipeline, and how you can build yours:\n\n**Use Real Pull Requests**: Extract recent, human-written PRs from your own repositories. Databricks' codebase spans over ten languages, including Scala, Go, Rust, Java, Python, and Bazel. Your benchmark must reflect your actual stack.**Enforce Strict Test-Based Scoring**: Never use an LLM as a judge to evaluate code quality. LLM judges are notoriously biased toward eloquent explanations rather than working code. Instead, score tasks purely on whether they pass your automated test suites.**Prevent Git Cheating**: During early runs, Databricks noticed models \"cheating\" by searching the Git history to find the original human solution. To fix this, you must truncate the entire Git history before handing the repository to the agent.**Route by Complexity**: Analyze your engineering tasks. Databricks used Unity AI Gateway to categorize their tasks, revealing that the vast majority do not require the most expensive models.\n\n```\nxychart-beta\n    title \"Distribution of Coding Tasks by Complexity\"\n    x-axis [Low, Medium, High]\n    y-axis \"Percentage of Tasks\" 0 --> 70\n    bar [19, 61, 12]\n```\n\nDo not route simple config changes to expensive frontier models. Push low and medium tasks to cheaper, highly efficient models like GPT 5.4 Mini or Haiku 4.5, and reserve the top tier for deep design explorations.\n\nThe Databricks benchmark proves that the future of agentic engineering is not about finding one master model. It is about building a flexible infrastructure, using tools like [SWE-bench](https://www.swebench.com) for inspiration but relying on internal pipelines, to route the right task to the right model-harness combination. By focusing on task-level costs and context management rather than raw token prices, engineering teams can dramatically lower their AI spend while shipping better code.\n\n## Sources & further reading\n\n-\n[Benchmarking coding agents on Databricks' multi-million line codebase](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase)— databricks.com -\n[Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase — brickster.ai](https://www.brickster.ai/news/databricks-blog/benchmarking-coding-agents-databricks-multi-million-line-codebase)— brickster.ai -\n[Digg](https://digg.com/tech/hwf03xak)— digg.com -\n[Databricks makes Chinese open-source model GLM 5.2 its default coding engine after it matched Opus at lower cost](https://the-decoder.com/databricks-makes-chinese-open-source-model-glm-5-2-its-default-coding-engine-after-it-matched-opus-at-lower-cost/)— the-decoder.com -\n[AI Coding on Databricks: Tools, Setup, and Best Practices](https://hiflylabs.com/blog/2026/4/17/ai-coding-databricks-tools-setup-best-practices)— hiflylabs.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/lessons-from-databricks-multi-million-line-agent-benchmark", "canonical_source": "https://sourcefeed.dev/a/lessons-from-databricks-multi-million-line-agent-benchmark", "published_at": "2026-07-09 13:05:20+00:00", "updated_at": "2026-07-09 13:11:11.948214+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-infrastructure", "ai-research"], "entities": ["Databricks", "Matei Zaharia", "Anthropic", "Claude 3.5 Sonnet", "Claude 3 Opus", "GLM 5.2", "OpenRouter", "Coinbase"], "alternates": {"html": "https://wpnews.pro/news/lessons-from-databricks-multi-million-line-agent-benchmark", "markdown": "https://wpnews.pro/news/lessons-from-databricks-multi-million-line-agent-benchmark.md", "text": "https://wpnews.pro/news/lessons-from-databricks-multi-million-line-agent-benchmark.txt", "jsonld": "https://wpnews.pro/news/lessons-from-databricks-multi-million-line-agent-benchmark.jsonld"}}