Databricks Benchmarks Coding Agents on Its Own Codebase

Databricks published an internal coding-agent benchmark on July 8 using tasks from its multi-million-line codebase, reporting that model choice, task difficulty, token use, and the agent harness affected cost and completion rates, with more than twofold cost differences for the same model in some harness comparisons. The company found that GLM 5.2 tied with Opus 4.8 in quality at $1.28 per task versus $1.94, and that Sonnet 5 cost $2.09 per task with an 81% score while Opus 4.8 cost $1.94 with 87%, due to token consumption. The results are self-reported on private workloads and have not been independently reproduced.

Databricks Benchmarks Coding Agents on Its Own Codebase Databricks published a July 8 internal benchmark of coding agents on tasks drawn from its multi-million-line codebase. The company reported that model choice, task difficulty, token use and the agent harness all changed cost and completion rates, including more than twofold cost differences for the same model in some harness comparisons. The results are self-reported and have not been independently reproduced. Databricks published an internal coding-agent benchmark on July 8 using tasks drawn from pull requests against its multi-million-line codebase. The company said the evaluation covered work in languages and systems including Python, Go, TypeScript, Scala, Rust, Java, Bazel and Protobuf. A separate developer-news report summarized the same findings after publication. The benchmark is useful as a production case study, but it is not an independent or generally reproducible leaderboard. Databricks did not publish the private task corpus or enough raw run data for outsiders to repeat the full evaluation, so its model and cost figures should be read as results on the company's own workloads. How Databricks built the evaluation Databricks said it derived tasks from recent, human-written pull requests, then reviewed the prompts and tests for accuracy. Test files were held out while agents worked, and completed changes were graded with those tests rather than an LLM judge. The company also reported fixing an early benchmark leak: because each task came from a merged change, agents could initially recover the known implementation from Git history. Databricks said it removed repository history from the working copy during later runs. That detail matters because an agent finding the original patch would measure retrieval of an answer rather than software-engineering ability. What changed cost and quality Databricks reported four broad findings: - •Its cost-versus-quality frontier included models from OpenAI, Anthropic and open-model providers, so no single provider dominated every price point. - •GLM 5.2 landed in the top capability tier in this internal evaluation. Databricks reported quality statistically tied with Opus 4.8 at $1.28 per task versus $1.94 for Opus. - •Per-token price did not predict end-to-end cost. The company said Sonnet 5 cost $2.09 per task and scored 81%, while Opus 4.8 cost $1.94 and scored 87%, because Sonnet consumed more tokens on these tasks. - •The harness materially affected efficiency. With the same model and reasoning setting, Databricks said Claude Code or Codex and Pi differed by more than twofold in cost in some comparisons while quality stayed similar; Pi supplied about one-third as much context per turn. For engineering teams, the practical takeaway is to evaluate the complete agent system on representative internal tasks. Model, harness, context policy, task mix and tests all shape the result. Databricks' figures provide hypotheses to test, not universal purchasing benchmarks. Key Points - 1Databricks built an internal benchmark from reviewed pull requests and held-out tests on its own multi-million-line codebase. - 2The company reported that model pricing alone did not predict task cost and that harness choice produced more than twofold cost differences in some same-model comparisons. - 3The model and cost results are self-reported on private workloads and have not been independently reproduced. Scoring Rationale The case study gives engineering teams concrete evidence that agent harnesses, context use and workload difficulty can matter as much as model selection. Its practical relevance is high, but the score is moderated because the corpus and complete raw results are private and the findings are not independently reproduced. Sources Primary source and supporting public references used for this report. Practice interview problems based on real data 1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with. Try 250 free problems /problems