# Introducing the Databricks AI Index: The state of enterprise AI tokenomics

> Source: <https://www.databricks.com/blog/introducing-databricks-ai-index-state-enterprise-ai-tokenomics>
> Published: 2026-11-16 17:00:00+00:00

A living index of real-world data on enterprise AI model adoption, spending patterns, and coding agent performance

Developers have more AI models and coding agents to choose from than ever before. Since May 2026, a new flagship proprietary or open-weight model has launched roughly every week, or every 3.7 days when regional labs are included. With so many options, knowing which models perform best, what they cost, and when to switch is becoming harder.

At Databricks, we have a unique perspective on this rapidly changing landscape. Thousands of enterprises use [Unity Gateway](https://www.databricks.com/product/artificial-intelligence/unity-gateway) to access and govern AI models, providing us with insights into real-world model adoption and spending. Databricks also heavily leverages AI internally across our 12,000+ employees, and we've invested substantial research in evaluating the quality and cost of new frontier models on enterprise tasks.

Today, we're launching the [Databricks AI Index](https://developers.databricks.com/ai-index), a public resource that brings these insights together. Customers have repeatedly asked us how new models compare and what we're seeing across enterprise AI adoption to help inform their own investments. Now, for the first time, we're making our internal evaluation suite permanently public, with weekly updates as we evaluate new proprietary and open-weight model releases.

The Index combines two complementary sources of data:

**Enterprise AI usage and spending.** Aggregated, anonymized data from 10,000+ companies using [Unity Gateway](https://www.databricks.com/product/artificial-intelligence/unity-gateway), spanning digital natives, Fortune 500, and Global 2000 enterprises, including traffic to external providers such as Anthropic and OpenAI. The Index tracks model adoption through token and spend shares across proprietary and open-weight models, as well as weekly AI spending patterns for individual users.

**Continuously updated coding-agent evaluations.** Building on the [coding-agent benchmark we published in July](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase), our latest evaluations draw on real coding-agent workflows from Databricks engineers, captured through Unity Gateway. These workflows span creating and reviewing pull requests, investigating issues, and tackling complex development tasks. Using tasks from our private codebase and withheld tests helps reduce the risk of benchmark contamination while enabling consistent comparisons of model-harness combinations and terms of task success rates and cost per completed task.

Together, these data sources show which models enterprises are adopting, how well different combinations of models and harnesses perform, and what they cost. Here are some highlights:

Enterprises are increasingly shifting workloads between model versions and providers. At mid-year, most organizations relied on a handful of Anthropic models for the majority of their work. Today, adoption is far more fragmented across model versions and providers alike. OpenAI's share of enterprise token traffic grew meaningfully over this period, driven largely by its 6 series Sol and Astra models. However, model quality continues to strongly influence provider market share; when Claude Opus 5.5 launched, momentum shifted back to Anthropic. In just six weeks, Opus 5.5 surged from 7.2% to 37.9% of proprietary-model tokens, lifting Anthropic's overall share from 57.6% to 68.8%.

The pattern of shifting workloads between models and providers aggressively matches our own AI strategy at Databricks, where we quickly evaluate and roll out new models. Two months ago, we introduced an internal [Day 1 model rollout process](https://www.databricks.com/blog/how-databricks-rolls-out-frontier-models-12000-employees-day-1) which lets us evaluate new releases within days. This process has helped us manage cost while increasing the quality of internal AI efforts. For example, we found Opus 5.5 delivered improved quality at 29% lower cost per session than its predecessor, helping inform our decision to make it the default model for Claude Code.

Enterprises are increasingly moving beyond a single AI provider. As recent as July, approximately 64% of enterprises using Unity Gateway relied on just one model provider. By November 2, that had fallen to 28% as more organizations adopted models from more 2 or more providers.

Open-weight models are gaining ground alongside this shift. Their share of enterprise token traffic grew from 7.3% in July to 30.9% in November, more than quadrupling in less than four months. Open-weight models are also becoming increasingly competitive with proprietary alternatives. In our coding quality benchmark, three open-weight models rank among the top 10, including DeepSeek V4.1 and GLM 5.3, which both rank third with a 93.1% pass rate. As we explored in our [research on open-weight models](https://www.databricks.com/blog/enterprise-case-open-weight-models-databricks), their growing capabilities, cost efficiency, and flexibility are making them an increasingly attractive option for enterprises.

Choosing the right coding agent means evaluating both the model and the harness running it. Our latest coding-agent benchmark evaluates models across nine AI labs using Claude Code and Codex. Five models from four labs (OpenAI, DeepSeek, Meta, and Anthropic) sit on the cost-quality frontier, delivering the best trade-offs between task success rates and cost.

The results show why model rankings alone don't tell the full story. Enterprises need to evaluate model and harness combinations against their own workloads, using real usage data and traces to understand the trade-offs between performance and cost.

We use this approach internally at Databricks. When evaluating whether to upgrade from Opus 4.8 to Opus 5, we analyzed the cost-quality Pareto frontier and replayed traces from our own workloads. The results showed the upgrade wasn't worth the additional cost for our use cases, so we decided to stay on 4.8.

Two models can deliver nearly the same benchmark performance at dramatically different costs. In our coding benchmark, GPT-6 Luna and Claude Opus 5.5 both achieved an 89.1% task success rate. However, Luna averaged just $0.05 per task compared with $1.62 for Opus 5.5, a roughly 32x difference.

Cost per task depends on both token prices and the amount of work a model must do to reach an answer. Extra turns, long outputs and failed attempts all increase cost. Gemini 3.8 Flash has a low token price, yet it averaged 140 turns per task and finished as one of the most expensive models in the benchmark at $11.97 per task. For coding agents, lower token prices don't guarantee lower costs when models require more work to complete a task.

Higher benchmark performance often comes at a steep price, and even small improvements in quality can carry high costs. Claude Fable 5.1 tops our benchmark with a 96.0% task success rate at $7.47 per task, while Muse Wonder scores 94.9% at just $1.53. That additional 1.1 percentage points of performance comes at nearly five times the cost. Enterprises appear to agree, and we saw Fable 5.1's share of proprietary tokens fall from 5.5% to 0.6% over 6 weeks since its launch.

The question for enterprises is how much additional quality a task actually requires. Premium models can be worth the cost for complex, high-stakes work, but less expensive alternatives may be sufficient for routine tasks. Intelligent model routing, such as [Smart Routing](https://www.databricks.com/blog/smart-routing-unity-ai-gateway-match-frontier-quality-30-lower-cost-task) in Unity Gateway, can help match tasks to appropriate models based on quality and cost.

AI spending varies dramatically across enterprise users. A developer running coding agents throughout the day has very different consumption patterns from those of an employee who asks an AI assistant a few questions a day. Our data highlights the stark difference: the median user spent just $2.49 on AI, while the 99th-percentile user spent $699.74, roughly 281x more.

Median weekly spending more than doubled from $1.10 over six weeks, while spending at the 90th and 99th percentile remained relatively flat. This suggests that AI consumption is growing among typical users, but a small group still accounts for most of the AI bill.

For enterprises, the trend makes visibility into the distribution of spending especially important. Understanding those differences helps organizations set more targeted budgets and spend controls without unnecessarily restricting usage.

The Databricks AI Index reveals industry-wide trends in model usage, performance, and costs. Databricks Unity Gateway helps enterprises respond to these trends across their own AI workloads:

**Access and switch models without lock-in.** Get centralized access to frontier, open-weight, and custom models through Databricks-hosted model serving or connections to external providers. With the [Unity Gateway CLI](https://docs.databricks.com/aws/en/ai-gateway/coding-agent-ug-cli), developers can also use approved models across supported coding harnesses such as Claude Code and Codex.

**Analyze your own AI tokenomics.** Use Unity Gateway [usage dashboards](https://docs.databricks.com/aws/en/ai-gateway/usage-tracking#built-in-usage-dashboard) and [unified traces](https://docs.databricks.com/aws/en/ai-gateway/unified-trace-table) to understand model consumption, spending, and coding agent activity across users, teams, and applications. You can analyze and replay traces to compare model performance and costs on your own workloads, just as Databricks does internally.

**Optimize model selection and costs.** Use [Smart Routing](https://www.databricks.com/blog/smart-routing-unity-ai-gateway-match-frontier-quality-30-lower-cost-task) to match coding agent tasks to cost-effective models based on quality and performance, and [apply budgets](https://docs.databricks.com/aws/en/ai-gateway/budgets) and spend caps to keep AI spending under control.

The Databricks AI Index combines aggregated, anonymized usage data from thousands of enterprises using Unity Gateway with coding agent benchmarks based on real pull requests from Databricks' private codebase. Model spending is estimated using publicly listed provider prices, and benchmark results are evaluated against hidden tests rather than an LLM judge. Results reflect the workloads and models evaluated and may vary across organizations. The current dataset covers September 21 through November 7, 2026.

Explore the [Databricks AI Index](https://developers.databricks.com/ai-index) to see the latest model usage trends, coding agent benchmarks, and cost comparisons. We'll continue updating the Index with the latest enterprise usage trends and coding-agent evaluations, giving developers and enterprises an ongoing resource for comparing models, performance, and cost.

Want to analyze your own AI usage, compare models, and optimize costs? Get started with the [Unity Gateway quickstart guide](https://docs.databricks.com/aws/en/unity-gateway/quickstart).

The Databricks AI Index is a public resource tracking enterprise AI model adoption, spending trends, and coding agent performance. It combines aggregated, anonymized usage data from thousands of enterprises using Databricks Unity Gateway with coding agent benchmarks based on real development tasks. The Index is updated weekly as new models are evaluated.

AI models for coding should be compared on task success rate, cost per task, and performance within different coding agent harnesses. The Databricks AI Index evaluates model-harness combinations using real coding workflows from Databricks' private codebase and hidden tests to measure quality and cost. Results can vary by workload, so enterprises should also evaluate models against their own development tasks.

Lower token prices don't always translate to lower costs because models vary in how much work they need to complete a task. Models that require more reasoning turns, generate longer outputs, or make repeated attempts can consume more tokens overall. Cost per completed task is often a more useful measure of AI efficiency than token price alone.

Yes, open-weight AI models are increasingly competitive with proprietary models in coding performance and cost efficiency. In the Databricks AI Index coding benchmark, three open-weight models rank among the top 10 for task success rate, including DeepSeek V4.1 at 93.1%. The best choice depends on the workload, required quality, and cost.

Enterprises can optimize AI costs by comparing model quality and cost on their own workloads, monitoring usage, and selecting models based on task requirements. Databricks Unity Gateway provides centralized model access, usage dashboards, unified traces, and budgets to help organizations manage AI spending. Its Smart Routing capability can also match coding agent tasks to cost-effective models based on quality and performance.

Subscribe to our blog and get the latest posts delivered to your inbox.
