In April 2026, Uber burned through its entire year’s AI budget by the middle of the month, driven largely by usage of Anthropic's Claude Code and Cursor. Nothing was broken. No one misused the tool. Engineers were doing exactly what every team's been told to do: adopt AI, use it well, use it wherever needed.
Uber's team wasn't being careless. They had no way to see what each prompt, retry, or agent step was costing until the total already reflected it. That blind spot, irrespective of the pricing tier, is where the real cost problem starts.
This post is based on Kong's webinar on token cost management. Watch it below, or keep reading for the breakdown.
This content contains a video which can not be displayed in Agent mode
What is token cost management? #
Token cost management is how a business tracks, controls, and governs what it spends on AI model usage, the same way it already tracks payroll or cloud spend.
That means visibility into who's spending what, control over how spend happens rather than just tracking it after the fact, ongoing management as usage shifts, and auditability to trace a dollar back to the request that spent it.
Get the balance wrong either way, and it costs you. Too tight, and engineers stop using the tools that make them productive. Too loose, and you get the kind of budget blowout Uber just went through. Businesses need a way to control both token cost and token behavior, not just watch the number climb.
Why does AI token expenditure get out of control so easily? #
Token spend gets out of control along two axes: behavior and governance.
Behavior is about balancing token quality against cost without having to micromanage the details of who's asking, what they're asking, and how. Prompt length, model choice, how chatty an agent is, and how many times a request gets retried all move the bill, independently of how much value actually got delivered. Traditional cloud costs track fairly predictably with usage you can forecast: more users, more servers, roughly linear math. However, AI tokens don't work that way. The same feature can cost 10 times more or less depending on which model answered it, and none of that shows up until the invoice does.
The other half is** governance,** and it has to cover three things at once:
-
Govern every asset. Every agent action depends on three things together: the models providing intelligence, the tools it calls (APIs, business processes, other agents), and the context it pulls from (MCP, APIs, event streams, data lakes). If you miss even one of them, governance becomes partial.
-
**Govern every angle. **Security, compliance, cost, resilience, visibility, performance, discoverability — the full surface, not a partial checklist.
-
Govern every workload. Wherever these workloads run, across clouds, model providers, on-prem, or data platforms, no environment should be exempted.
Truth be told,organizations aren't covering all three today. Kong's AI Governance Gap ReportAI Governance Gap Report analyzed millions of live production API calls and found that 99% of organizations don't have AI-specific governance controls in place. Also, 62% organizations are already spending across multiple models at once, which means the surface area for untracked cost keeps growing faster than most teams can keep up with.
What causes runaway AI token costs? #
Runaway AI token costs usually trace back to four missing guardrails:
-
**Ungoverned Choice: **Every request — simple or complex — hits the same model, so you pay frontier-model prices to answer questions a cheaper model could handle just as well.
-
Unchecked Spend: Spend surfaces in a report weeks later, long after the decision that caused it. Without real-time alerting for AI spend overruns, teams cannot course-correct mid-sprint.
-
Unknown Quality. A "cheap" model that needs two retries to produce a usable answer often costs more than the expensive one that got it right the first time.
-
Siloed Visibility: One central team owns the whole picture, so nobody closer to the usage can see what's actually driving it.
That last point compounds fast. Usage gets spread across local environments, cloud environments, and multiple model providers with no central view, creating a form of "shadow token consumption" — spend nobody can see, and so nobody can plan around. To prevent shadow token consumption, organizations must centralize their telemetry so every token is tracked back to a specific user or application.
How do you control token AI costs without slowing teams down? #
The answer isn't spending less but spending ***intentionally. *** The first step to this is matching each request to the model it actually needs and giving teams the feedback to self-correct in real time.
In practice, effective AI cost management means:
-
Routing requests to cost-appropriate models based on task complexity. For example a question about how to make spaghetti doesn't need the same model as an enterprise architecture decision. So, using the expensive model for both just wastes tokens.
-
Making cost a live signal at the request layer, not a line item that shows up in next month's report. This enables immediate intervention to stop engineers from overusing tokens on low-priority tasks.
-
Giving individual teams their own usage view, so they can adjust their own behavior instead of waiting for a central team to flag it.
It's the difference between "go build, go spend, max out your tokens" and building with a plan that has margins baked in from the start.
Where should AI cost controls live in your stack? #
Honestly, your AI stack doesn't live in one place. It spans your entire estate: different clouds, different model providers, on-prem, the edge, wherever a request happens to originate. Controlling cost means controlling it everywhere at once, across models, agents, edges, clouds, and APIs, not just the pieces that are easiest to reach.
That's what an AI control tower does. It gives you one governed, holistic view of cost across your assets, angles, and workloads.
-
Routes by cost and intent. A gateway with semantic and intent-based routing gives every application one endpoint to call. And then decides which model actually answers based on the complexity of the request.
-
Turns token counts into dollars, broken down by who spent them. Gateway-level observability, built on something like OpenTelemetry, shows consumption per team, per application, and per model, converted into an actual dollar figure, not a token count someone has to translate by hand at the end of the month.
-
Enforces rate limits before anyone has to ask for a budget increase. AI rate limiting sets soft limits that alert a team as it approaches its budget, and hard limits that cut off consumption entirely. Budgets and wallets work the same way at the account level, capping what a team can draw down before anyone has to step in manually.
-
Meters and attributes spent per team, in real time. Defining entitlements per team or project, then tracking consumption against them as it happens, is what turns raw cost data into chargeback, showback, or even real-time invoicing, instead of a spreadsheet someone reconciles weeks later.
On the contrary, a dashboard can't do any of this. It pulls in logs after the fact and displays them for someone to review. This can be useful for history, but it can only tell you that one of the four things above should have happened. It can't make any of them happen.
Truth be told, this is the same shift API governance already went through: traffic used to get logged and reviewed later, now it gets controlled as it happens. Token governance is following the same path. And the fastest way isn't rebuilding your whole stack, it's turning on one of these four at the gateway you already have, starting with rate limits or cost-based routing.
Getting started with token cost management #
Route every model request through the gateway first. From there, build outward. Match requests to the right model for the task, replace after-the-fact reports with real-time cost signals, and give teams their own visibility instead of leaving one team to own a black box.
How do I cap OpenAI and Anthropic usage costs?
To effectively cap usage costs for providers like OpenAI and Anthropic, you should route all LLM requests through an AI gateway rather than connecting applications directly to the provider's API. At the gateway level, you can enforce strict token budgets, set up team-level wallets, and implement real-time rate limiting to ensure no single application or user exceeds their allocated spend for GPT-4 or Claude models.
What is the difference between rate limits and budgets for AI cost control?
Rate limits control the velocity of your AI spend by restricting how many requests or tokens a user can consume within a short timeframe (e.g., tokens per minute). Budgets control the total volume of your spend by setting a hard financial ceiling over a longer period (e.g., dollars per month). Effective AI token cost management requires both: rate limits to prevent sudden spikes, and budgets to prevent long-term overruns.
How can I prevent shadow token consumption?
Shadow token consumption occurs when developers use unauthorized API keys or route requests through unmonitored local environments. You can prevent this by requiring all AI traffic to pass through a centralized AI gateway. This provides a single control plane where every token is authenticated, logged, and attributed to a specific team or project, eliminating blind spots in your AI spend.
What is an AI gateway and why is it essential for token budgeting?
An AI gateway is an architectural layer that sits between your applications and the AI models they interact with. It is essential for token budgeting because it acts as an active enforcement point. Unlike a dashboard that only reports on costs after they have occurred, an AI gateway intercepts the request in real-time, checking the team's token budget and blocking the request if the funds are exhausted.
An AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That "what it knows" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,
Hugo Guerrero
Multi-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s
Hugo Guerrero
Multi-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:
Hugo Guerrero
The Anatomy of Architectural Complexity Modern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity. North-South API Traf
Kong
Executive Summary AI adoption has moved past the "honeymoon phase" and into the "operational chaos" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and "Shadow AI" usage, the need for a centralized control plane has be
Kong
You Can't Govern What You Can't See AI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from
Augusto Marietti
Your infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho
Hugo Guerrero
An AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That "what it knows" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,
Hugo Guerrero
Multi-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s
Hugo Guerrero
Multi-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:
Hugo Guerrero
The Anatomy of Architectural Complexity Modern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity. North-South API Traf
Kong
Executive Summary AI adoption has moved past the "honeymoon phase" and into the "operational chaos" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and "Shadow AI" usage, the need for a centralized control plane has be
Kong
You Can't Govern What You Can't See AI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from
Augusto Marietti
Your infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho
Hugo Guerrero
An AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That "what it knows" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,
Hugo Guerrero
Multi-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s
Hugo Guerrero
Multi-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:
Hugo Guerrero
The Anatomy of Architectural Complexity Modern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity. North-South API Traf
Kong
Executive Summary AI adoption has moved past the "honeymoon phase" and into the "operational chaos" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and "Shadow AI" usage, the need for a centralized control plane has be
Kong
You Can't Govern What You Can't See AI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from
Augusto Marietti
Your infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho
Hugo Guerrero
Ready to see Kong in action? #
Get a personalized walkthrough of Kong's platform tailored to your architecture, use cases, and scale requirements.