{"slug": "ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it", "title": "AI Token Cost Management: Why AI Spend Gets Out of Control (and How To Fix It)", "summary": "Kong's AI Governance Gap Report, which analyzed millions of live production API calls, found that 99% of organizations lack AI-specific governance controls and 62% already spend across multiple models at once, according to a Kong webinar on token cost management. The report cites Uber burning through its entire 2026 AI budget by mid-April 2026, driven largely by usage of Anthropic's Claude Code and Cursor, as an example of ungoverned token spend. Kong attributes runaway costs to four missing guardrails: ungoverned model choice, unchecked spend with no real-time alerting, unknown quality, and incomplete governance across assets, angles, and workloads.", "body_md": "In April 2026, Uber burned through its entire year’s AI budget by the middle of the month, driven largely by usage of Anthropic's Claude Code and Cursor. Nothing was broken. No one misused the tool. Engineers were doing exactly what every team's been told to do: adopt AI, use it well, use it wherever needed.\n\nUber's team wasn't being careless. They had no way to see what each prompt, retry, or agent step was costing until the total already reflected it. That blind spot, irrespective of the pricing tier, is where the real cost problem starts.\n\n*This post is based on Kong's webinar on token cost management. Watch it below, or keep reading for the breakdown.*\n\n**This content contains a video which can not be displayed in Agent mode**\n\n## What is token cost management?\n\nToken cost management is how a business tracks, controls, and governs what it spends on AI model usage, the same way it already tracks payroll or cloud spend.\n\nThat means visibility into who's spending what, control over how spend happens rather than just tracking it after the fact, ongoing management as usage shifts, and auditability to trace a dollar back to the request that spent it.\n\nGet the balance wrong either way, and it costs you. Too tight, and engineers stop using the tools that make them productive. Too loose, and you get the kind of budget blowout Uber just went through. Businesses need a way to control both token cost and token behavior, not just watch the number climb.\n\n## Why does AI token expenditure get out of control so easily?\n\nToken spend gets out of control along two axes: **behavior and governance.**\n\n**Behavior** is about balancing token quality against cost without having to micromanage the details of who's asking, what they're asking, and how. Prompt length, model choice, how chatty an agent is, and how many times a request gets retried all move the bill, independently of how much value actually got delivered. Traditional cloud costs track fairly predictably with usage you can forecast: more users, more servers, roughly linear math. However, AI tokens don't work that way. The same feature can cost 10 times more or less depending on which model answered it, and none of that shows up until the invoice does.\n\nThe other half is** governance,** and it has to cover three things at once:\n\n- **Govern every asset.** Every agent action depends on three things together: the models providing intelligence, the tools it calls (APIs, business processes, other agents), and the context it pulls from (MCP, APIs, event streams, data lakes). If you miss even one of them, governance becomes partial.\n\n- **Govern every angle. **Security, compliance, cost, resilience, visibility, performance, discoverability — the full surface, not a partial checklist.\n\n- **Govern every workload.** Wherever these workloads run, across clouds, model providers, on-prem, or data platforms, no environment should be exempted.\n\nTruth be told,organizations aren't covering all three today. Kong's [_AI Governance Gap Report_](https://konghq.com/resources/reports/enterprise-ai-governance-gap-report)_AI Governance Gap Report_ analyzed millions of live production API calls and found that 99% of organizations don't have AI-specific governance controls in place. Also, 62% organizations are already spending across multiple models at once, which means the surface area for untracked cost keeps growing faster than most teams can keep up with.\n\n## What causes runaway AI token costs?\n\nRunaway AI token costs usually trace back to four missing guardrails:\n\n- **Ungoverned Choice: **Every request — simple or complex — hits the same model, so you pay frontier-model prices to answer questions a cheaper model could handle just as well.\n\n- **Unchecked Spend:** Spend surfaces in a report weeks later, long after the decision that caused it. Without real-time alerting for AI spend overruns, teams cannot course-correct mid-sprint.\n\n- **Unknown Quality.** A \"cheap\" model that needs two retries to produce a usable answer often costs more than the expensive one that got it right the first time.\n\n- **Siloed Visibility:** One central team owns the whole picture, so nobody closer to the usage can see what's actually driving it.\n\nThat last point compounds fast. Usage gets spread across local environments, cloud environments, and multiple model providers with no central view, creating a form of \"shadow token consumption\" — spend nobody can see, and so nobody can plan around. To prevent shadow token consumption, organizations must centralize their telemetry so every token is tracked back to a specific user or application.\n\n## How do you control token AI costs without slowing teams down?\n\nThe answer isn't spending less but spending ***intentionally. *** The first step to this is matching each request to the model it actually needs and giving teams the feedback to self-correct in real time.\n\nIn practice, effective AI cost management means:\n\n- Routing requests to cost-appropriate models based on task complexity. For example a question about how to make spaghetti doesn't need the same model as an enterprise architecture decision. So, using the expensive model for both just wastes tokens.\n\n- Making cost a live signal at the request layer, not a line item that shows up in next month's report. This enables immediate intervention to stop engineers from overusing tokens on low-priority tasks.\n\n- Giving individual teams their own usage view, so they can adjust their own behavior instead of waiting for a central team to flag it.\n\nIt's the difference between *\"go build, go spend, max out your tokens\"* and building with a plan that has margins baked in from the start.\n\n## Where should AI cost controls live in your stack?\n\nHonestly, your AI stack doesn't live in one place. It spans your entire estate: different clouds, different model providers, on-prem, the edge, wherever a request happens to originate. Controlling cost means controlling it everywhere at once, across models, agents, edges, clouds, and APIs, not just the pieces that are easiest to reach.\n\nThat's what an AI control tower does. It gives you one governed, holistic view of cost across your assets, angles, and workloads.\n\n- **Routes by cost and intent.** A gateway with semantic and intent-based routing gives every application one endpoint to call. And then decides which model actually answers based on the complexity of the request.\n\n- **Turns token counts into dollars, broken down by who spent them.** Gateway-level observability, built on something like OpenTelemetry, shows consumption per team, per application, and per model, converted into an actual dollar figure, not a token count someone has to translate by hand at the end of the month.\n\n- **Enforces rate limits before anyone has to ask for a budget increase.** AI rate limiting sets soft limits that alert a team as it approaches its budget, and hard limits that cut off consumption entirely. Budgets and wallets work the same way at the account level, capping what a team can draw down before anyone has to step in manually.\n\n- **Meters and attributes spent per team, in real time.** Defining entitlements per team or project, then tracking consumption against them as it happens, is what turns raw cost data into chargeback, showback, or even real-time invoicing, instead of a spreadsheet someone reconciles weeks later.\n\nOn the contrary, a dashboard can't do any of this. It pulls in logs after the fact and displays them for someone to review. This can be useful for history, but it can only tell you that one of the four things above should have happened. It can't make any of them happen.\n\nTruth be told, this is the same shift API governance already went through: traffic used to get logged and reviewed later, now it gets controlled as it happens. Token governance is following the same path. And the fastest way isn't rebuilding your whole stack, it's turning on one of these four at the gateway you already have, starting with rate limits or cost-based routing.\n\n## Getting started with token cost management\n\nRoute every model request through the gateway first. From there, build outward. Match requests to the right model for the task, replace after-the-fact reports with real-time cost signals, and give teams their own visibility instead of leaving one team to own a black box.\n\n**How do I cap OpenAI and Anthropic usage costs?**\n\nTo effectively cap usage costs for providers like OpenAI and Anthropic, you should route all LLM requests through an AI gateway rather than connecting applications directly to the provider's API. At the gateway level, you can enforce strict token budgets, set up team-level wallets, and implement real-time rate limiting to ensure no single application or user exceeds their allocated spend for GPT-4 or Claude models.\n\n**What is the difference between rate limits and budgets for AI cost control?**\n\nRate limits control the *velocity* of your AI spend by restricting how many requests or tokens a user can consume within a short timeframe (e.g., tokens per minute). Budgets control the *total volume* of your spend by setting a hard financial ceiling over a longer period (e.g., dollars per month). Effective AI token cost management requires both: rate limits to prevent sudden spikes, and budgets to prevent long-term overruns.\n\n**How can I prevent shadow token consumption?**\n\nShadow token consumption occurs when developers use unauthorized API keys or route requests through unmonitored local environments. You can prevent this by requiring all AI traffic to pass through a centralized AI gateway. This provides a single control plane where every token is authenticated, logged, and attributed to a specific team or project, eliminating blind spots in your AI spend.\n\n**What is an AI gateway and why is it essential for token budgeting?**\n\nAn AI gateway is an architectural layer that sits between your applications and the AI models they interact with. It is essential for token budgeting because it acts as an active enforcement point. Unlike a dashboard that only reports on costs after they have occurred, an AI gateway intercepts the request in real-time, checking the team's token budget and blocking the request if the funds are exhausted.\n\nAn AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That \"what it knows\" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,\n\nHugo Guerrero\n\n# Your Multi-Agent System Is Only as Reliable as Its Context Layer\n\nMulti-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s\n\nHugo Guerrero\n\n# The Architecture Decision Your Multi-Agent System Will Live With\n\nMulti-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:\n\nHugo Guerrero\n\n# From Microservices to AI Traffic — Kong as the Unified Control Plane\n\nThe Anatomy of Architectural Complexity\nModern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity.\nNorth-South API Traf\n\nKong\n\n# Managing the Chaos: How AI Gateways Enable Scalable AI Connectivity\n\nExecutive Summary\nAI adoption has moved past the \"honeymoon phase\" and into the \"operational chaos\" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and \"Shadow AI\" usage, the need for a centralized control plane has be\n\nKong\n\n# A New Dawn: Enterprise AI's Shadow — Trillions of Tokens, Zero Governance\n\nYou Can't Govern What You Can't See\nAI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from\n\nAugusto Marietti\n\n# Stop Patching. Start Building: The Kong Context Mesh Stack\n\nYour infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho\n\nHugo Guerrero\n\n# How AI Agents Communicate: Managing Context in Multi-Agent Workflows\n\nAn AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That \"what it knows\" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,\n\nHugo Guerrero\n\n# Your Multi-Agent System Is Only as Reliable as Its Context Layer\n\nMulti-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s\n\nHugo Guerrero\n\n# The Architecture Decision Your Multi-Agent System Will Live With\n\nMulti-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:\n\nHugo Guerrero\n\n# From Microservices to AI Traffic — Kong as the Unified Control Plane\n\nThe Anatomy of Architectural Complexity\nModern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity.\nNorth-South API Traf\n\nKong\n\n# Managing the Chaos: How AI Gateways Enable Scalable AI Connectivity\n\nExecutive Summary\nAI adoption has moved past the \"honeymoon phase\" and into the \"operational chaos\" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and \"Shadow AI\" usage, the need for a centralized control plane has be\n\nKong\n\n# A New Dawn: Enterprise AI's Shadow — Trillions of Tokens, Zero Governance\n\nYou Can't Govern What You Can't See\nAI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from\n\nAugusto Marietti\n\n# Stop Patching. Start Building: The Kong Context Mesh Stack\n\nYour infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho\n\nHugo Guerrero\n\n# How AI Agents Communicate: Managing Context in Multi-Agent Workflows\n\nAn AI agent isn't magic. It's a reasoning engine that makes decisions based on what it knows at a given moment. That \"what it knows\" is its context — the information available to it right now. Context is everything. Give an agent the wrong context,\n\nHugo Guerrero\n\n# Your Multi-Agent System Is Only as Reliable as Its Context Layer\n\nMulti-agent workflows live and die on context. Every agent-to-agent call and every agent-to-tool call is either a retrieval — fetching information the agent needs — or a mutation — changing state that downstream agents will depend on. At prototype s\n\nHugo Guerrero\n\n# The Architecture Decision Your Multi-Agent System Will Live With\n\nMulti-agent systems are, at their core, context distribution systems. Every agent in your workflow is a consumer and producer of context. The interesting architectural questions are all about how that context moves. Two operations drive everything:\n\nHugo Guerrero\n\n# From Microservices to AI Traffic — Kong as the Unified Control Plane\n\nThe Anatomy of Architectural Complexity\nModern architectures now juggle three distinct traffic patterns. Each brings unique demands. Traditional approaches treat them separately. This separation creates unnecessary complexity.\nNorth-South API Traf\n\nKong\n\n# Managing the Chaos: How AI Gateways Enable Scalable AI Connectivity\n\nExecutive Summary\nAI adoption has moved past the \"honeymoon phase\" and into the \"operational chaos\" phase. As enterprises juggle multiple LLM providers, skyrocketing token costs, and \"Shadow AI\" usage, the need for a centralized control plane has be\n\nKong\n\n# A New Dawn: Enterprise AI's Shadow — Trillions of Tokens, Zero Governance\n\nYou Can't Govern What You Can't See\nAI spending will reach $2.59 trillion in 2026. I regularly like to share what we're seeing in production at Kong. Not projections or analyst forecasts, but actual traffic flowing through Kong AI Gateway from\n\nAugusto Marietti\n\n# Stop Patching. Start Building: The Kong Context Mesh Stack\n\nYour infrastructure already has the raw materials: compute (VMs, containers, serverless), event streaming (Kafka, Kinesis, Pub/Sub, RabbitMQ), data stores (warehouses, databases, object storage), and AI endpoints (any hosted or self-hosted LLM). Tho\n\nHugo Guerrero\n\n## Ready to see Kong in action?\n\nGet a personalized walkthrough of Kong's platform tailored to your architecture, use cases, and scale requirements.", "url": "https://wpnews.pro/news/ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it", "canonical_source": "https://konghq.com/blog/enterprise/ai-token-cost-management", "published_at": "2026-09-24 16:05:00+00:00", "updated_at": "2026-09-24 16:31:12.538389+00:00", "lang": "en", "topics": ["ai-policy", "ai-infrastructure", "mlops", "large-language-models", "ai-tools"], "entities": ["Kong", "Uber", "Anthropic", "Claude Code", "Cursor", "AI Governance Gap Report"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it", "markdown": "https://wpnews.pro/news/ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it.md", "text": "https://wpnews.pro/news/ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it.txt", "jsonld": "https://wpnews.pro/news/ai-token-cost-management-why-ai-spend-gets-out-of-control-and-how-to-fix-it.jsonld"}}