Most teams find out their LLM costs are a problem the same way: the invoice shows up, it's bigger than expected, and nobody can say exactly why. Not because the spend is random. Because nobody is watching the layer where the spend actually happens.
Here's the thing that trips people up. LLM cost tracking usually lives inside each application: one service, one API key, one line item. That works fine when you have one app calling one model. It falls apart the moment you have multiple apps, multiple teams, and multiple models sharing the same provider accounts. You get a total number and no way to decompose it. You can't tell which feature is expensive, which team owns the spike, or which model tier is overkill for the job.
The fix isn't better logging inside each app. It's moving cost control to the layer that sits between all your apps and the model providers, an AI gateway. That's infrastructure, not application code, and it's the only place that sees every request across the whole stack.
Route by complexity, not by default. If every request goes to the same frontier model regardless of whether it's a one-line FAQ answer or a multi-step reasoning task, you're paying frontier prices for commodity work. A routing layer evaluates each request and sends the easy stuff to a cheaper model. Something like this, conceptually:
routing_rules:
- match: token_estimate < 500 and task_type == "classification"
route_to: small-model-tier
- match: task_type in ["code_gen", "multi_step_reasoning"]
route_to: frontier-model-tier
- default: mid-tier
No SDK changes on the application side. Your app still sends a normal request, the gateway decides where it goes.
Cache semantically, not just literally. Exact-match caching only catches identical strings. Semantic caching uses embedding similarity, so "what's your refund policy" and "how do I return something" hit the same cached answer instead of two separate model calls. In high-traffic apps, a meaningful chunk of traffic is redundant in exactly this way, and every cache hit is a model call you didn't pay for.
Cap spend in tokens, not requests. A rate limit measured in request count misses the real cost driver. One short prompt and one massive context-stuffed prompt count the same under request limits but can differ by two orders of magnitude in actual cost. Token budgets, set per user, per app, or per agent session, are the circuit breaker that stops a broken retry loop or a misbehaving agent before it turns into a five-figure surprise.
Fail over to something, not to a retry storm. When a provider degrades or rate-limits you, application-level retry logic tends to just hammer the same endpoint again at full price. A gateway-level fallback chain routes to an alternate provider, or a cheaper model, automatically. You define the chain once and stop babysitting incidents.
Attribute everything, or you're guessing. This is the one that makes the other four actionable. Tag every request with its source (API key, team, service identity) and you can finally answer "who's spending what and why." Without that, cost conversations with engineering leads are vibes, not evidence. This is really just observability applied to spend, and it's what turns "our AI bill is high" into "team X's retrieval step is calling the model three times per request when it needs one."
Agentic workflows add another layer of cost surface. An agent doesn't make one model call, it chains tool calls, retrieves context, and often invokes the model multiple times to complete a single task. If nothing bounds how many tool calls an agent can make per session, a support-ticket agent that should cost a few cents can quietly rack up dozens of calls before it returns an answer.
This is where cost control and security start to overlap. Uncapped agent tool-calling is both a budget problem and an attack surface, since the same lack of limits that lets a bug run wild also lets a malicious prompt trigger expensive chains on purpose. Gateways that operate at the Model Context Protocol layer can bound tool calls, data source access, and context size per session, which closes both gaps with the same control.
Inference cost isn't shrinking as fast as usage is growing. Per-token prices have fallen sharply over the past few years according to a16z's analysis of inference pricing trends, but total spend keeps climbing anyway because the volume of calls is growing faster than the price per call is dropping. That means the lever isn't waiting for models to get cheaper. It's controlling how many expensive calls you're making in the first place.
None of the five mechanisms above require rewriting your application. They require a layer in front of it that can see, route, cache, and cap every request before it reaches a model provider. If you're still debugging your AI bill from provider dashboards and app-level logs, that's the gap.
Disclosure: I work with NeuralTrust, which builds TrustGate, an open source AI gateway that implements routing, semantic caching, token budgets, fallback chains, and cost attribution at the infrastructure layer.