My first real LLM bill surprise wasn't dramatic. No infinite loop, no viral spike. A retry policy I'd written months earlier met a flaky endpoint, and a background job quietly re-sent the same long prompt all night. The bill was just... 40x normal. Nothing "failed", so nothing alerted.
I've spent the last few weeks building a hard spending cap for LLM calls, and most of what I learned wasn't about caps at all. It was about how many small boring ways money leaks when the meter only exists on the provider's side. Here's the list I wish someone had handed me.
Billing dashboards are rear-view mirrors
Every provider dashboard answers "how much did you spend?" hours later, after aggregation. Nobody answers "how much are you spending right now, from which feature?" By the time the dashboard shows the leak, the leak already happened. If you take one thing from this post: put the meter inside your process, at the call site, where it can act before the request instead of reporting after it.
Total spend is a useless number
"$120 today" tells you nothing actionable. The question that matters is which feature is leaking. The fix is one tag per call (feature: "chat", feature: "summarizer") and a per-feature breakdown. The first time I ran a breakdown on a real app, a background enrichment job I'd forgotten about turned out to be 60% of spend. Attribution isn't a nice-to-have. It's the entire diagnosis.
Every provider reports usage differently, and they all lie a little
OpenAI gives you prompt_tokens and completion_tokens. Anthropic gives input_tokens/output_tokens. Gemini nests everything in usageMetadata. Bedrock and Cohere have their own shapes. Then it gets worse: cached input tokens bill at a different rate, and reasoning tokens are billed as output even though you never see them. Two providers even disagree on whether reasoning tokens are counted inside the output number at all — xAI and Gemini report them outside it, so if you naively bill the output count you're undercounting exactly when reasoning models get expensive. Budget a week of reading API docs that contradict the actual responses.
Streaming hides the receipt until the very end
With stream: true, usage arrives (if at all) in the final chunk. OpenAI won't even send it unless you pass stream_options: { include_usage: true }. Anthropic splits it across message_start and message_delta events you have to assemble, and the delta is cumulative, so adding instead of replacing double-counts. Gemini sends a running usageMetadata where only the last value is real. Any cost tracking that ignores streaming is blind to the calls most modern apps actually make.
A cap that checks after the call is a receipt, not a guard
The naive cap goes: call, add to a counter, compare. It always overshoots by one call. Concurrency makes it much worse. Ten parallel requests all read the same counter, all pass the check, all land. I only really fixed this with an atomic reserve-then-settle: estimate the cost, reserve it against the cap before the call (one atomic operation, Lua script if the counter lives in Redis), settle the difference after. If you run multiple workers the counter has to live in shared storage anyway, otherwise each worker politely enforces its own private budget while the fleet blows through ten of them.
Retry storms are the leak nobody instruments
The single most expensive bug class I found wasn't prompts. It was retries: exponential backoff on a 429 while the request is actually succeeding on the provider's side, queue re-drives, at-least-once delivery re-running completed jobs. Track consecutive failures per feature and alert when the streak gets weird. A hard cap turns this failure mode from "unbounded bill" into "capped error you notice in the morning", and that trade is almost always correct for background workloads.
Caps and traces are two sides of the same thing
A reader of my first post put it better than I had: cost caps and execution traces are both answers to "what is my agent doing when I'm not looking?" The cap is the brake. A per-call spend event (feature, model, usd, running total) piped into the logs you already have is the gauge. You want both, because a brake without a gauge just means you get stopped without knowing why.
I ended up packaging all of this into a small MIT-licensed library: zero-dependency hard caps that hold under concurrent workers, per-feature attribution, streaming usage, and adapters for Vercel AI SDK, LangChain.js, LlamaIndex.TS and Mastra — budget-guard. But honestly, even if you never install anything: tag your calls, meter before the request, and go look at what your retries did last night.