{"slug": "what-i-got-wrong-estimating-llm-costs-before-the-request", "title": "What I got wrong estimating LLM costs before the request", "summary": "A developer built a cross-provider LLM gateway that refuses requests before they are sent when today's spend plus in-flight reservations plus an estimate would exceed a user-set daily cap, returning HTTP 402 instead of letting the call through. Testing against OpenAI, Anthropic and Gemini APIs with a $0.001 daily cap, the developer found six estimation pitfalls, including that max_tokens acts as a budget reservation (a 10,000-token request on gpt-6-luna reserved $0.005, five times the cap), that reasoning models can burn the entire output limit on thinking and return an empty reply, and that chat-format overhead caused input estimates to miss consistently (1 token estimated versus 7 billed). The gateway now rejects unpriced models with a 400 UNKNOWN_MODEL and reserves images at each model's worst case rather than a flat 4,000 tokens.", "body_md": "Ever since AI has taken over the internet, I've been wondering how we can optimize the full ability of AI in our life. Therefore I began building a small gateway that sits between your code and the OpenAI, Anthropic and Gemini APIs. It has one job: if a request would push today's spend past a daily cap you set, refuse it before it goes out.\n\nWhy bother? Because the stories are always the same. An agent gets stuck in a loop, keeps retrying, every retry sends a longer conversation, and nobody notices until the bill shows up. Logs and alerts tell you what happened. I wanted something that says no before it happens.\n\n(To be fair to OpenAI: they now have [hard spend limits](https://developers.openai.com/api/docs/guides/spend-limits), and you should turn them on. But they're monthly, they're OpenAI only, and their own docs say enforcement isn't instant, so spend can go slightly over. I wanted a daily cap that works across providers.)\n\nThe idea is simple: today's spend + whatever is still running + an estimate for this call. Over the cap? Refuse.\n\nThe estimate is where I kept getting it wrong. Here are six places, with real numbers from my own test account.\n\nBefore a request goes out you know the prompt. You don't know the answer. So the only honest number is the ceiling: `max_tokens` × the output price. No `max_tokens`? I assume 4,096. Above 10,000? I clamp it down, so one request can't reserve something silly.\n\nTo test it I set my own daily cap to $0.001 (yes, a tenth of a cent) and sent `gpt-6-luna` with `max_tokens: 10000`. The output reserve alone is 10,000 × $0.50 per million = $0.005, five times the cap. The gateway said no with a 402. OpenAI never saw the request. Cost: $0.\n\nWhat surprised me: once a cap sits in front, `max_tokens` stops being a length setting and becomes a budget setting. Plenty of clients send a huge number by default. Under a small cap, they get refused.\n\nThis one confused me for a minute. To keep a test cheap I sent `gpt-6-luna` a short prompt with `max_tokens: 20`. I got a 200 back... and an empty answer. `finish_reason: \"length\"`. Usage said 7 in, 20 out.\n\nAll 20 output tokens went to reasoning. Nothing was left for the reply.\n\nOn OpenAI's reasoning models the output limit covers the thinking plus the visible text. For the budget math that's fine, since the reserve is still the ceiling. For whoever gets the empty reply, not so fine. If you use reasoning models, leave real headroom in `max_tokens`, or you'll pay for a call that returns nothing.\n\nI count input with the o200k tokenizer over the message text. Same short test: I estimated 1 input token. OpenAI billed 7.\n\nThe gap is the formatting the chat format wraps around each message, which I wasn't counting.\n\nIn money it's nothing ($0.0000101 estimated, $0.0000107 actual). But it always misses in the same direction, and it grows with every message. Agent conversations have a lot of messages. I haven't fixed it yet. It's next on my list, and I'd rather say that than pretend it's done.\n\nMy first version had what I thought was a clever fallback: unknown model? Price it like the most expensive one I know and let it through. Safe, right?\n\nNot really.\n\n`gpt-4o-2024-05-13` was $5 / $15 per million tokens; `gpt-4o` is $2.50 / $10.`o4-mini`, `o3-mini` and `gpt-4.1-nano` on October 23, 2026, and `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `o3` and `o3-pro` go on December 11 (\nSo now it's strict. If the model isn't in my price table, the request gets a 400 `UNKNOWN_MODEL` before it's sent, with a note to use the name without a date. My rule now: if I can't price it, I can't cap it. The downside is a chore: every time I add a model, I check the provider's deprecation page first.\n\nVersion one: if a request had any image in it, I counted the whole thing as 4,000 input tokens, text included. So a big image was under-counted and the text was ignored completely. Not great.\n\nNow I count the text normally and reserve each image at that model's worst case, from the provider's own rules. That's up to 36,000 tokens per image on some newer OpenAI models, and 4,784 on newer Claude models. One image on `gpt-6-luna` reserves about $0.0036.\n\nSafe, but the real charge is usually much lower. With a small cap, image-heavy traffic runs out of room because of reservations, not because of real spend. The reserve is swapped for real usage once the call finishes, but until then it takes up space.\n\nLong agent runs re-read the same context again and again, and providers charge much less for cached input. My estimate charges all input at full price.\n\nSomeone on the OpenAI forum posted a 13-hour, two-agent run where 95.4% of the tokens were cached re-reads ([thread](https://community.openai.com/t/two-agents-one-night-158-million-tokens-what-it-cost-and-what-we-are-fixing/1398068)). On a run like that, my check would hit the cap long before the real bill does. A reply in that thread put it better than I can: price everything at full rate and you stop too early; assume the cache and you get surprised when it misses.\n\nI picked \"stop too early\". The better fix is probably to look at each session's recent cache-hit rate and estimate from that. I haven't built it.\n\nEverything above leans toward refusing too early. A refused call costs nothing; a missed one costs money. That's basically the whole philosophy.\n\nIt's still an estimate, not a billing guarantee. Input can come in low (see #3), and the cap only sees traffic that actually goes through it.\n\nIf I had to give one piece of advice: decide what your gate does when it can't price a call, before anything else.\n\nI'm curious how others handle this. If you've solved the per-message overhead or cache-aware estimates, how did you do it?", "url": "https://wpnews.pro/news/what-i-got-wrong-estimating-llm-costs-before-the-request", "canonical_source": "https://dev.to/elson_pang_69c87e7fb31893/what-i-got-wrong-estimating-llm-costs-before-the-request-1fla", "published_at": "2026-11-06 16:12:22+00:00", "updated_at": "2026-11-06 16:24:34.920700+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["OpenAI", "Anthropic", "Google Gemini", "gpt-6-luna", "gpt-4o", "o3-mini"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-i-got-wrong-estimating-llm-costs-before-the-request", "markdown": "https://wpnews.pro/news/what-i-got-wrong-estimating-llm-costs-before-the-request.md", "text": "https://wpnews.pro/news/what-i-got-wrong-estimating-llm-costs-before-the-request.txt", "jsonld": "https://wpnews.pro/news/what-i-got-wrong-estimating-llm-costs-before-the-request.jsonld"}}