Five ways your LLM cost tracking is lying to you

A developer building an LLM observability service discovered five silent failures in cost tracking for OpenAI and Anthropic APIs. Key issues include missing usage data for streaming requests, incorrect handling of prompt caching discounts, and provider-specific billing quirks. The developer provides fixes such as injecting stream_options and normalizing cache tokens into three buckets.

Your monthly OpenAI or Anthropic invoice tells you how much you spent. It doesn't tell you which feature spent it, which model, or why last Tuesday cost three times as much as Monday. So at some point you or your team will build a metering layer: wrap the client, read usage off the response, multiply by a price table, ship it to a database. I did exactly that over the past few months while building an LLM observability service, and my numbers were wrong in five different ways before they were right. Every one of these failures was silent. No exception, no alert, just numbers that were quietly too low or too high. This is the list I wish someone had handed me. OpenAI's Chat Completions API returns no usage data at all for streaming requests unless you pass stream options: { include usage: true } . No error, no warning. The stream just never contains token counts. If your metering reads usage off the chunks, every streaming call gets recorded as 0 tokens, $0. And since chat UIs are almost always streaming, that's most of your traffic. This one bit me twice in the same audit. First finding: all streaming calls in my own dashboard were $0. Second, nastier finding: I had a budget-gate feature that blocks calls once spend crosses a limit, and it waved every streaming call straight through — because as far as it could tell, streaming was free. The fix is to inject the option in your wrapper when the caller didn't set it: js let injected = false; if params.stream && params.stream options?.include usage === undefined { params = { ...params, stream options: { ...params.stream options, include usage: true }, }; injected = true; } But there's a trap inside the fix. With include usage on, OpenAI appends one extra chunk at the end of the stream that carries usage and has an empty choices array . Any downstream code that does chunk.choices 0 .delta — which is most example code on the internet — will throw on it. So if js for await const chunk of upstream { if chunk.usage observedUsage = chunk.usage; // Usage-only chunk we asked for: record it, don't leak it. if injected && chunk.usage && chunk.choices?.length ?? 0 === 0 continue; yield chunk; } If the caller set include usage themselves, pass the chunk through — they asked for it. Prompt caching is where the naive formula prompt tokens × input rate falls apart, and it fails in opposite directions depending on the provider. OpenAI : prompt tokens includes cached reads, and the cached portion is billed at a discount half the input rate, as of this writing . The cached count is tucked away in usage.prompt tokens details.cached tokens . If you ignore it, you over-count , and the more effective your prompt caching is, the more wrong your numbers get. Which is perverse: the caching you added to save money makes your dashboard say you're spending more. Anthropic : the opposite layout. input tokens excludes cache activity; cache reads and writes arrive in separate fields cache read input tokens , cache creation input tokens . Reads are heavily discounted, but writes cost more than the base input rate 1.25× for the default 5-minute TTL, at current rates . Ignore those fields and you under-count . Gemini has its own context-caching discount on top of that. There is no shortcut here: you have to normalize per provider. What worked for me was normalizing everything into three buckets — uncached input, cached reads, cached writes — and pricing each bucket with a per-provider multiplier: js const uncached = Math.max 0, promptTokens - cachedRead - cachedWrite ; const inputCost = uncached rate.input + cachedRead rate.input mult.read + // e.g. 0.5 OpenAI , 0.1 Anthropic cachedWrite rate.input mult.write; // e.g. 1.25 Anthropic 5-min cache write const cost = inputCost + completionTokens rate.output; The multipliers change Gemini cut its cache-read rate substantially in 2026 , so keep them in one table you can update, not scattered through the code. Cloudflare Workers, AWS Lambda, and edge runtimes kill outstanding work the moment your handler returns. If your metering sends records fire-and-forget, or buffers them for a batched flush that fires on a size threshold or an idle timer, a short-lived handler exits before either trigger fires. The records evaporate. No error anywhere, because the process that would have logged the error is already gone. Of everything in this post, this was the failure that scared me most. Not because the data loss was large, but because the metering itself failed and nothing told me . A monitoring system that silently stops monitoring is worse than no monitoring — you still trust it. Two fixes, use either: export default { async fetch req: Request, env: Env, ctx: ExecutionContext { const client = wrap new OpenAI { apiKey: env.OPENAI API KEY } ; try { return await handleRequest req, client ; } finally { // Keeps the runtime alive past the response without delaying it: ctx.waitUntil flushClient client ; // ...or, if you don't have ctx: await flushClient client ; } }, }; The point is that your metering library must expose an awaitable flush. If you're evaluating an SDK for serverless use and it doesn't document one, assume it loses data there. Users close tabs mid-stream. Code break s out of the loop after finding what it needed. The provider bills you for every token generated up to the disconnect. But if your recording logic sits after the consumption loop, it never runs. In JavaScript, breaking out of a for await calls .return on the generator, and everything after the loop body is skipped. So: real money spent, zero recorded. Same silent-under-count family as pitfall 1. The pattern that covers all three exits completion, error, early break is a finally with an idempotent record call: js let recorded = false; const recordOnce = = { if recorded return; recorded = true; save observedSoFar ; // partial usage is better than no record }; try { for await const chunk of upstream { observe chunk ; yield chunk; } recordOnce ; } catch err { recorded = true; saveErrorRecord err, observedSoFar ; throw err; } finally { recordOnce ; // early break / abandoned iterator lands here } One more wrinkle a code review caught in mine: openai-node 's Stream treats controller.abort as a normal end of iteration. The loop exits cleanly, so without an extra check you'd record a truncated response as a successful, complete one. Check stream.controller.signal.aborted before marking the record as completed. If you can afford it, there's a stronger version: tee the stream and drain an observation branch independently of the user-facing branch. Then even an early break on the user side leaves the observation side to run to completion and capture the final usage chunk. Costs some buffering memory; worth it for accuracy. Hardcoded per-model rates go stale every time a provider ships a model. The dangerous failure mode isn't the staleness itself. It's what your code does when it looks up a model it doesn't know. If the answer is "silently return $0", then the day someone switches to a new model, your cost graph drops to zero and everyone celebrates the wrong thing. The floor for handling this: js const entry = lookupPricing provider, model ; if entry { warnOnce unknown ${provider} model "${model}" — pricing returned 0. + Update the pricing table. , ; return 0; } You still record $0 there's no honest number to put there , but you warn, loudly and once per model, so a human finds out the table needs updating. A silent zero and a zero with a warning look identical in the database. Operationally they are very different things. Warn once per unknown model, not per call. The first version of my warning fired on every call and turned the logs into noise nobody read. If you're building or buying an LLM metering layer, walk through these: include usage handled, and is the usage-only chunk kept away from choices 0 consumers? prompt tokens while Anthropic reports cache activity in separate fields? ctx.waitUntil or a finally ?None of these is hard on its own. The problem is that they all fail silently, so you tend to find them one production surprise at a time. Disclosure: I built Argosvix https://argosvix.com , an LLM observability service whose SDK handles all five of these. Everything above works fine self-rolled too.