{"slug": "five-ways-your-llm-cost-tracking-is-lying-to-you", "title": "Five ways your LLM cost tracking is lying to you", "summary": "A developer building an LLM observability service discovered five silent failures in cost tracking for OpenAI and Anthropic APIs. Key issues include missing usage data for streaming requests, incorrect handling of prompt caching discounts, and provider-specific billing quirks. The developer provides fixes such as injecting stream_options and normalizing cache tokens into three buckets.", "body_md": "Your monthly OpenAI or Anthropic invoice tells you *how much* you spent. It doesn't tell you which feature spent it, which model, or why last Tuesday cost three times as much as Monday. So at some point you (or your team) will build a metering layer: wrap the client, read `usage`\n\noff the response, multiply by a price table, ship it to a database.\n\nI did exactly that over the past few months while building an LLM observability service, and my numbers were wrong in five different ways before they were right. Every one of these failures was silent. No exception, no alert, just numbers that were quietly too low or too high. This is the list I wish someone had handed me.\n\nOpenAI's Chat Completions API returns **no usage data at all** for streaming requests unless you pass `stream_options: { include_usage: true }`\n\n. No error, no warning. The stream just never contains token counts.\n\nIf your metering reads `usage`\n\noff the chunks, every streaming call gets recorded as 0 tokens, $0. And since chat UIs are almost always streaming, that's most of your traffic.\n\nThis one bit me twice in the same audit. First finding: all streaming calls in my own dashboard were $0. Second, nastier finding: I had a budget-gate feature that blocks calls once spend crosses a limit, and it waved every streaming call straight through — because as far as it could tell, streaming was free.\n\nThe fix is to inject the option in your wrapper when the caller didn't set it:\n\n``` js\nlet injected = false;\nif (params.stream && params.stream_options?.include_usage === undefined) {\n  params = {\n    ...params,\n    stream_options: { ...params.stream_options, include_usage: true },\n  };\n  injected = true;\n}\n```\n\nBut there's a trap inside the fix. With `include_usage`\n\non, OpenAI appends one extra chunk at the end of the stream that carries `usage`\n\nand has an **empty choices array**. Any downstream code that does\n\n`chunk.choices[0].delta`\n\n— which is most example code on the internet — will throw on it. So if \n\n``` js\nfor await (const chunk of upstream) {\n  if (chunk.usage) observedUsage = chunk.usage;\n  // Usage-only chunk we asked for: record it, don't leak it.\n  if (injected && chunk.usage && (chunk.choices?.length ?? 0) === 0) continue;\n  yield chunk;\n}\n```\n\nIf the caller set `include_usage`\n\nthemselves, pass the chunk through — they asked for it.\n\nPrompt caching is where the naive formula `prompt_tokens × input_rate`\n\nfalls apart, and it fails in *opposite directions* depending on the provider.\n\n**OpenAI**: `prompt_tokens`\n\n**includes** cached reads, and the cached portion is billed at a discount (half the input rate, as of this writing). The cached count is tucked away in `usage.prompt_tokens_details.cached_tokens`\n\n. If you ignore it, you **over-count**, and the more effective your prompt caching is, the more wrong your numbers get. Which is perverse: the caching you added to save money makes your dashboard say you're spending more.\n\n**Anthropic**: the opposite layout. `input_tokens`\n\n**excludes** cache activity; cache reads and writes arrive in separate fields (`cache_read_input_tokens`\n\n, `cache_creation_input_tokens`\n\n). Reads are heavily discounted, but writes cost *more* than the base input rate (1.25× for the default 5-minute TTL, at current rates). Ignore those fields and you **under-count**.\n\nGemini has its own context-caching discount on top of that. There is no shortcut here: you have to normalize per provider.\n\nWhat worked for me was normalizing everything into three buckets — uncached input, cached reads, cached writes — and pricing each bucket with a per-provider multiplier:\n\n``` js\nconst uncached = Math.max(0, promptTokens - cachedRead - cachedWrite);\nconst inputCost =\n  uncached   * rate.input +\n  cachedRead * rate.input * mult.read +   // e.g. 0.5 (OpenAI), 0.1 (Anthropic)\n  cachedWrite * rate.input * mult.write;  // e.g. 1.25 (Anthropic 5-min cache write)\nconst cost = inputCost + completionTokens * rate.output;\n```\n\nThe multipliers change (Gemini cut its cache-read rate substantially in 2026), so keep them in one table you can update, not scattered through the code.\n\nCloudflare Workers, AWS Lambda, and edge runtimes kill outstanding work the moment your handler returns. If your metering sends records fire-and-forget, or buffers them for a batched flush that fires on a size threshold or an idle timer, a short-lived handler exits before either trigger fires. The records evaporate. No error anywhere, because the process that would have logged the error is already gone.\n\nOf everything in this post, this was the failure that scared me most. Not because the data loss was large, but because the metering itself failed and *nothing told me*. A monitoring system that silently stops monitoring is worse than no monitoring — you still trust it.\n\nTwo fixes, use either:\n\n```\nexport default {\n  async fetch(req: Request, env: Env, ctx: ExecutionContext) {\n    const client = wrap(new OpenAI({ apiKey: env.OPENAI_API_KEY }));\n    try {\n      return await handleRequest(req, client);\n    } finally {\n      // Keeps the runtime alive past the response without delaying it:\n      ctx.waitUntil(flushClient(client));\n      // ...or, if you don't have ctx: await flushClient(client);\n    }\n  },\n};\n```\n\nThe point is that your metering library must *expose* an awaitable flush. If you're evaluating an SDK for serverless use and it doesn't document one, assume it loses data there.\n\nUsers close tabs mid-stream. Code `break`\n\ns out of the loop after finding what it needed. The provider bills you for every token generated up to the disconnect. But if your recording logic sits *after* the consumption loop, it never runs. In JavaScript, breaking out of a `for await`\n\ncalls `.return()`\n\non the generator, and everything after the loop body is skipped.\n\nSo: real money spent, zero recorded. Same silent-under-count family as pitfall 1.\n\nThe pattern that covers all three exits (completion, error, early break) is a `finally`\n\nwith an idempotent record call:\n\n``` js\nlet recorded = false;\nconst recordOnce = () => {\n  if (recorded) return;\n  recorded = true;\n  save(observedSoFar); // partial usage is better than no record\n};\n\ntry {\n  for await (const chunk of upstream) {\n    observe(chunk);\n    yield chunk;\n  }\n  recordOnce();\n} catch (err) {\n  recorded = true;\n  saveErrorRecord(err, observedSoFar);\n  throw err;\n} finally {\n  recordOnce(); // early break / abandoned iterator lands here\n}\n```\n\nOne more wrinkle a code review caught in mine: `openai-node`\n\n's `Stream`\n\ntreats `controller.abort()`\n\nas a *normal* end of iteration. The loop exits cleanly, so without an extra check you'd record a truncated response as a successful, complete one. Check `stream.controller.signal.aborted`\n\nbefore marking the record as completed.\n\nIf you can afford it, there's a stronger version: `tee()`\n\nthe stream and drain an observation branch independently of the user-facing branch. Then even an early break on the user side leaves the observation side to run to completion and capture the final usage chunk. Costs some buffering memory; worth it for accuracy.\n\nHardcoded per-model rates go stale every time a provider ships a model. The dangerous failure mode isn't the staleness itself. It's what your code does when it looks up a model it doesn't know. If the answer is \"silently return $0\", then the day someone switches to a new model, your cost graph drops to zero and everyone celebrates the wrong thing.\n\nThe floor for handling this:\n\n``` js\nconst entry = lookupPricing(provider, model);\nif (!entry) {\n  warnOnce(\n    `unknown ${provider} model \"${model}\" — pricing returned 0. ` +\n    `Update the pricing table.`,\n  );\n  return 0;\n}\n```\n\nYou still record $0 (there's no honest number to put there), but you warn, loudly and once per model, so a human finds out the table needs updating. A silent zero and a zero with a warning look identical in the database. Operationally they are very different things.\n\n(Warn *once* per unknown model, not per call. The first version of my warning fired on every call and turned the logs into noise nobody read.)\n\nIf you're building or buying an LLM metering layer, walk through these:\n\n`include_usage`\n\nhandled, and is the usage-only chunk kept away from `choices[0]`\n\nconsumers?`prompt_tokens`\n\nwhile Anthropic reports cache activity in separate fields?`ctx.waitUntil`\n\nor a `finally`\n\n?None of these is hard on its own. The problem is that they all fail silently, so you tend to find them one production surprise at a time.\n\nDisclosure: I built [Argosvix](https://argosvix.com), an LLM observability service whose SDK handles all five of these. Everything above works fine self-rolled too.", "url": "https://wpnews.pro/news/five-ways-your-llm-cost-tracking-is-lying-to-you", "canonical_source": "https://dev.to/yutomakihara/five-ways-your-llm-cost-tracking-is-lying-to-you-191n", "published_at": "2026-07-13 12:58:54+00:00", "updated_at": "2026-07-13 13:18:10.355154+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure", "ai-products"], "entities": ["OpenAI", "Anthropic", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/five-ways-your-llm-cost-tracking-is-lying-to-you", "markdown": "https://wpnews.pro/news/five-ways-your-llm-cost-tracking-is-lying-to-you.md", "text": "https://wpnews.pro/news/five-ways-your-llm-cost-tracking-is-lying-to-you.txt", "jsonld": "https://wpnews.pro/news/five-ways-your-llm-cost-tracking-is-lying-to-you.jsonld"}}