I run a multi-provider LLM gateway in production (OpenAI, Anthropic, Google, DeepSeek and a dozen others behind one endpoint) with prepaid, per-token billing. Getting the metering correct took more iterations than the entire proxy itself. Here is what I wish someone had told me.
One request is never one price
A single chat request produces up to six differently priced components: text input, cached input, cache writes, output, reasoning tokens, and tool units (web search calls, image generations). We ended up writing one charge row per component, each with its own rate, summed per request trace. Every attempt to shortcut this with a single "tokens x price" row broke on the next provider quirk.
Streaming usage is scattered, not at the end
If you only read usage from the final SSE chunk, you will undercount. OpenAI sends usage in the last chunk when you ask for it via stream_options. Anthropic reports input and cache tokens in message_start and the rest in message_delta. The robust pattern: accumulate every usage payload seen anywhere in the stream, keep cached and uncached input separate end to end, and only price the sum at the very end. Client aborts silently eat your money
When a user hits stop (or closes the tab) and your proxy stops reading the upstream response, the provider still bills you, but you never see the final usage frame, so you record zero. We keep draining the provider stream after the client disconnects, purely to capture usage. Test "user pressed stop" and "tab closed" separately; they behaved differently for us.
Check the balance before the request, settle after
Post-paid metering plus streaming is how you end up funding a stranger's agent loop. We do a preflight check: estimate the worst case (prompt tokens plus max_tokens times the output rate), refuse if the balance cannot cover it, then settle against real usage afterwards. Two hard-won details: cap max_tokens globally or a single request defeats any sane check, and treat agent recursion limits as a cost brake, because one rate-limited tool sent an agent into a loop that burned real money before we capped it.
Currency is its own minefield
If your customers pay in more than one currency, store the FX rate used at grant time and derive display costs from the USD source of truth. We had a bug where a display field was multiplied instead of divided by the rate; the balance math was correct the whole time, but every non-USD row displayed costs off by fx squared. Reconciliation jobs that assert granted - consumed == balance per batch caught it. All of this runs in production at kral.ai, a managed LibreChat with this billing layer built in; the longer write-up is in LibreChat for the enterprise: SSO, teams and per-token billing. Happy to answer questions about any of the failure modes in the comments.
All of this runs in production at kral.ai, a managed LibreChat with this billing layer built in; the longer write-up is in LibreChat for the enterprise: SSO, teams and per-token billing. Happy to answer questions about any of the failure modes in the comments.