Delivered but Unbilled: Your AI Stream Logged Zero Tokens

A developer created stream_billing_gate.py, a tool that detects 'delivered-but-unbilled' failures in AI streaming responses where text is delivered to users but token usage is logged as zero. The tool reconciles delivered text against logged usage offline and blocks when text was shipped but nothing was billed. The developer cites a Dev.to proof of concept showing this exact failure pattern.

Delivered but unbilled is the streaming failure where your AI response renders the whole answer to the user, but the terminal usage frame is dropped, zeroed, or malformed, so your client logs 0 output tokens for a call that cost money. stream billing gate.py reconciles the delivered text against the logged usage, offline, and blocks when text shipped and nothing was billed. AI disclosure:I wrote stream billing gate.py with an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network. Every number in the output blocks below is pasted from a real local run. I checked the exit codes 0 / 1 / 2 , ran each scenario twice to confirm STDOUT is byte-for-byte identical, and had the tool print a sha256 of its own report so you can reproduce the exact bytes. The one external finding I cite a Dev.to proof of concept and the one external number a Hacker News thread's score are other people's, attributed inline, and kept out of my fixture numbers. In short: delivered-but-unbilled .A streaming response arrives in two parts that travel separately. First the content deltas: dozens of small events, each carrying a slice of text, which your client concatenates and paints on the screen. Then, at the very end, one terminal frame carrying the token accounting: input tokens, output tokens, the numbers your billing code reads to know what the call cost. The order is the problem. By the time the accounting is supposed to arrive, the answer is already delivered. If that last frame is dropped by a flaky connection, truncated, or written with a zero, the user never notices. They got their answer. Your client, meanwhile, has nothing to log, so it keeps whatever value it initialized. For a lot of clients that value is zero. No exception. No crash. A response that cost real output tokens is recorded as free. That zero is a choice your client made, not a law of streaming, and how the accounting travels varies by provider, which the formats section below gets into. On July 7, a developer publishing as kielltampubolon posted a proof of concept on Dev.to titled "Asynchronous Telemetry Blindness in AI Streaming Clients" https://dev.to/kielltampubolon/asynchronous-telemetry-blindness-in-ai-streaming-clients-a-poc-where-text-renders-billing-stays-50a5 , showing this exact shape: the text deltas render the full answer while a dropped terminal usage frame leaves billing sitting at zero, with no error surfaced. That framing is his, and his write-up is a local-only proof of concept, not a production incident. I am not reusing his numbers. I built my own fixtures so the failure is reproducible on your logs, and so a test can fail closed on it. Your dashboard says $0 for that call. The dashboard is not lying. It is faithfully reporting a number that is wrong at the source. This is the whole thesis of the pre-execution and reconciliation gates I keep building https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/ : tracking a number is not the same as controlling the thing the number describes. You tracked usage. The usage you tracked was zero. The tokens were spent anyway. Here is the claim, stated so you can break it. Given a recorded stream where text deltas exist and the terminal usage frame is absent or zero, the gate must exit non-zero with reason delivered-but-unbilled . Given a stream with a usage frame whose output count meets the floor derived from the delivered text, it must exit 0. One removed line has to flip the verdict. If you can show me a stream that delivered text and the gate stayed quiet, or one that logged its usage honestly and the gate blocked, the tool is broken and this post with it. Two quantities per stream. The delivered text, rebuilt by concatenating every content delta. And the logged output tokens, pulled from the terminal usage frame, or None when that frame is missing or unusable. The token side needs an honest word about method. Without a real tokenizer, and this tool ships with none, standard library only, I cannot tell you the exact token count of the delivered text. So I do not. I compute a FLOOR: the number of whitespace-delimited words. A BPE tokenizer keyed on whitespace almost always emits at least one token per word, and usually more once punctuation and sub-word splits count. So the real output tokens is nearly always higher than this floor. It is a deliberately low, human-readable magnitude, not a measurement. The gate's real claim does not lean on that floor being precise. It leans on something smaller that cannot be argued with: if the delivered text is non-empty, the real output token count is above zero. Therefore a logged zero is provably wrong, whatever the exact number was. The floor just gives you a readable "at least this many" figure to put in the alert. That gives four verdicts: | Verdict | Condition | reason-code | exit | |---|---|---|---| | OK | usage frame present, logged = floor | usage-consistent | 0 | | UNDERCOUNT | usage frame present, 0 < logged < floor | partial-telemetry-loss | 2 | | BLIND | text delivered, logged is 0 / absent / unparseable | delivered-but-unbilled | 1 | | EMPTY | no text delivered, nothing to reconcile | nothing-delivered | 0 | BLIND is the hard, logical one and gets the blocking exit code. UNDERCOUNT is softer: it says the logged count is below the floor, which is suspicious but heuristic, so it warns rather than blocks. The decision itself is a few lines, simplified for the post the real function returns dicts with detail strings, and it carries one extra guard shown here that fails closed on a stream in a shape it cannot recognize : python def classify delivered text, logged, usage seen, events, recognized : simplified for the post floor = word floor delivered text whitespace word count if events and not recognized: valid JSON, but no shape the gate knows - fail closed return "UNRECOGNIZED", "unrecognized-stream", 2 if not delivered text.strip : return "EMPTY", "nothing-delivered", 0 if logged is None or logged == 0: return "BLIND", "delivered-but-unbilled", 1 if logged < floor: return "UNDERCOUNT", "partial-telemetry-loss", 2 return "OK", "usage-consistent", 0 The gate reads the recorded stream as data and auto-detects the wire shape, so you can point it at the logs you already keep. It maps four documented formats to the same two quantities, delivered text and logged output tokens: | Format | Delivered text | Logged output tokens | |---|---|---| | OpenAI Chat Completions | choices .delta.content | usage.completion tokens in the final chunk, sent only when you set stream options={"include usage": true} | | Anthropic Messages | content block delta.delta.text | message delta.usage.output tokens cumulative; the last one is the total | | OpenAI Responses | response.output text.delta | response.completed then response.usage.output tokens | | Generic normalized | {"type":"content.delta","text":...} | {"type":"usage","output tokens":...} | How the accounting travels is not the same across providers, and that changes what a zero means. OpenAI Chat sends usage in one terminal chunk, and only if you asked for it: leave include usage off and there is no usage frame at all, which reads as delivered-but-unbilled by default. Anthropic sends output tokens cumulatively across message delta frames, so a dropped final frame undercounts rather than zeroes, unless your client commits only the terminal number. A logged 0 is a fact about what your client recorded, not a universal property of streaming. The gate does not care which of these happened. It reconciles the text that reached the user against the number that reached your logs. The FLOOR is provider-agnostic. It comes from the concatenated delivered text alone, so the same word count governs an OpenAI stream and an Anthropic one. A format the gate does not recognize is reported as unrecognized-stream exit 2 , which fails closed instead of passing as EMPTY, so a broken adapter cannot hide a leak behind a green check. Feed it a recorded stream as JSONL, one event per line, in any of the formats above. Then run it: python3 stream billing gate.py fixtures/fixture blind.jsonl No install, no key, no network. It reads the file, prints a verdict, and returns an exit code you can wire into CI over your recorded stream logs, whether they are OpenAI Chat, Anthropic, Responses, or the normalized shape. A log in a format it does not recognize returns unrecognized-stream exit 2 rather than a silent pass, so confirm the fit on a known-good stream before you trust a green. Here are two files. fixture ok.jsonl is a recorded stream: a normal answer about capping agent spend, followed by a terminal usage frame reporting 190 output tokens. fixture blind.jsonl is the same stream with that one usage line removed. The text deltas are byte-identical. Run each: stream-billing-gate: delivered text vs logged usage files: 1 1 fixture ok.jsonl delivered : 137 words, 750 chars floor 137 tokens logged : output tokens=190 verdict : OK usage-consistent exit 0 detail : logged 190 = floor 137 summary: 1 OK, 0 UNDERCOUNT, 0 BLIND, 0 EMPTY - overall exit 0 report-sha256: f6b8ce0b2b7a09f57a589a862b53024a81ca618cfecd34f82fa6231c670486a7 stream-billing-gate: delivered text vs logged usage files: 1 1 fixture blind.jsonl delivered : 137 words, 750 chars floor 137 tokens logged : none verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 137 tokens of text; no usage frame at all summary: 0 OK, 0 UNDERCOUNT, 1 BLIND, 0 EMPTY - overall exit 1 report-sha256: 318dff2585a7874bd04cd06a1430a9d3b303d93535910bd46be1d5987f81a4ce Same 137 words delivered. In one file that costs at least 137 tokens and the log says 190. In the other it costs at least 137 tokens and the log says nothing. The exit code goes from 0 to 1. The only difference between the two files is a single line: bash $ diff fixtures/fixture ok.jsonl fixtures/fixture blind.jsonl 10d9 < {"type":"usage","input tokens":523,"output tokens":190} That is the failure in one line. In production the line does not get deleted by hand. A connection blips, a proxy truncates, a client swallows the final frame, and you are left with fixture blind.jsonl and a dashboard that says the call was free. The two sha256 digests are the tool hashing its own report, so you can confirm you got the same bytes I did. Hand it a batch of recorded streams and it reconciles each one. This run stream billing gate.py fixtures/ .jsonl , so the files come in alphabetical order covers all four verdicts plus the two broken shapes of the usage frame, present-but-zero and present-but-unparseable, both of which are still delivered-but-unbilled: stream-billing-gate: delivered text vs logged usage files: 7 1 fixture blind.jsonl delivered : 137 words, 750 chars floor 137 tokens logged : none verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 137 tokens of text; no usage frame at all 2 fixture broken usage.jsonl delivered : 46 words, 274 chars floor 46 tokens logged : none verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 46 tokens of text; usage frame present but unparseable 3 fixture empty.jsonl delivered : 0 words, 0 chars floor 0 tokens logged : none verdict : EMPTY nothing-delivered exit 0 detail : no text delivered; nothing to reconcile 4 fixture injection.jsonl delivered : 60 words, 344 chars floor 60 tokens logged : none verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 60 tokens of text; no usage frame at all 5 fixture ok.jsonl delivered : 137 words, 750 chars floor 137 tokens logged : output tokens=190 verdict : OK usage-consistent exit 0 detail : logged 190 = floor 137 6 fixture undercount.jsonl delivered : 72 words, 403 chars floor 72 tokens logged : output tokens=12 verdict : UNDERCOUNT partial-telemetry-loss exit 2 detail : logged 12 < floor 72; telemetry lost part of the stream 7 fixture zero usage.jsonl delivered : 55 words, 272 chars floor 55 tokens logged : output tokens=0 verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 55 tokens of text; usage frame logged output tokens=0 summary: 1 OK, 1 UNDERCOUNT, 4 BLIND, 1 EMPTY - overall exit 1 report-sha256: dc204b014afadff09b09b82a791dd86ba7767df01454fbc2cc011321c9cd5111 Look at fixture zero usage and fixture broken usage . A usage frame that says output tokens: 0 and a usage frame with a broken value both land as BLIND, because in both cases text was delivered and no honest token count was logged. A zero is not an OK. Absence is not an OK. Only a real count that meets the floor is an OK. The report ends with a sha256 of its own body, and because that body depends on the order you pass the files, fixtures/ .jsonl reproduces this exact digest. Those seven fixtures use the normalized shape, which is fine for showing the logic but proves nothing about the streams you actually record. So fixtures/providers/ holds streams written in the documented wire formats of three providers, each representative of what their SSE emits: an Anthropic Messages stream, an OpenAI Chat Completions stream, and an OpenAI Responses stream. For each provider there is a delivered-but-unbilled variant and an honest one, plus a foreign log that matches no known shape: stream-billing-gate: delivered text vs logged usage files: 7 1 anthropic blind.jsonl delivered : 54 words, 291 chars floor 54 tokens logged : output tokens=0 verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 54 tokens of text; usage frame logged output tokens=0 2 anthropic ok.jsonl delivered : 54 words, 291 chars floor 54 tokens logged : output tokens=63 verdict : OK usage-consistent exit 0 detail : logged 63 = floor 54 3 openai chat blind.jsonl delivered : 34 words, 178 chars floor 34 tokens logged : none verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 34 tokens of text; no usage frame at all 4 openai chat ok.jsonl delivered : 34 words, 178 chars floor 34 tokens logged : output tokens=64 verdict : OK usage-consistent exit 0 detail : logged 64 = floor 34 5 responses blind.jsonl delivered : 32 words, 172 chars floor 32 tokens logged : output tokens=0 verdict : BLIND delivered-but-unbilled exit 1 detail : delivered = 32 tokens of text; usage frame logged output tokens=0 6 responses ok.jsonl delivered : 32 words, 172 chars floor 32 tokens logged : output tokens=54 verdict : OK usage-consistent exit 0 detail : logged 54 = floor 32 7 unknown schema.jsonl delivered : 0 words, 0 chars floor 0 tokens logged : none verdict : UNRECOGNIZED unrecognized-stream exit 2 detail : 4 event s parsed, none matched a known stream shape; map your adapter's text and usage fields before trusting a pass summary: 3 OK, 0 UNDERCOUNT, 3 BLIND, 0 EMPTY, 1 UNRECOGNIZED - overall exit 1 report-sha256: ec93efeb393fc0606656eb9f5447ca1dbd9837ee5a51258dc0689b3d7a022aa5 Three providers, three delivered-but-unbilled streams, three blocks. The Anthropic case logs output tokens: 0 in its terminal message delta ; the OpenAI Chat case ran without include usage so no usage frame lands at all; the Responses case reports a zero in response.completed . Each honest twin logs a real count above the floor and ships. And unknown schema.jsonl is the case the earlier version of this tool got wrong: a valid-JSON log in a shape the gate cannot map. It used to read as EMPTY and exit 0, which is exactly the silent pass this whole post is against. Now it fails closed as unrecognized-stream , exit 2, so a broken adapter shows up loud instead of hiding the leak. The fixture injection case is a small joke that matters. Its delivered text contains a line addressed to any monitoring tool reading the log: it claims the response was served for free and instructs the reader to report usage-consistent and exit 0. The gate reads that line the same way it reads every other line, as text to be counted. It does not execute it. So the injected instruction just adds to the delivered token count, and the stream still gets blocked, exit 1. This is the security posture written down: stream content is untrusted input. A recorded stream can carry anything, including a sentence designed to talk your tooling into standing down. A gate that treats logs as data and never as commands is not optional when the logs come off the open internet. I would rather draw the borders myself than have you find them. It does not compute dollars. The floor is not the vendor's invoice. The gate catches delivered-but-zero-logged, the failure where accounting collapses to nothing while text shipped. It does not tell you the exact amount you were overcharged or undercharged. For that you need the price policy, and reconciling a billed amount against declared rates is a different tool that checks which model answered and whether the charge reconciles https://finops.spinov.online/blog/model-receipt-probe/ . The floor is conservative, and UNDERCOUNT is a heuristic. BLIND rests on a logical fact: text delivered means tokens above zero. UNDERCOUNT rests on a statistical one: a logged count below the word floor is suspicious. Pathological text could in principle sit near the boundary, which is exactly why UNDERCOUNT warns exit 2 and does not block. The floor is whitespace word count, so it barely moves for scripts that do not put spaces between words; a paragraph of Chinese or Japanese collapses toward a floor of 1, and UNDERCOUNT is effectively inert there while BLIND still holds. A non-integer count is treated the same conservative way: a float like 190.0 or a string "190" is dropped as unparseable and reads as BLIND, a rare false positive if some client logs honest floats. If you need exact counts, run the delivered text through the real tokenizer for your model. This tool is the cheap first pass that needs no tokenizer to catch the zero. It cannot tell you why the frame is missing. A dropped connection and a vendor that ate the usage frame produce the same evidence: delivered text, no matching usage. The gate reports the fact, not the cause. It also fails closed on bad input: no arguments, a missing file, or unparseable lines return a non-zero exit rather than a silent pass, and a stream in a shape it does not recognize returns unrecognized-stream exit 2 , not a silent EMPTY. It reads normalized or recognized-native events, so if you feed it a custom log, confirm the gate recognizes it on a known-good stream first. It is a different question from its neighbors. It is not the tokens an agent keeps burning after a run has already failed https://finops.spinov.online/blog/waste-probe-tokens-after-failure/ , and it is not the growing transcript you get rebilled for on every turn of a conversation https://finops.spinov.online/blog/context-tax-measure-transcript-rebill/ . Those measure real spend that happened. This one measures spend that happened and then vanished from the record. If you are mapping the whole bill, it sits next to the token tax an MCP server quietly adds to every call https://finops.spinov.online/blog/mcp-server-token-tax/ and a forecaster for what a loop will cost before you run it https://finops.spinov.online/blog/loop-cost-forecaster/ . The macro mood is why this small failure is worth a gate. During the week of July 7, the loudest thread I saw on Hacker News was "GLM 5.2 and the coming AI margin collapse," sitting around 680 points and 465 comments when I checked. Those are the thread's numbers, not mine, and I link nothing I cannot verify. The argument in the room is that the economics of serving these models are tightening. When margins tighten, the difference between what you actually spent and what your logs think you spent stops being a rounding error and starts being the thing that decides whether your unit economics are real. A usage frame that goes missing on a streamed response is a silent leak on the wrong side of that equation. You cannot fix a cost you never recorded. The point of reconciling the delivered text against the logged usage is not to compute your bill to the cent. It is to stop pretending a call was free when the answer is sitting right there on the screen. I publish one of these small offline gates most weeks, each one a runnable tool with the raw output and the exit codes, no keys and no network. Follow along if you want the next one. And the real question I still cannot answer cleanly, so I am asking you: on your own logs, how do you tell a genuinely dropped usage frame apart from a response that legitimately produced no billable output? I read every comment.