{"slug": "delivered-but-unbilled-your-ai-stream-logged-zero-tokens", "title": "Delivered but Unbilled: Your AI Stream Logged Zero Tokens", "summary": "A developer created stream_billing_gate.py, a tool that detects 'delivered-but-unbilled' failures in AI streaming responses where text is delivered to users but token usage is logged as zero. The tool reconciles delivered text against logged usage offline and blocks when text was shipped but nothing was billed. The developer cites a Dev.to proof of concept showing this exact failure pattern.", "body_md": "Delivered but unbilled is the streaming failure where your AI response renders the whole answer to the user, but the terminal usage frame is dropped, zeroed, or malformed, so your client logs 0 output tokens for a call that cost money. `stream_billing_gate.py`\n\nreconciles the delivered text against the logged usage, offline, and blocks when text shipped and nothing was billed.\n\nAI disclosure:I wrote`stream_billing_gate.py`\n\nwith an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only, no network. Every number in the output blocks below is pasted from a real local run. I checked the exit codes (0 / 1 / 2), ran each scenario twice to confirm STDOUT is byte-for-byte identical, and had the tool print a sha256 of its own report so you can reproduce the exact bytes. The one external finding I cite (a Dev.to proof of concept) and the one external number (a Hacker News thread's score) are other people's, attributed inline, and kept out of my fixture numbers.\n\n**In short:**\n\n`delivered-but-unbilled`\n\n.A streaming response arrives in two parts that travel separately. First the content deltas: dozens of small events, each carrying a slice of text, which your client concatenates and paints on the screen. Then, at the very end, one terminal frame carrying the token accounting: input tokens, output tokens, the numbers your billing code reads to know what the call cost.\n\nThe order is the problem. By the time the accounting is supposed to arrive, the answer is already delivered. If that last frame is dropped by a flaky connection, truncated, or written with a zero, the user never notices. They got their answer. Your client, meanwhile, has nothing to log, so it keeps whatever value it initialized. For a lot of clients that value is zero. No exception. No crash. A response that cost real output tokens is recorded as free. That zero is a choice your client made, not a law of streaming, and how the accounting travels varies by provider, which the formats section below gets into.\n\nOn July 7, a developer publishing as kielltampubolon posted a proof of concept on Dev.to titled [\"Asynchronous Telemetry Blindness in AI Streaming Clients\"](https://dev.to/kielltampubolon/asynchronous-telemetry-blindness-in-ai-streaming-clients-a-poc-where-text-renders-billing-stays-50a5), showing this exact shape: the text deltas render the full answer while a dropped terminal usage frame leaves billing sitting at zero, with no error surfaced. That framing is his, and his write-up is a local-only proof of concept, not a production incident. I am not reusing his numbers. I built my own fixtures so the failure is reproducible on your logs, and so a test can fail closed on it.\n\nYour dashboard says $0 for that call. The dashboard is not lying. It is faithfully reporting a number that is wrong at the source. This is the whole thesis of the [pre-execution and reconciliation gates I keep building](https://finops.spinov.online/blog/pre-execution-gate-for-ai-agents/): tracking a number is not the same as controlling the thing the number describes. You tracked usage. The usage you tracked was zero. The tokens were spent anyway.\n\nHere is the claim, stated so you can break it. Given a recorded stream where text deltas exist and the terminal usage frame is absent or zero, the gate must exit non-zero with reason `delivered-but-unbilled`\n\n. Given a stream with a usage frame whose output count meets the floor derived from the delivered text, it must exit 0. One removed line has to flip the verdict. If you can show me a stream that delivered text and the gate stayed quiet, or one that logged its usage honestly and the gate blocked, the tool is broken and this post with it.\n\nTwo quantities per stream. The delivered text, rebuilt by concatenating every content delta. And the logged output tokens, pulled from the terminal usage frame, or `None`\n\nwhen that frame is missing or unusable.\n\nThe token side needs an honest word about method. Without a real tokenizer, and this tool ships with none, standard library only, I cannot tell you the exact token count of the delivered text. So I do not. I compute a FLOOR: the number of whitespace-delimited words. A BPE tokenizer keyed on whitespace almost always emits at least one token per word, and usually more once punctuation and sub-word splits count. So the real `output_tokens`\n\nis nearly always higher than this floor. It is a deliberately low, human-readable magnitude, not a measurement.\n\nThe gate's real claim does not lean on that floor being precise. It leans on something smaller that cannot be argued with: if the delivered text is non-empty, the real output token count is above zero. Therefore a logged zero is provably wrong, whatever the exact number was. The floor just gives you a readable \"at least this many\" figure to put in the alert.\n\nThat gives four verdicts:\n\n| Verdict | Condition | reason-code | exit |\n|---|---|---|---|\n| OK | usage frame present, `logged >= floor`\n|\n`usage-consistent` |\n0 |\n| UNDERCOUNT | usage frame present, `0 < logged < floor`\n|\n`partial-telemetry-loss` |\n2 |\n| BLIND | text delivered, `logged` is 0 / absent / unparseable |\n`delivered-but-unbilled` |\n1 |\n| EMPTY | no text delivered, nothing to reconcile | `nothing-delivered` |\n0 |\n\nBLIND is the hard, logical one and gets the blocking exit code. UNDERCOUNT is softer: it says the logged count is below the floor, which is suspicious but heuristic, so it warns rather than blocks. The decision itself is a few lines, simplified for the post (the real function returns dicts with detail strings, and it carries one extra guard shown here that fails closed on a stream in a shape it cannot recognize):\n\n``` python\ndef classify(delivered_text, logged, usage_seen, events, recognized):  # simplified for the post\n    floor = word_floor(delivered_text)                                 # whitespace word count\n    if events and not recognized:            # valid JSON, but no shape the gate knows -> fail closed\n        return \"UNRECOGNIZED\", \"unrecognized-stream\", 2\n    if not delivered_text.strip():\n        return \"EMPTY\", \"nothing-delivered\", 0\n    if logged is None or logged == 0:\n        return \"BLIND\", \"delivered-but-unbilled\", 1\n    if logged < floor:\n        return \"UNDERCOUNT\", \"partial-telemetry-loss\", 2\n    return \"OK\", \"usage-consistent\", 0\n```\n\nThe gate reads the recorded stream as data and auto-detects the wire shape, so you can point it at the logs you already keep. It maps four documented formats to the same two quantities, delivered text and logged output tokens:\n\n| Format | Delivered text | Logged output tokens |\n|---|---|---|\n| OpenAI Chat Completions | `choices[].delta.content` |\n`usage.completion_tokens` in the final chunk, sent only when you set `stream_options={\"include_usage\": true}`\n|\n| Anthropic Messages | `content_block_delta.delta.text` |\n`message_delta.usage.output_tokens` (cumulative; the last one is the total) |\n| OpenAI Responses | `response.output_text.delta` |\n`response.completed` then `response.usage.output_tokens`\n|\n| Generic normalized | `{\"type\":\"content.delta\",\"text\":...}` |\n`{\"type\":\"usage\",\"output_tokens\":...}` |\n\nHow the accounting travels is not the same across providers, and that changes what a zero means. OpenAI Chat sends usage in one terminal chunk, and only if you asked for it: leave `include_usage`\n\noff and there is no usage frame at all, which reads as delivered-but-unbilled by default. Anthropic sends output tokens cumulatively across `message_delta`\n\nframes, so a dropped final frame undercounts rather than zeroes, unless your client commits only the terminal number. A logged 0 is a fact about what your client recorded, not a universal property of streaming. The gate does not care which of these happened. It reconciles the text that reached the user against the number that reached your logs.\n\nThe FLOOR is provider-agnostic. It comes from the concatenated delivered text alone, so the same word count governs an OpenAI stream and an Anthropic one. A format the gate does not recognize is reported as `unrecognized-stream`\n\n(exit 2), which fails closed instead of passing as EMPTY, so a broken adapter cannot hide a leak behind a green check.\n\nFeed it a recorded stream as JSONL, one event per line, in any of the formats above. Then run it:\n\n```\npython3 stream_billing_gate.py fixtures/fixture_blind.jsonl\n```\n\nNo install, no key, no network. It reads the file, prints a verdict, and returns an exit code you can wire into CI over your recorded stream logs, whether they are OpenAI Chat, Anthropic, Responses, or the normalized shape. A log in a format it does not recognize returns `unrecognized-stream`\n\n(exit 2) rather than a silent pass, so confirm the fit on a known-good stream before you trust a green.\n\nHere are two files. `fixture_ok.jsonl`\n\nis a recorded stream: a normal answer about capping agent spend, followed by a terminal usage frame reporting 190 output tokens. `fixture_blind.jsonl`\n\nis the same stream with that one usage line removed. The text deltas are byte-identical. Run each:\n\n```\nstream-billing-gate: delivered text vs logged usage\nfiles: 1\n\n[1] fixture_ok.jsonl\n    delivered : 137 words, 750 chars (floor 137 tokens)\n    logged    : output_tokens=190\n    verdict   : OK (usage-consistent) exit 0\n    detail    : logged 190 >= floor 137\n\nsummary: 1 OK, 0 UNDERCOUNT, 0 BLIND, 0 EMPTY  ->  overall exit 0\nreport-sha256: f6b8ce0b2b7a09f57a589a862b53024a81ca618cfecd34f82fa6231c670486a7\nstream-billing-gate: delivered text vs logged usage\nfiles: 1\n\n[1] fixture_blind.jsonl\n    delivered : 137 words, 750 chars (floor 137 tokens)\n    logged    : none\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 137 tokens of text; no usage frame at all\n\nsummary: 0 OK, 0 UNDERCOUNT, 1 BLIND, 0 EMPTY  ->  overall exit 1\nreport-sha256: 318dff2585a7874bd04cd06a1430a9d3b303d93535910bd46be1d5987f81a4ce\n```\n\nSame 137 words delivered. In one file that costs at least 137 tokens and the log says 190. In the other it costs at least 137 tokens and the log says nothing. The exit code goes from 0 to 1. The only difference between the two files is a single line:\n\n``` bash\n$ diff fixtures/fixture_ok.jsonl fixtures/fixture_blind.jsonl\n10d9\n< {\"type\":\"usage\",\"input_tokens\":523,\"output_tokens\":190}\n```\n\nThat is the failure in one line. In production the line does not get deleted by hand. A connection blips, a proxy truncates, a client swallows the final frame, and you are left with `fixture_blind.jsonl`\n\nand a dashboard that says the call was free. The two sha256 digests are the tool hashing its own report, so you can confirm you got the same bytes I did.\n\nHand it a batch of recorded streams and it reconciles each one. This run (`stream_billing_gate.py fixtures/*.jsonl`\n\n, so the files come in alphabetical order) covers all four verdicts plus the two broken shapes of the usage frame, present-but-zero and present-but-unparseable, both of which are still delivered-but-unbilled:\n\n```\nstream-billing-gate: delivered text vs logged usage\nfiles: 7\n\n[1] fixture_blind.jsonl\n    delivered : 137 words, 750 chars (floor 137 tokens)\n    logged    : none\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 137 tokens of text; no usage frame at all\n\n[2] fixture_broken_usage.jsonl\n    delivered : 46 words, 274 chars (floor 46 tokens)\n    logged    : none\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 46 tokens of text; usage frame present but unparseable\n\n[3] fixture_empty.jsonl\n    delivered : 0 words, 0 chars (floor 0 tokens)\n    logged    : none\n    verdict   : EMPTY (nothing-delivered) exit 0\n    detail    : no text delivered; nothing to reconcile\n\n[4] fixture_injection.jsonl\n    delivered : 60 words, 344 chars (floor 60 tokens)\n    logged    : none\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 60 tokens of text; no usage frame at all\n\n[5] fixture_ok.jsonl\n    delivered : 137 words, 750 chars (floor 137 tokens)\n    logged    : output_tokens=190\n    verdict   : OK (usage-consistent) exit 0\n    detail    : logged 190 >= floor 137\n\n[6] fixture_undercount.jsonl\n    delivered : 72 words, 403 chars (floor 72 tokens)\n    logged    : output_tokens=12\n    verdict   : UNDERCOUNT (partial-telemetry-loss) exit 2\n    detail    : logged 12 < floor 72; telemetry lost part of the stream\n\n[7] fixture_zero_usage.jsonl\n    delivered : 55 words, 272 chars (floor 55 tokens)\n    logged    : output_tokens=0\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 55 tokens of text; usage frame logged output_tokens=0\n\nsummary: 1 OK, 1 UNDERCOUNT, 4 BLIND, 1 EMPTY  ->  overall exit 1\nreport-sha256: dc204b014afadff09b09b82a791dd86ba7767df01454fbc2cc011321c9cd5111\n```\n\nLook at `fixture_zero_usage`\n\nand `fixture_broken_usage`\n\n. A usage frame that says `output_tokens: 0`\n\nand a usage frame with a broken value both land as BLIND, because in both cases text was delivered and no honest token count was logged. A zero is not an OK. Absence is not an OK. Only a real count that meets the floor is an OK. The report ends with a sha256 of its own body, and because that body depends on the order you pass the files, `fixtures/*.jsonl`\n\nreproduces this exact digest.\n\nThose seven fixtures use the normalized shape, which is fine for showing the logic but proves nothing about the streams you actually record. So `fixtures/providers/`\n\nholds streams written in the documented wire formats of three providers, each representative of what their SSE emits: an Anthropic Messages stream, an OpenAI Chat Completions stream, and an OpenAI Responses stream. For each provider there is a delivered-but-unbilled variant and an honest one, plus a foreign log that matches no known shape:\n\n```\nstream-billing-gate: delivered text vs logged usage\nfiles: 7\n\n[1] anthropic_blind.jsonl\n    delivered : 54 words, 291 chars (floor 54 tokens)\n    logged    : output_tokens=0\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 54 tokens of text; usage frame logged output_tokens=0\n\n[2] anthropic_ok.jsonl\n    delivered : 54 words, 291 chars (floor 54 tokens)\n    logged    : output_tokens=63\n    verdict   : OK (usage-consistent) exit 0\n    detail    : logged 63 >= floor 54\n\n[3] openai_chat_blind.jsonl\n    delivered : 34 words, 178 chars (floor 34 tokens)\n    logged    : none\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 34 tokens of text; no usage frame at all\n\n[4] openai_chat_ok.jsonl\n    delivered : 34 words, 178 chars (floor 34 tokens)\n    logged    : output_tokens=64\n    verdict   : OK (usage-consistent) exit 0\n    detail    : logged 64 >= floor 34\n\n[5] responses_blind.jsonl\n    delivered : 32 words, 172 chars (floor 32 tokens)\n    logged    : output_tokens=0\n    verdict   : BLIND (delivered-but-unbilled) exit 1\n    detail    : delivered >= 32 tokens of text; usage frame logged output_tokens=0\n\n[6] responses_ok.jsonl\n    delivered : 32 words, 172 chars (floor 32 tokens)\n    logged    : output_tokens=54\n    verdict   : OK (usage-consistent) exit 0\n    detail    : logged 54 >= floor 32\n\n[7] unknown_schema.jsonl\n    delivered : 0 words, 0 chars (floor 0 tokens)\n    logged    : none\n    verdict   : UNRECOGNIZED (unrecognized-stream) exit 2\n    detail    : 4 event(s) parsed, none matched a known stream shape; map your adapter's text and usage fields before trusting a pass\n\nsummary: 3 OK, 0 UNDERCOUNT, 3 BLIND, 0 EMPTY, 1 UNRECOGNIZED  ->  overall exit 1\nreport-sha256: ec93efeb393fc0606656eb9f5447ca1dbd9837ee5a51258dc0689b3d7a022aa5\n```\n\nThree providers, three delivered-but-unbilled streams, three blocks. The Anthropic case logs `output_tokens: 0`\n\nin its terminal `message_delta`\n\n; the OpenAI Chat case ran without `include_usage`\n\nso no usage frame lands at all; the Responses case reports a zero in `response.completed`\n\n. Each honest twin logs a real count above the floor and ships. And `unknown_schema.jsonl`\n\nis the case the earlier version of this tool got wrong: a valid-JSON log in a shape the gate cannot map. It used to read as EMPTY and exit 0, which is exactly the silent pass this whole post is against. Now it fails closed as `unrecognized-stream`\n\n, exit 2, so a broken adapter shows up loud instead of hiding the leak.\n\nThe `fixture_injection`\n\ncase is a small joke that matters. Its delivered text contains a line addressed to any monitoring tool reading the log: it claims the response was served for free and instructs the reader to report `usage-consistent`\n\nand exit 0. The gate reads that line the same way it reads every other line, as text to be counted. It does not execute it. So the injected instruction just adds to the delivered token count, and the stream still gets blocked, exit 1.\n\nThis is the security posture written down: stream content is untrusted input. A recorded stream can carry anything, including a sentence designed to talk your tooling into standing down. A gate that treats logs as data and never as commands is not optional when the logs come off the open internet.\n\nI would rather draw the borders myself than have you find them.\n\n**It does not compute dollars.** The floor is not the vendor's invoice. The gate catches delivered-but-zero-logged, the failure where accounting collapses to nothing while text shipped. It does not tell you the exact amount you were overcharged or undercharged. For that you need the price policy, and reconciling a billed amount against declared rates is [a different tool that checks which model answered and whether the charge reconciles](https://finops.spinov.online/blog/model-receipt-probe/).\n\n**The floor is conservative, and UNDERCOUNT is a heuristic.** BLIND rests on a logical fact: text delivered means tokens above zero. UNDERCOUNT rests on a statistical one: a logged count below the word floor is suspicious. Pathological text could in principle sit near the boundary, which is exactly why UNDERCOUNT warns (exit 2) and does not block. The floor is whitespace word count, so it barely moves for scripts that do not put spaces between words; a paragraph of Chinese or Japanese collapses toward a floor of 1, and UNDERCOUNT is effectively inert there while BLIND still holds. A non-integer count is treated the same conservative way: a float like `190.0`\n\nor a string `\"190\"`\n\nis dropped as unparseable and reads as BLIND, a rare false positive if some client logs honest floats. If you need exact counts, run the delivered text through the real tokenizer for your model. This tool is the cheap first pass that needs no tokenizer to catch the zero.\n\n**It cannot tell you why the frame is missing.** A dropped connection and a vendor that ate the usage frame produce the same evidence: delivered text, no matching usage. The gate reports the fact, not the cause. It also fails closed on bad input: no arguments, a missing file, or unparseable lines return a non-zero exit rather than a silent pass, and a stream in a shape it does not recognize returns `unrecognized-stream`\n\n(exit 2), not a silent EMPTY. It reads normalized or recognized-native events, so if you feed it a custom log, confirm the gate recognizes it on a known-good stream first.\n\n**It is a different question from its neighbors.** It is not [the tokens an agent keeps burning after a run has already failed](https://finops.spinov.online/blog/waste-probe-tokens-after-failure/), and it is not [the growing transcript you get rebilled for on every turn of a conversation](https://finops.spinov.online/blog/context-tax-measure-transcript-rebill/). Those measure real spend that happened. This one measures spend that happened and then vanished from the record. If you are mapping the whole bill, it sits next to [the token tax an MCP server quietly adds to every call](https://finops.spinov.online/blog/mcp-server-token-tax/) and [a forecaster for what a loop will cost before you run it](https://finops.spinov.online/blog/loop-cost-forecaster/).\n\nThe macro mood is why this small failure is worth a gate. During the week of July 7, the loudest thread I saw on Hacker News was \"GLM 5.2 and the coming AI margin collapse,\" sitting around 680 points and 465 comments when I checked. Those are the thread's numbers, not mine, and I link nothing I cannot verify. The argument in the room is that the economics of serving these models are tightening. When margins tighten, the difference between what you actually spent and what your logs think you spent stops being a rounding error and starts being the thing that decides whether your unit economics are real.\n\nA usage frame that goes missing on a streamed response is a silent leak on the wrong side of that equation. You cannot fix a cost you never recorded. The point of reconciling the delivered text against the logged usage is not to compute your bill to the cent. It is to stop pretending a call was free when the answer is sitting right there on the screen.\n\nI publish one of these small offline gates most weeks, each one a runnable tool with the raw output and the exit codes, no keys and no network. Follow along if you want the next one. And the real question I still cannot answer cleanly, so I am asking you: on your own logs, how do you tell a genuinely dropped usage frame apart from a response that legitimately produced no billable output? I read every comment.", "url": "https://wpnews.pro/news/delivered-but-unbilled-your-ai-stream-logged-zero-tokens", "canonical_source": "https://dev.to/alex_spinov/delivered-but-unbilled-your-ai-stream-logged-zero-tokens-3c99", "published_at": "2026-07-10 04:24:37+00:00", "updated_at": "2026-07-10 05:05:28.717750+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-products"], "entities": ["stream_billing_gate.py", "kielltampubolon", "Dev.to", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/delivered-but-unbilled-your-ai-stream-logged-zero-tokens", "markdown": "https://wpnews.pro/news/delivered-but-unbilled-your-ai-stream-logged-zero-tokens.md", "text": "https://wpnews.pro/news/delivered-but-unbilled-your-ai-stream-logged-zero-tokens.txt", "jsonld": "https://wpnews.pro/news/delivered-but-unbilled-your-ai-stream-logged-zero-tokens.jsonld"}}