# We audited 110 AI usage tools. Here is where the numbers go wrong. > Source: > Published: 2026-09-22 23:12:53+00:00 AI products are moving from flat subscriptions to usage-based and outcome-based billing: per-resolution support agents, premium request quotas, per-token gateway billing. Every one of those numbers is produced by a meter, and the meter is usually the vendor's own. We spent the last month auditing the open-source side of that world — 110 tools that count tokens, track costs, or enforce budgets — and this post is the field report. The short version: **45+ verified bugs, five recurring families, 23 fixes already merged upstream** (including langfuse at 34k stars and codeburn at 11k). An independent auditor reproduced our entire conformance suite and confirmed one cache-accounting path that under-reports usage by roughly 99%. Every claim below links to a reproducible check. **1. Stale pricing tables.** The most common real bug, and the most boring: pricing tables that lack the current generation of models. In one batch of seven tools, five shipped outdated or missing rate rows. The tool computes confidently from numbers that stopped being true months ago — and every downstream total inherits the error. **2. Provider-wrong cache multipliers.** Cache reads and writes are billed at different multipliers per provider. A common shortcut hardcodes one provider's ratios and applies them to everyone. We found a tool applying Anthropic's cache-read discount to OpenAI models, underestimating cache reads by 5x. The billing math is wrong in a way that looks plausible on every individual line. **3. Retry double-counting in stream aggregation.** When a streaming request is retried byte-identical, some aggregators count both attempts. In one public corpus of 604 re-emitted events — 46% of the corpus — the bytes were identical, and naive aggregation doubled them. Conversely, tools that deduplicate too aggressively miss genuine retried work. Both directions lose money for someone. **4. Absent treated as zero.** When a usage field is missing, `or 0` turns "unknown" into "free". It reads like defensive coding and behaves like a discount nobody authorized. The fix is semantic, not syntactic: absent must stay absent, and rollups must be allowed to say UNPROVABLE instead of zero. **5. Window boundary errors.** Quota windows anchored to wall-clock time with fixtures pinned to absolute dates — the tool passes tests for 30 days and then fails everywhere at once. We know this one intimately because our own CI did it to us. The part we are proudest of is not our numbers — it is that an external auditor, working separately, reproduced our full conformance suite (236 checks) and then contributed his own quantified audit of a commercial provider's cache accounting: four code paths failing, usage under-reported by 98.9% and 95.1% on the affected paths, pinned to an exact commit. That report is now in our evidence tree with named credit. A third party spending real effort to check your work is worth more than any testimonial. 23 fixes across the ecosystem carry the audit's fingerprints — pricing table updates, cache multipliers corrected per provider, retry-collapse guards with regression fixtures, absent-vs-zero semantics made explicit. None of these were adversarial finds. Every one started as a thread where the maintainer was already discussing the problem, which is the only place we work. We checked 20 commercial vendors for one thing: a published process for what happens when the meter is wrong — a dispute path, a correction policy, anything. None had one. When the vendor runs the meter, grades its own homework, and publishes no correction process, "trust us" is the whole control framework. Two minimum standards would change that: a named dispute path, and machine-checkable billing disclosures. The [conformance pack](https://github.com/roy-tong/AgentMeasure) is open source (MIT). Export your usage data, run the checker locally — nothing leaves your machine — and it separates logical operations from physical attempts, cache reads from cache writes, and absent from zero, with every verdict traceable to a named rule. If a number on your invoice survives an independent recount, you can defend it. If it doesn't, now you have the receipt. *The full audit report with per-tool findings: [campaigns/audit-report-2026-09.md](https://github.com/roy-tong/AgentMeasure/blob/main/campaigns/audit-report-2026-09.md). The settlement spec behind the checker: [AMS-1](https://github.com/roy-tong/AgentMeasure/blob/main/standard/SETTLEMENT.md).*