{"slug": "a-cost-cap-that-over-bills-by-100x-is-an-availability-bug", "title": "A cost cap that over-bills by 100x is an availability bug", "summary": "Klorn, an email classification service, suffered a cost cap bug that over-billed by up to 100x due to substring matching in its pricing table. The developer had previously fixed the table to over-estimate unknown models, but this caused cheap fallback models to be charged at higher rates, leading to premature spending caps and potential service outages. The bug highlights the importance of precise model rate mapping in cost control systems.", "body_md": "A cost cap is supposed to be the boring part. You pick a number, you stop spending past it, you go back to work. Mine did the opposite: it became the thing most likely to take my service down, and it did that while the actual bill stayed under a dollar.\n\nThe bug was a substring match in a pricing table. The interesting part is not the bug — it is that I had already \"fixed\" this table once, in the safe direction, and the safe direction is what broke it.\n\nKlorn classifies every inbound email into exactly one of four tiers. Every classification is one LLM call, so cost scales with mail volume, and mail volume is not something I control. That is the whole reason a ceiling exists.\n\nThere are three of them:\n\n``` js\n// packages/api/src/config.ts\nexport const DAILY_COST_CAP_CENTS = intEnv(\"DAILY_COST_CAP_CENTS\", 100);        // $1.00 / user / day\nexport const FREE_DAILY_COST_CAP_CENTS = intEnv(\"FREE_DAILY_COST_CAP_CENTS\", 10); // $0.10 free tier\nexport const GLOBAL_DAILY_COST_CAP_CENTS = intEnv(\"GLOBAL_DAILY_COST_CAP_CENTS\", 5000); // $50 / day fleet-wide\n```\n\nThe per-user caps are the real guard. The global one is the fatal-bill backstop — the thing that stops a runaway loop from producing a five-figure invoice while I sleep.\n\nNone of these can meter a real invoice in real time. They estimate: token counts times a per-model rate, accumulated into a ledger. So there is a table mapping model id to price, and everything downstream trusts it.\n\nThe original table had a single rate for paid models. Whatever the model was, it got billed at roughly Gemini Flash prices.\n\nThat is fine right up until a fallback chain routes a request to Claude Sonnet, which is 10x Flash on input and 6x on output. The ledger records Flash. The cap sees a number far below the truth, never fires, and the protection you think you have is decorative. This is the failure mode everyone anticipates when they build a cost cap: **under-billing, which costs money.**\n\nSo I fixed it in the obvious direction — a family table, and an explicit decision about what to do with models the table has never seen:\n\n```\n/**\n * Unknown paid models price at the sonnet tier, NOT at flash rates: a new\n * frontier model landing in CHAT_MODEL before this table learns it must be\n * over-billed against the caps, never 20x under-billed (the original bug).\n */\nconst DEFAULT_MODEL_RATE: ModelRateUsdPerMTok = { input: 3, output: 15 };\n```\n\nUnknown model? Charge it like Sonnet. Over-estimate on purpose. If I am wrong, I am wrong in the direction that stops spending too early rather than too late.\n\nThat reasoning is correct, and it is also how I shipped the second bug.\n\nKlorn's judge has a fallback chain — cheap paid SKUs it drops to when the primary model is rate-limited or down. Those SKUs are genuinely cheap. Not \"cheaper,\" cheap enough to change the shape of the arithmetic.\n\nThe table matched them by substring. First row where every needle appears in the model id wins. So:\n\n| Model | Real rate (in/out per 1M) | What the table charged | Off by |\n|---|---|---|---|\n`openai/gpt-oss-120b` |\n$0.037 / $0.17 | $2.50 / $10.00 — matched the generic `[\"gpt\"]` row |\n~67x / ~59x |\n`qwen/qwen3.7-flash` |\n$0.03 / $0.13 | $3.00 / $15.00 — matched nothing, took the Sonnet default | ~100x / ~115x |\n`mistralai/mistral-nemo` |\n$0.019 / $0.03 | $2.00 / $6.00 — matched the generic `[\"mistral\"]` row |\n~105x / ~200x |\n\nEvery one of these is an over-estimate. Every one of them is \"safe\" by the rule I had just written down.\n\nNow put it against the load. Measured, not guessed: a classification is about 1,000 prompt tokens and 100 completion tokens. At 100 users doing 100 emails a day, that is 10,000 classifications, which at the primary model's real rates is about **$5.50/day** — call it $7/day with reply drafts and briefings on top. Against a $50/day ceiling that is roughly 7x headroom, which is the amount of headroom I wanted.\n\nNow suppose the primary is rate-limited and the fleet spends the day on `qwen/qwen3.7-flash`\n\n, the cheap middle of the chain. The real cost of that same work *drops*, because the fallback is cheaper than the primary: 10,000 calls at $0.03/$0.13 per 1M is about **$0.43/day**. The ledger, pricing it at the Sonnet default of $3/$15, records **$45/day**.\n\nThe same work. Two numbers, a hundredfold apart, and the cap acts on the wrong one.\n\n$45 against a $50 ceiling means the fleet trips the global cap inside a single day — and on any day with more mail than average, it trips mid-morning. When it trips, classification stops. For everyone. Not because anything cost too much, but because a substring did not match.\n\nThis is the part I got wrong, and it generalizes past my codebase.\n\nI had been reasoning about the cap as a **financial** control, where the two failure modes are \"spent too much\" and \"spent too little,\" and only one of them is dangerous. Under that model, over-estimating is free.\n\nBut a cap is not only a financial control. It is a **runtime dependency of the request path**. Every classification asks it for permission. Which means the error budget has two axes, not one:\n\nA protective ceiling that fires during normal operation has stopped protecting and started breaking. And over-billing is the sneakier of the two, because it never shows up on a bill you can go look at. The one artifact that would tell you something is wrong — a large invoice — is precisely the thing that does not happen. You get an outage with a healthy credit card.\n\nThere is a second-order version of this too. I had separately raised the global cap from $10/day to $50/day, correctly, because $10 left barely one heavy day of headroom against $7/day steady state. A protective ceiling that fires in normal operation is not a ceiling, it is a scheduled outage. But raising the ceiling also hides mis-metering for longer: with a $10 cap the 100x error would have tripped in an hour and I would have found it immediately. Slack in the system is not free either — it buys you time and it buys the bug time.\n\nThe fix is unglamorous — explicit rows above the generic ones, because first match wins:\n\n```\n// Cheap fallback SKUs, priced from the live OpenRouter catalog 2026-08-10\n// and rounded up. These rows exist because the generic family rows below\n// mis-price them by 60-100x [...] Over-billing is \"safe\" for a protective\n// cap in isolation, but at 100 users it burns the daily caps ~100x too fast\n// and the fleet stops classifying mid-day — the caps stop protecting and\n// start breaking. Specific rows MUST stay above generic.\n{ match: [\"gpt-oss\"], rate: { input: 0.05, output: 0.2 } },\n{ match: [\"qwen\"], rate: { input: 0.1, output: 0.3 } },\n{ match: [\"mistral\", \"nemo\"], rate: { input: 0.05, output: 0.1 } },\n{ match: [\"nemotron\"], rate: { input: 0.1, output: 0.4 } },\n```\n\nNote these are still rounded *up* from the real catalog prices — between 1.2x and 3.3x, depending on the row. That is deliberate: the catalog moves, and a table that under-bills is the first bug again. Over-estimating by 3x is a rounding policy against a moving target. Over-estimating by 100x is an outage. The direction was never the problem; the magnitude was.\n\nOrdering is now load-bearing, which means a comment is not enough. It is pinned by a table-driven test:\n\n```\n// packages/api/src/__tests__/model-fallback.test.ts\ndescribe(\"fallback-chain SKUs are metered at their real rates (100-user economics)\", () => {\n  it.each([\n    [\"openai/gpt-oss-120b\", 0.05, 0.2],\n    [\"qwen/qwen3.7-flash\", 0.1, 0.3],\n    [\"mistralai/mistral-nemo\", 0.05, 0.1],\n    [\"google/gemini-3.1-flash-lite\", 0.25, 1.5],\n    [\"openai/gpt-5-nano\", 0.1, 0.4],\n  ])(\"%s is priced at $%s/$%s per M\", (model, input, output) => {\n    expect(resolveModelRateUsdPerMTok(model)).toEqual({ input, output });\n  });\n\n  it(\"keeps the sonnet-tier default for genuinely unknown models\", () => {\n    expect(resolveModelRateUsdPerMTok(\"acme/brand-new-frontier\")).toEqual({\n      input: 3,\n      output: 15,\n    });\n  });\n});\n```\n\nEvery entry in the chain is in that table now, not just the three that were wrong — because the next person to add a chain entry should see a row missing, not guess. And the second test pins the unknown-model default deliberately, so a future contributor does not \"simplify\" the over-billing away.\n\nThe operational rule that falls out: **any model added to the fallback chain must be added to the rate table in the same change.** A chain entry without a rate row is not a missing optimization, it is a latent 100x metering error pointed at your availability.\n\nThree things, in order of how much they cost to learn.\n\n**Ask what your cap actually gates.** If the answer is \"the request path,\" it is a dependency, and dependencies get availability budgets, not just correctness checks. I had been treating mine as an accounting feature.\n\n**Write down which direction of error is safe, and then check whether that is still true at scale.** \"Over-estimate on purpose\" was correct in isolation and wrong at 10,000 calls a day. The rule did not change; the multiplier did. Any heuristic phrased as a direction rather than a magnitude has this failure mode waiting in it.\n\n**Instrument the ratio, not the total.** A dashboard of daily spend would have shown $45 and looked like a busy day. What would have caught this in minutes is metered cost divided by a sanity estimate of real cost — a number that should sit near 1 and was sitting near 100. I did not have that, and the thing that eventually surfaced it was working through the unit economics on paper before the user count made it urgent.\n\nThe whole thing is open source under AGPL if you want to read the table, the ordering constraint, or the tests around it — [github.com/k08200/klorn](https://github.com/k08200/klorn). The pricing table and the fallback resolver live in `packages/api/src/llm/model-fallback.ts`\n\n.\n\nIf you run LLM calls behind a spend ceiling, go check what your table does with the cheapest model in your fallback chain. The expensive models are the ones you remembered to price.", "url": "https://wpnews.pro/news/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug", "canonical_source": "https://dev.to/k08200/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug-4f43", "published_at": "2026-08-15 07:05:00+00:00", "updated_at": "2026-08-15 07:10:47.998743+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["Klorn", "Gemini Flash", "Claude Sonnet", "OpenAI", "Qwen", "Mistral"], "alternates": {"html": "https://wpnews.pro/news/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug", "markdown": "https://wpnews.pro/news/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug.md", "text": "https://wpnews.pro/news/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug.txt", "jsonld": "https://wpnews.pro/news/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug.jsonld"}}