# A cost cap that over-bills by 100x is an availability bug

> Source: <https://dev.to/k08200/a-cost-cap-that-over-bills-by-100x-is-an-availability-bug-4f43>
> Published: 2026-08-15 07:05:00+00:00

A cost cap is supposed to be the boring part. You pick a number, you stop spending past it, you go back to work. Mine did the opposite: it became the thing most likely to take my service down, and it did that while the actual bill stayed under a dollar.

The bug was a substring match in a pricing table. The interesting part is not the bug — it is that I had already "fixed" this table once, in the safe direction, and the safe direction is what broke it.

Klorn classifies every inbound email into exactly one of four tiers. Every classification is one LLM call, so cost scales with mail volume, and mail volume is not something I control. That is the whole reason a ceiling exists.

There are three of them:

``` js
// packages/api/src/config.ts
export const DAILY_COST_CAP_CENTS = intEnv("DAILY_COST_CAP_CENTS", 100);        // $1.00 / user / day
export const FREE_DAILY_COST_CAP_CENTS = intEnv("FREE_DAILY_COST_CAP_CENTS", 10); // $0.10 free tier
export const GLOBAL_DAILY_COST_CAP_CENTS = intEnv("GLOBAL_DAILY_COST_CAP_CENTS", 5000); // $50 / day fleet-wide
```

The per-user caps are the real guard. The global one is the fatal-bill backstop — the thing that stops a runaway loop from producing a five-figure invoice while I sleep.

None of these can meter a real invoice in real time. They estimate: token counts times a per-model rate, accumulated into a ledger. So there is a table mapping model id to price, and everything downstream trusts it.

The original table had a single rate for paid models. Whatever the model was, it got billed at roughly Gemini Flash prices.

That is fine right up until a fallback chain routes a request to Claude Sonnet, which is 10x Flash on input and 6x on output. The ledger records Flash. The cap sees a number far below the truth, never fires, and the protection you think you have is decorative. This is the failure mode everyone anticipates when they build a cost cap: **under-billing, which costs money.**

So I fixed it in the obvious direction — a family table, and an explicit decision about what to do with models the table has never seen:

```
/**
 * Unknown paid models price at the sonnet tier, NOT at flash rates: a new
 * frontier model landing in CHAT_MODEL before this table learns it must be
 * over-billed against the caps, never 20x under-billed (the original bug).
 */
const DEFAULT_MODEL_RATE: ModelRateUsdPerMTok = { input: 3, output: 15 };
```

Unknown model? Charge it like Sonnet. Over-estimate on purpose. If I am wrong, I am wrong in the direction that stops spending too early rather than too late.

That reasoning is correct, and it is also how I shipped the second bug.

Klorn's judge has a fallback chain — cheap paid SKUs it drops to when the primary model is rate-limited or down. Those SKUs are genuinely cheap. Not "cheaper," cheap enough to change the shape of the arithmetic.

The table matched them by substring. First row where every needle appears in the model id wins. So:

| Model | Real rate (in/out per 1M) | What the table charged | Off by |
|---|---|---|---|
`openai/gpt-oss-120b` |
$0.037 / $0.17 | $2.50 / $10.00 — matched the generic `["gpt"]` row |
~67x / ~59x |
`qwen/qwen3.7-flash` |
$0.03 / $0.13 | $3.00 / $15.00 — matched nothing, took the Sonnet default | ~100x / ~115x |
`mistralai/mistral-nemo` |
$0.019 / $0.03 | $2.00 / $6.00 — matched the generic `["mistral"]` row |
~105x / ~200x |

Every one of these is an over-estimate. Every one of them is "safe" by the rule I had just written down.

Now put it against the load. Measured, not guessed: a classification is about 1,000 prompt tokens and 100 completion tokens. At 100 users doing 100 emails a day, that is 10,000 classifications, which at the primary model's real rates is about **$5.50/day** — call it $7/day with reply drafts and briefings on top. Against a $50/day ceiling that is roughly 7x headroom, which is the amount of headroom I wanted.

Now suppose the primary is rate-limited and the fleet spends the day on `qwen/qwen3.7-flash`

, the cheap middle of the chain. The real cost of that same work *drops*, because the fallback is cheaper than the primary: 10,000 calls at $0.03/$0.13 per 1M is about **$0.43/day**. The ledger, pricing it at the Sonnet default of $3/$15, records **$45/day**.

The same work. Two numbers, a hundredfold apart, and the cap acts on the wrong one.

$45 against a $50 ceiling means the fleet trips the global cap inside a single day — and on any day with more mail than average, it trips mid-morning. When it trips, classification stops. For everyone. Not because anything cost too much, but because a substring did not match.

This is the part I got wrong, and it generalizes past my codebase.

I had been reasoning about the cap as a **financial** control, where the two failure modes are "spent too much" and "spent too little," and only one of them is dangerous. Under that model, over-estimating is free.

But a cap is not only a financial control. It is a **runtime dependency of the request path**. Every classification asks it for permission. Which means the error budget has two axes, not one:

A protective ceiling that fires during normal operation has stopped protecting and started breaking. And over-billing is the sneakier of the two, because it never shows up on a bill you can go look at. The one artifact that would tell you something is wrong — a large invoice — is precisely the thing that does not happen. You get an outage with a healthy credit card.

There is a second-order version of this too. I had separately raised the global cap from $10/day to $50/day, correctly, because $10 left barely one heavy day of headroom against $7/day steady state. A protective ceiling that fires in normal operation is not a ceiling, it is a scheduled outage. But raising the ceiling also hides mis-metering for longer: with a $10 cap the 100x error would have tripped in an hour and I would have found it immediately. Slack in the system is not free either — it buys you time and it buys the bug time.

The fix is unglamorous — explicit rows above the generic ones, because first match wins:

```
// Cheap fallback SKUs, priced from the live OpenRouter catalog 2026-08-10
// and rounded up. These rows exist because the generic family rows below
// mis-price them by 60-100x [...] Over-billing is "safe" for a protective
// cap in isolation, but at 100 users it burns the daily caps ~100x too fast
// and the fleet stops classifying mid-day — the caps stop protecting and
// start breaking. Specific rows MUST stay above generic.
{ match: ["gpt-oss"], rate: { input: 0.05, output: 0.2 } },
{ match: ["qwen"], rate: { input: 0.1, output: 0.3 } },
{ match: ["mistral", "nemo"], rate: { input: 0.05, output: 0.1 } },
{ match: ["nemotron"], rate: { input: 0.1, output: 0.4 } },
```

Note these are still rounded *up* from the real catalog prices — between 1.2x and 3.3x, depending on the row. That is deliberate: the catalog moves, and a table that under-bills is the first bug again. Over-estimating by 3x is a rounding policy against a moving target. Over-estimating by 100x is an outage. The direction was never the problem; the magnitude was.

Ordering is now load-bearing, which means a comment is not enough. It is pinned by a table-driven test:

```
// packages/api/src/__tests__/model-fallback.test.ts
describe("fallback-chain SKUs are metered at their real rates (100-user economics)", () => {
  it.each([
    ["openai/gpt-oss-120b", 0.05, 0.2],
    ["qwen/qwen3.7-flash", 0.1, 0.3],
    ["mistralai/mistral-nemo", 0.05, 0.1],
    ["google/gemini-3.1-flash-lite", 0.25, 1.5],
    ["openai/gpt-5-nano", 0.1, 0.4],
  ])("%s is priced at $%s/$%s per M", (model, input, output) => {
    expect(resolveModelRateUsdPerMTok(model)).toEqual({ input, output });
  });

  it("keeps the sonnet-tier default for genuinely unknown models", () => {
    expect(resolveModelRateUsdPerMTok("acme/brand-new-frontier")).toEqual({
      input: 3,
      output: 15,
    });
  });
});
```

Every entry in the chain is in that table now, not just the three that were wrong — because the next person to add a chain entry should see a row missing, not guess. And the second test pins the unknown-model default deliberately, so a future contributor does not "simplify" the over-billing away.

The operational rule that falls out: **any model added to the fallback chain must be added to the rate table in the same change.** A chain entry without a rate row is not a missing optimization, it is a latent 100x metering error pointed at your availability.

Three things, in order of how much they cost to learn.

**Ask what your cap actually gates.** If the answer is "the request path," it is a dependency, and dependencies get availability budgets, not just correctness checks. I had been treating mine as an accounting feature.

**Write down which direction of error is safe, and then check whether that is still true at scale.** "Over-estimate on purpose" was correct in isolation and wrong at 10,000 calls a day. The rule did not change; the multiplier did. Any heuristic phrased as a direction rather than a magnitude has this failure mode waiting in it.

**Instrument the ratio, not the total.** A dashboard of daily spend would have shown $45 and looked like a busy day. What would have caught this in minutes is metered cost divided by a sanity estimate of real cost — a number that should sit near 1 and was sitting near 100. I did not have that, and the thing that eventually surfaced it was working through the unit economics on paper before the user count made it urgent.

The whole thing is open source under AGPL if you want to read the table, the ordering constraint, or the tests around it — [github.com/k08200/klorn](https://github.com/k08200/klorn). The pricing table and the fallback resolver live in `packages/api/src/llm/model-fallback.ts`

.

If you run LLM calls behind a spend ceiling, go check what your table does with the cheapest model in your fallback chain. The expensive models are the ones you remembered to price.