{"slug": "we-stress-tested-ai-spend-limits-under-bursts-and-outages", "title": "We stress-tested AI spend limits under bursts and outages", "summary": "A stress test of LiteLLM and Revenium found that LiteLLM, configured with Redis and its fail-closed setting, held every $0.05 per-customer budget to within about 2.5% even through a database outage, while Revenium's accounting matched the testers' own tally to within $0.00004 but overshot budgets by 117% at its default 60-second rule refresh and 14.5% at a 10-second refresh. On LiteLLM's default setup, spend went unrecorded, budgets were handed back after a restart, and customers slipped one call past their limit, with a single large call near the limit overshooting by 15.6%. The full test suite, which ran gpt-4o-mini calls through a Python harness at a $0.05 budget per customer and included a 250-customer concurrency run, cost under $2 in OpenAI usage.", "body_md": "# We Stress-Tested AI Spend Limits. Here's Where They Held and Where They Slipped.\n\nIf you sell AI features and bill your customers for usage, a budget limit is a promise about your margins. We wanted to know how well that promise holds when things get messy: parallel calls, oversized responses, backend outages, restarts, and hundreds of customers at once. So we tested two tools, LiteLLM, the popular open-source proxy, and Revenium, a commercial AI cost platform.\n\nThe short answer: configured well, LiteLLM held every budget we set to within about 2.5%, even through a database outage. But several of its safest behaviors are off by default, and on the default setup we saw spend go unrecorded, budgets handed back after a restart, and customers slipping one call past their limit. Revenium's accounting was accurate to a fraction of a cent, but its enforcement works from a copy of the rules it refreshes on a timer, so overshoot grew with how fast a customer was spending.\n\nThe full set of tests cost under $2 in OpenAI usage. Here's what we did and what we found.\n\n## How we tested\n\nWe wrote a small Python harness that sends calls to gpt-4o-mini on behalf of test customers, each with its own spend limit, and logs every call's cost, latency and whether it was blocked. Most tests used a $0.05 budget per customer, small enough to cross in about 35 calls. Every test started with a fresh customer at $0, and a stop-loss in the harness aborted any run that reached three times its budget.\n\n- **LiteLLM** ran as a self-hosted proxy with Postgres, using per-customer`max_budget` limits. We tested its default setup, then with its fail-closed setting, then with Redis.\n- **Revenium** ran through its Python SDK with a live budget rule: total cost, $0.05 per day, grouped by customer, with blocking turned on. We tested its default 60-second rule refresh and a 10-second refresh.\n\nThe tests:\n\n1. **Calls one at a time** until the customer is blocked.\n2. **A 20-call burst** , fired at once after spending 90% of the budget.\n3. **One large call** near the limit, allowed up to 16,000 output tokens.\n4. **A cancelled stream** , to see whether partial output is counted.\n5. **Backend outages:** stopping LiteLLM's database, or cutting Revenium's connection, partway through a run.\n6. **250 customers at once,** each with a tiny budget, to see how limits behave at the scale of a real SaaS product.\n\n## Results at a glance\n\n| Overshoot past a $0.05 budget, by test |  |  | \n|---|---|---|\n| Test | Tool | Overshoot past budget | \n|---|---|---|\n| Calls one at a time | LiteLLM | 2.4% | \n| Calls one at a time, 60 s refresh | Revenium | 117% | \n| Calls one at a time, 10 s refresh | Revenium | 14.5% | \n| 20-call burst | LiteLLM | 2.5% | \n| 20-call burst, 10 s refresh | Revenium | 51% | \n| One large call near the limit | LiteLLM | 15.6% | \n| One large call near the limit | Revenium | 4.8% | \n| Cut off while under budget | Revenium | at least 198% | \n\nTwo things don't fit in a chart. Revenium's recorded spend matched our own tally to within $0.00004 in normal operation, so its accounting is solid; its overshoot comes from timing. And with Redis and its fail-closed setting turned on, LiteLLM's default-setup gaps closed: the large call was rejected before it reached OpenAI, and nothing was lost in an outage.\n\n## What held up\n\n**LiteLLM handled concurrency better than we expected.** We predicted a 20-call burst would sail past the budget, since each call would check the same counter before any of them finished. Instead, only 4 of the 20 got through. Reading the source explains why: before forwarding a call, LiteLLM reserves its worst-case cost (input tokens plus `max_tokens`) against the customer's budget, and every new call sees the reservations of calls still in flight. When a call finishes, the reservation settles to the real cost. A cancelled stream was settled to its input cost, which is close to what it actually cost.\n\n**LiteLLM's fail-closed setting stops oversized calls.** With `fail_closed_budget_enforcement: true`, a call whose worst-case cost didn't fit the remaining budget was rejected in 25 ms, before it reached OpenAI.\n\n**LiteLLM with Redis survived every outage we tried.** With `REDIS_HOST`/` REDIS_PORT` set and `use_redis_transaction_buffer: true`, spend recorded while Postgres was down was re-queued and written once it came back, and a customer over budget stayed blocked after a proxy restart. With 250 customers active at once, every one was blocked right on schedule.\n\n**The proxy is cheap in latency.** LiteLLM added about 28 ms at the median and 114 ms at the 95th percentile, small next to the model's own response time.\n\n**Revenium's books were accurate.** In normal operation, its recorded spend matched ours to within $0.00004, and once it blocked a customer, it kept blocking them even after we cut its connection.\n\n## Where limits slipped\n\n### LiteLLM, on its default setup\n\n- **Oversized calls get through.** When a call's worst-case cost doesn't fit the remaining budget, LiteLLM shrinks the reservation to whatever is left and lets the call through. Our 16,000-token call at 95% of budget ended 15.6% over.\n- **An outage can lose spend for good.** With Postgres stopped, LiteLLM kept blocking from its in-memory total, but its batched spend writes failed and were dropped after a few retries. Once Postgres came back, it had recorded $0.0196 of $0.0512 actually spent.\n- **A restart can hand budget back.** After a proxy restart, LiteLLM reloaded the lower database figure, and a customer who had really spent past their limit was allowed through again.\n- **Hundreds of customers strain the counter cache.** LiteLLM's spend counter cache holds 200 entries ([issue #40221](https://github.com/BerriAI/litellm/issues/40221) ). With 250 customers active, 98 of them, the oldest entries in the cache, each got one extra call past their budget. The effect was small in our test, but it grows with how far the database lags behind real spend.\n\nAll four disappeared with Redis and the fail-closed setting turned on. Several related LiteLLM issues are open, including [#35529](https://github.com/BerriAI/litellm/issues/35529), [#34238](https://github.com/BerriAI/litellm/issues/34238) and [#43732](https://github.com/BerriAI/litellm/issues/43732); we haven't reproduced those.\n\n### Revenium\n\n- **Enforcement lags by one rule refresh.** The SDK decides locally, in under a millisecond, using a copy of the rules it refreshes on a timer. In our tests the block landed about one refresh interval after spend crossed the limit, so overshoot works out to roughly spending rate times refresh interval. At the 60-second default, an agent spending $1 a second could run about $60 past its limit, however large the budget.\n- **Bursts aren't held back.** All 20 calls in the burst went through, because each one checked the same cached copy.\n- **A disconnected SDK can't start blocking.** When we cut its connection while a customer was under budget, it never blocked them; spend reached three times the budget before our stop-loss ended the run. Usage reports wait in a memory-only buffer, so they arrive once the connection returns but are lost if the process exits first.\n- **Two setup traps.** New rules start in shadow mode, which only records would-be blocks. And with SDK 0.12.0 on Revenium's cloud API, enforcement only worked once we set`REVENIUM_ENFORCEMENT_BASE_URL=https://api.revenium.ai/profitstream` . The SDK's README lists this value but marks it optional, and the default produces a rules URL that returns 404. When that happened, every call was allowed and the SDK logged a warning that the rules couldn't be refreshed, without mentioning the 404; the cause showed only at debug level. We configured from the SDK source and missed the README line. Revenium pointed out that the setting is documented in the SDK README but marked optional, and is updating it to be more clear for future installers. Revenium then fixed this in SDK 0.12.1, released October 31, 2026, which also updates the README to document the default address. We confirmed the fix: the default address now includes`/profitstream` , an address that answers 404 is retried once with`/profitstream` added or removed, and a rules fetch refused with a 401, 403 or 404 is logged at WARNING with the URL and status.\n\n## If you run one of these today\n\n### LiteLLM\n\n- Run Redis and set `REDIS_HOST` and`REDIS_PORT` , so spend counters survive a proxy restart and aren't limited by the in-memory cache.\n- Set `general_settings: use_redis_transaction_buffer: true` , so spend writes that fail during a database outage are retried instead of dropped.\n- Turn on persistence in Redis itself, since it's now part of your accounting.\n- Set `general_settings: fail_closed_budget_enforcement: true` if you'd rather reject a call than let it run past the budget.\n- Handle HTTP 422 as a budget block in your app. LiteLLM returns 422 \"ExceededBudget\", not 429, so retry logic built for rate limits won't catch it.\n- Always set `max_tokens` . LiteLLM's reservation uses it as the worst case.\n\n### Revenium\n\n- Use Revenium Python SDK 0.12.1 or later, which addresses the enforcement URL. On 0.12.0, set `REVENIUM_ENFORCEMENT_BASE_URL=https://api.revenium.ai/profitstream` .\n- After creating a rule, turn off shadow mode and confirm the summary says \"Block (live)\".\n- Check that blocking actually works with a throwaway customer and a tiny budget before relying on it.\n- Consider `REVENIUM_CB_FAIL_MODE=closed` while testing, so a missing or unreachable rules endpoint blocks calls instead of allowing them.\n- Lower `REVENIUM_CB_POLL_INTERVAL_SECONDS` if overshoot matters more than refresh traffic. At 10 seconds our overshoot fell from 117% to 14.5%.\n- Set `REVENIUM_SHUTDOWN_TIMEOUT_SECONDS` so buffered usage has time to send when a process exits.\n- Treat Revenium's limits as soft for fast-moving agents or bursts, and add a cap of your own, such as a per-run call or token limit, where that matters.\n\n## Caveats\n\n- **Tiny budgets inflate percentages.** A $0.05 budget makes a few cents of overshoot look dramatic. For Revenium, the dollar figure, roughly spending rate times refresh interval, is the one to carry over to your own setup.\n- **Mostly single runs.** Each scenario ran once or twice on one machine with one model. Treat the numbers as directional.\n- **Specific versions.** Results are for LiteLLM v1.106.6 and Revenium Python SDK 0.12.0. Revenium's SDK 0.12.1 fixes the address issue described above; the rest of the Revenium behavior we describe is unchanged in 0.12.1. Both projects ship often, and other details may change.\n- **One way to cut a connection.** Our first Revenium outage test changed the SDK's address, which wiped its rule cache in a way a real outage wouldn't. We reran it by routing traffic through a proxy we could switch off, and report only the rerun.\n- **We told the vendors first.** We ran these tests on October 29, 2026, and reported the SDK 0.12.0 setup issue to Revenium on October 30, with steps to reproduce and logs. Their support team replied the same day, and the summary of their response above is in words they approved. They shipped a fix in SDK 0.12.1 the next day.\n- **We're not neutral.** We're exploring a product in this space, described below. The harness is[available on GitHub](https://github.com/petreli12/stipend-check) , and the raw results are available on request.\n\n## Check your own setup\n\nWe turned these tests into a free, open-source tool, **stipend-check**. Point it at your LiteLLM proxy and in about two minutes it reports which safe settings are off and runs a small live test with a throwaway customer, capped at five cents. For Revenium it checks your settings and that the SDK can actually read your rules, with no spend. [Get stipend-check on GitHub](https://github.com/petreli12/stipend-check).\n\n## What we're building\n\nThe tests left us with a bigger question. Enforcement is the last step; before it, a company billing customers for AI usage needs to know what each customer actually costs, which customers are unprofitable or abusing a free tier, and whether its pricing makes sense. That's what we're working on, under the name Stipend.\n\nIf you sell AI usage to your customers and any of this sounds familiar, we'd like to hear how you handle it today. We're looking for a few design partners to try early versions and shape what we build.", "url": "https://wpnews.pro/news/we-stress-tested-ai-spend-limits-under-bursts-and-outages", "canonical_source": "https://stipend-ai.com/blog/ai-spend-limits-tested/", "published_at": "2026-11-06 18:10:14+00:00", "updated_at": "2026-11-06 18:20:10.607639+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "large-language-models", "mlops"], "entities": ["LiteLLM", "Revenium", "OpenAI", "gpt-4o-mini", "Postgres", "Redis"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-stress-tested-ai-spend-limits-under-bursts-and-outages", "markdown": "https://wpnews.pro/news/we-stress-tested-ai-spend-limits-under-bursts-and-outages.md", "text": "https://wpnews.pro/news/we-stress-tested-ai-spend-limits-under-bursts-and-outages.txt", "jsonld": "https://wpnews.pro/news/we-stress-tested-ai-spend-limits-under-bursts-and-outages.jsonld"}}