In my last post, we talked about key cache invalidation — the silent production killer that turns your gateway into a 502 factory. Today I want to talk about something equally dangerous but far more insidious: cost traps.
These aren't bugs. They're not crashes. Your gateway runs fine. Your users are happy. Then finance sends you a Slack message: "Why did our OpenAI bill jump 4x last month?"
I've been running LiteLLM Proxy in production for multiple teams across three companies. Here are the five cost traps I've personally been burned by — each with the config that would have saved me thousands of dollars.
num_retries=3
Actually Means 15 LiteLLM's retry logic is smart. Too smart. When a request fails, it retries. When the retried request hits a fallback model and that fails, it retries again. If you've configured a fallback chain of 5 models with 3 retries each, a single user request can trigger up to 15 upstream API calls — and you pay for every single one, including the ones that errored out after consuming tokens.
The default num_retries
in LiteLLM is 3. Most teams set it and forget it. But retries multiply across your fallback chain. Here's the math:
Request → Model A fails → Retry A (1) → Retry A (2) → Retry A (3)
→ Fallback to Model B → Fails → Retry B (1) → Retry B (2) → Retry B (3)
→ Fallback to Model C → Fails → Retry C (1) → Retry C (2) → Retry C (3)
Total upstream calls: 9 retries + 3 initial = 12 billable calls for 1 user request
If Model C is GPT-4o and each retry consumes 2K input tokens before timing out, that's 24K tokens on a single failed request.
Cap total attempts across the entire chain, not just per-model:
litellm_settings:
num_retries: 2 # per-model retries
max_fallbacks: 2 # hard cap on fallback chain depth
retry_after: 5 # respect 429 Retry-After headers
allowed_fails: 3 # circuit breaker: after 3 fails, stop entirely
model_list:
- model_name: gpt-4o
litellm_params:
model: gpt-4o
max_retries: 2 # override: fewer retries on expensive models
- model_name: gpt-4o-fallback
litellm_params:
model: gpt-4o-mini # cheap fallback, not another expensive model
The key insight: your fallback should be cheaper than your primary, not equally expensive. If GPT-4o fails, fall back to GPT-4o-mini, not to Claude Opus.
This is the trap that cost me $2,300 in a single weekend. A well-meaning engineer configured a fallback chain that looked like this:
router_settings:
fallbacks:
- "gpt-4o-mini": ["gpt-4o"]
- "gpt-4o": ["claude-3-5-sonnet"]
- "claude-3-5-sonnet": ["claude-3-opus"]
The logic seemed sound: "If the cheap model fails, try the better one." But here's what actually happened: GPT-4o-mini was rate-limited during a traffic spike (429s everywhere), so every single request fell through to GPT-4o and then to Claude 3.5 Sonnet. For 6 hours, we were running 100% of our traffic on the most expensive models in the chain.
Rate limits are per-model, not per-gateway. When you hit OpenAI's TPM limit on gpt-4o-mini
, LiteLLM dutifully falls back. But if the traffic spike is caused by overall volume (not a model-specific outage), the fallback model gets the same volume that caused the 429 in the first place. You're not solving the problem — you're just paying 10x more to have it on a different model.
Structure fallbacks by cost tier, not by capability tier:
router_settings:
fallbacks:
- "gpt-4o-mini": ["gemini-1.5-flash", "claude-3-haiku"]
- "gpt-4o": ["claude-3-5-sonnet", "gemini-1.5-pro"]
cooldown_time: 60
Also add alerting. If your fallback rate exceeds 5% of total traffic, something is structurally wrong:
from litellm.integrations.custom_logger import CustomLogger
import litellm
class FallbackAlertLogger(CustomLogger):
def log_pre_api_call(self, model, messages, kwargs):
if kwargs.get("metadata", {}).get("fallback_idx", 0) > 0:
self.fallback_counter.inc()
if self.fallback_counter._value.get() / self.total_counter._value.get() > 0.05:
self.alert_webhook.send(
"⚠️ Fallback rate >5% — check rate limits on primary models"
)
Most teams don't enable LiteLLM's built-in caching because "our prompts are dynamic." But in practice, a huge percentage of your traffic is near-identical: system prompts are the same, the first 500 tokens of user messages are often boilerplate, and many users ask the exact same questions.
I audited one team's traffic and found that 34% of their requests were exact duplicates of requests made in the last hour. They were paying OpenAI ~$400/day for identical completions.
LiteLLM has Redis caching built in. But it's disabled by default, and the documentation buries it under "Advanced Settings." Most engineers set up the proxy, test it, ship it, and never circle back.
Enable Redis caching with a sensible TTL. This is a 30-second config change that can cut your bill by 30-50%:
litellm_settings:
cache: true
cache_params:
type: "redis"
host: "your-redis-host"
port: 6379
namespace: "litellm_cache"
ttl: 3600 # 1 hour for exact matches
cache_key_include_models: true # don't share cache across models
model_list:
- model_name: gpt-4o
litellm_params:
model: gpt-4o
cache: true # enable per-model
For even bigger savings, use prompt caching with providers that support it (Claude, GPT-4o). LiteLLM supports this natively:
import litellm
response = litellm.completion(
model="claude-3-5-sonnet",
messages=[
{"role": "user", "content": [
{"type": "text", "text": "<long_system_prompt>", "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": user_input}
]}
]
)
Real numbers from my audit: After enabling Redis cache with a 1-hour TTL, that team went from $400/day to $180/day. A 55% reduction for a config change that took less than a minute.
An intern pushes a while True
loop to a staging environment. It doesn't crash — it just calls your gateway 4,000 times per minute with a 4K-token prompt. By the time PagerDuty fires, you've spent $847 in 12 minutes.
This isn't hypothetical. This is a Tuesday.
LiteLLM's default configuration has no budget enforcement. The max_budget
field exists but most teams never configure it because they're focused on getting the gateway working, not on constraining it.
Set budgets at three levels: per-key, per-team, and global:
litellm_settings:
max_budget: 500 # $500/day global cap
budget_duration: "1d"
rpm_limit: 1000 # requests per minute, global
general_settings:
master_key: sk-1234
database_url: "postgresql://..."
alerting: ["slack"]
alerting_threshold: 0.8 # alert at 80% of budget
When creating virtual keys for teams or individual developers:
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"max_budget": 50,
"budget_duration": "1d",
"rpm_limit": 100,
"tpm_limit": 50000,
"models": ["gpt-4o-mini", "gpt-4o"],
"metadata": {"team": "frontend"}
}'
And set up a webhook to catch budget breaches:
litellm_settings:
proxy_budget_respecting_alerting:
- webhook_url: "https://hooks.slack.com/services/..."
The intern's loop? With a $50/day key budget and 100 RPM limit, it would have been throttled after 100 calls and blocked entirely after $50. Total damage: about $0.80.
Streaming mode (stream=True
) is great for UX. Users see tokens appear in real-time. But here's what most teams don't realize: when a streaming request is interrupted mid-stream, you still pay for the entire generation.
User starts a request → GPT-4o begins streaming a 2,000-token response → user navigates away after 50 tokens → the client connection drops → but the upstream API call completes fully → you pay for all 2,000 tokens.
At scale, this is devastating. I've seen teams where 23% of their token spend was on tokens that no user ever saw because the client disconnected early.
LiteLLM (and most API gateways) doesn't automatically cancel the upstream request when the client disconnects during streaming. The gateway is acting as a proxy — it's happily receiving tokens from OpenAI and trying to forward them, even though nobody's listening.
Enable client disconnect detection and upstream cancellation:
litellm_settings:
stream_options:
include_usage: true # get token counts in the final chunk
callbacks: stream_cost_logger
router_settings:
streaming_client_disconnect: true # LiteLLM 1.40+
If you're on an older version or need more control, add a custom middleware:
from litellm.proxy.custom_proxy_admin_logic import CustomProxyAdminLogic
class StreamCancellationMiddleware(CustomProxyAdminLogic):
async def async_pre_call(self, user_api_key_dict, cache, data, call_type):
if data.get("stream"):
data["metadata"] = data.get("metadata", {})
data["metadata"]["stream_start_time"] = time.time()
return data
async def async_log_stream_event(self, logging_obj, response, start_time, end_time):
if hasattr(response, 'usage'):
total_tokens = response.usage.get('completion_tokens', 0)
if logging_obj.stream_connection_broken:
self.metrics.abandoned_stream_tokens.inc(total_tokens)
self.alert(
f"Abandoned stream: {total_tokens} tokens paid but undelivered"
)
Also, consider setting max_tokens
conservatively for streaming endpoints:
model_list:
- model_name: gpt-4o-stream
litellm_params:
model: gpt-4o
max_tokens: 1000 # cap generation length
stream: true
stream_options:
include_usage: true
After implementing stream cancellation, that 23% wasted spend dropped to under 2%.
Notice the theme: every one of these traps is a sensible default that becomes dangerous at scale. Retries are good — until they multiply across fallbacks. Fallbacks are good — until they funnel traffic to premium models. Caching is optional — until it's costing you 30% of your bill.
The fix is never "disable the feature." It's always "add constraints." Budgets, caps, cooldowns, TTLs. The gateway works for you, not the other way around.
If you're deploying LiteLLM or any AI API gateway, do a quick audit:
If any of these questions made you nervous, you might want to check out the ** AI API Gateway Pitfall Map** — a one-page production survival guide I put together that covers these traps (and a few more) in a format you can print and pin above your desk. It's the checklist I wish I'd had before I learned these lessons the expensive way.
Have you hit any of these traps in production? Or found others I missed? Drop a comment — I'm collecting war stories for a follow-up post.
Tags: #litellm #ai #devops #costoptimization
Before you ship, run through this 43-point checklist covering auth, cost control, caching, fallbacks, security, monitoring, and production readiness. It's free — grab it here:
👉 Free Pre-Deployment Checklist (PDF)
And if you want the full pitfall map with detailed fixes for each trap above, that's here: AI API Gateway Pitfall Map ($9)