# The 5 Cost Traps That Will Quietly Bleed Your AI API Gateway Dry (And How to Fix Them) > Source: > Published: 2026-06-22 01:47:21+00:00 In my last post, we talked about key cache invalidation — the silent production killer that turns your gateway into a 502 factory. Today I want to talk about something equally dangerous but far more insidious: **cost traps**. These aren't bugs. They're not crashes. Your gateway runs fine. Your users are happy. Then finance sends you a Slack message: *"Why did our OpenAI bill jump 4x last month?"* I've been running LiteLLM Proxy in production for multiple teams across three companies. Here are the five cost traps I've personally been burned by — each with the config that would have saved me thousands of dollars. `num_retries=3` Actually Means 15 LiteLLM's retry logic is smart. Too smart. When a request fails, it retries. When the retried request hits a fallback model and *that* fails, it retries again. If you've configured a fallback chain of 5 models with 3 retries each, a single user request can trigger up to **15 upstream API calls** — and you pay for every single one, including the ones that errored out after consuming tokens. The default `num_retries` in LiteLLM is 3. Most teams set it and forget it. But retries multiply across your fallback chain. Here's the math: ``` Request → Model A fails → Retry A (1) → Retry A (2) → Retry A (3) → Fallback to Model B → Fails → Retry B (1) → Retry B (2) → Retry B (3) → Fallback to Model C → Fails → Retry C (1) → Retry C (2) → Retry C (3) Total upstream calls: 9 retries + 3 initial = 12 billable calls for 1 user request ``` If Model C is GPT-4o and each retry consumes 2K input tokens before timing out, that's 24K tokens on a *single failed request*. Cap total attempts across the entire chain, not just per-model: ``` # litellm_config.yaml litellm_settings: num_retries: 2 # per-model retries max_fallbacks: 2 # hard cap on fallback chain depth retry_after: 5 # respect 429 Retry-After headers allowed_fails: 3 # circuit breaker: after 3 fails, stop entirely model_list: - model_name: gpt-4o litellm_params: model: gpt-4o max_retries: 2 # override: fewer retries on expensive models - model_name: gpt-4o-fallback litellm_params: model: gpt-4o-mini # cheap fallback, not another expensive model ``` The key insight: **your fallback should be cheaper than your primary**, not equally expensive. If GPT-4o fails, fall back to GPT-4o-mini, not to Claude Opus. This is the trap that cost me $2,300 in a single weekend. A well-meaning engineer configured a fallback chain that looked like this: ``` # THE EXPENSIVE WAY — do not do this router_settings: fallbacks: - "gpt-4o-mini": ["gpt-4o"] - "gpt-4o": ["claude-3-5-sonnet"] - "claude-3-5-sonnet": ["claude-3-opus"] ``` The logic seemed sound: "If the cheap model fails, try the better one." But here's what actually happened: GPT-4o-mini was rate-limited during a traffic spike (429s everywhere), so **every single request fell through to GPT-4o and then to Claude 3.5 Sonnet**. For 6 hours, we were running 100% of our traffic on the most expensive models in the chain. Rate limits are per-model, not per-gateway. When you hit OpenAI's TPM limit on `gpt-4o-mini` , LiteLLM dutifully falls back. But if the traffic spike is caused by overall volume (not a model-specific outage), the fallback model gets the same volume that caused the 429 in the first place. You're not solving the problem — you're just paying 10x more to have it on a different model. Structure fallbacks by **cost tier**, not by capability tier: ``` # THE SMART WAY — fallback within price tier, not up router_settings: fallbacks: # Tier 1: Cheap models (fallback to other cheap models) - "gpt-4o-mini": ["gemini-1.5-flash", "claude-3-haiku"] # Tier 2: Mid-tier models (fallback to other mid-tier) - "gpt-4o": ["claude-3-5-sonnet", "gemini-1.5-pro"] # NEVER fall up from cheap to expensive # If all cheap models fail, return an error, don't escalate # Add a cooldown so the same model isn't retried immediately cooldown_time: 60 ``` Also add alerting. If your fallback rate exceeds 5% of total traffic, something is structurally wrong: ``` # prometheus metric in your LiteLLM custom callback from litellm.integrations.custom_logger import CustomLogger import litellm class FallbackAlertLogger(CustomLogger): def log_pre_api_call(self, model, messages, kwargs): if kwargs.get("metadata", {}).get("fallback_idx", 0) > 0: # This is a fallback call, not the primary self.fallback_counter.inc() # Alert if fallback rate > 5% if self.fallback_counter._value.get() / self.total_counter._value.get() > 0.05: self.alert_webhook.send( "⚠️ Fallback rate >5% — check rate limits on primary models" ) ``` Most teams don't enable LiteLLM's built-in caching because "our prompts are dynamic." But in practice, a huge percentage of your traffic is **near-identical**: system prompts are the same, the first 500 tokens of user messages are often boilerplate, and many users ask the exact same questions. I audited one team's traffic and found that **34% of their requests were exact duplicates** of requests made in the last hour. They were paying OpenAI ~$400/day for identical completions. LiteLLM has Redis caching built in. But it's disabled by default, and the documentation buries it under "Advanced Settings." Most engineers set up the proxy, test it, ship it, and never circle back. Enable Redis caching with a sensible TTL. This is a 30-second config change that can cut your bill by 30-50%: ``` litellm_settings: cache: true cache_params: type: "redis" host: "your-redis-host" port: 6379 namespace: "litellm_cache" # Cache settings ttl: 3600 # 1 hour for exact matches # For semantic caching (similar but not identical prompts): # semantic_cache: true # similarity_threshold: 0.8 # Cache based on messages content, not just the full request cache_key_include_models: true # don't share cache across models model_list: - model_name: gpt-4o litellm_params: model: gpt-4o cache: true # enable per-model ``` For even bigger savings, use **prompt caching** with providers that support it (Claude, GPT-4o). LiteLLM supports this natively: ``` python import litellm # Enable prompt caching for Claude response = litellm.completion( model="claude-3-5-sonnet", messages=[ {"role": "user", "content": [ {"type": "text", "text": "", "cache_control": {"type": "ephemeral"}}, {"type": "text", "text": user_input} ]} ] ) # Claude charges 90% less for cached input tokens ``` **Real numbers from my audit**: After enabling Redis cache with a 1-hour TTL, that team went from $400/day to $180/day. A 55% reduction for a config change that took less than a minute. An intern pushes a `while True` loop to a staging environment. It doesn't crash — it just calls your gateway 4,000 times per minute with a 4K-token prompt. By the time PagerDuty fires, you've spent $847 in 12 minutes. This isn't hypothetical. This is a Tuesday. LiteLLM's default configuration has **no budget enforcement**. The `max_budget` field exists but most teams never configure it because they're focused on getting the gateway working, not on constraining it. Set budgets at three levels: per-key, per-team, and global: ``` # Per-virtual-key budget (when creating keys via /key/generate) # This is your first line of defense litellm_settings: # Global budget — emergency brake max_budget: 500 # $500/day global cap budget_duration: "1d" # Rate limiting rpm_limit: 1000 # requests per minute, global general_settings: master_key: sk-1234 database_url: "postgresql://..." # Enable budget tracking alerting: ["slack"] alerting_threshold: 0.8 # alert at 80% of budget ``` When creating virtual keys for teams or individual developers: ``` # Create a key with a $50 daily budget and 100 RPM curl -X POST http://localhost:4000/key/generate \ -H "Authorization: Bearer sk-1234" \ -H "Content-Type: application/json" \ -d '{ "max_budget": 50, "budget_duration": "1d", "rpm_limit": 100, "tpm_limit": 50000, "models": ["gpt-4o-mini", "gpt-4o"], "metadata": {"team": "frontend"} }' ``` And set up a webhook to catch budget breaches: ``` # In your LiteLLM proxy config litellm_settings: proxy_budget_respecting_alerting: - webhook_url: "https://hooks.slack.com/services/..." # This fires BEFORE the request is sent when a key is over budget # LiteLLM will return a 429 to the client, not forward to the provider ``` The intern's loop? With a $50/day key budget and 100 RPM limit, it would have been throttled after 100 calls and blocked entirely after $50. Total damage: about $0.80. Streaming mode (`stream=True` ) is great for UX. Users see tokens appear in real-time. But here's what most teams don't realize: **when a streaming request is interrupted mid-stream, you still pay for the entire generation.** User starts a request → GPT-4o begins streaming a 2,000-token response → user navigates away after 50 tokens → the client connection drops → but the upstream API call completes fully → you pay for all 2,000 tokens. At scale, this is devastating. I've seen teams where **23% of their token spend was on tokens that no user ever saw** because the client disconnected early. LiteLLM (and most API gateways) doesn't automatically cancel the upstream request when the client disconnects during streaming. The gateway is acting as a proxy — it's happily receiving tokens from OpenAI and trying to forward them, even though nobody's listening. Enable client disconnect detection and upstream cancellation: ``` litellm_settings: # Cancel upstream request when client disconnects during streaming stream_options: include_usage: true # get token counts in the final chunk # Custom callback to track abandoned streams callbacks: stream_cost_logger router_settings: # Close upstream connection when client disconnects streaming_client_disconnect: true # LiteLLM 1.40+ ``` If you're on an older version or need more control, add a custom middleware: ``` python from litellm.proxy.custom_proxy_admin_logic import CustomProxyAdminLogic class StreamCancellationMiddleware(CustomProxyAdminLogic): async def async_pre_call(self, user_api_key_dict, cache, data, call_type): if data.get("stream"): # Mark the start time data["metadata"] = data.get("metadata", {}) data["metadata"]["stream_start_time"] = time.time() return data async def async_log_stream_event(self, logging_obj, response, start_time, end_time): # Log how many tokens were actually consumed vs delivered if hasattr(response, 'usage'): total_tokens = response.usage.get('completion_tokens', 0) # If stream ended early (client disconnect), log it if logging_obj.stream_connection_broken: self.metrics.abandoned_stream_tokens.inc(total_tokens) self.alert( f"Abandoned stream: {total_tokens} tokens paid but undelivered" ) ``` Also, consider setting `max_tokens` conservatively for streaming endpoints: ``` model_list: - model_name: gpt-4o-stream litellm_params: model: gpt-4o max_tokens: 1000 # cap generation length stream: true stream_options: include_usage: true ``` After implementing stream cancellation, that 23% wasted spend dropped to under 2%. Notice the theme: **every one of these traps is a sensible default that becomes dangerous at scale.** Retries are good — until they multiply across fallbacks. Fallbacks are good — until they funnel traffic to premium models. Caching is optional — until it's costing you 30% of your bill. The fix is never "disable the feature." It's always "add constraints." Budgets, caps, cooldowns, TTLs. The gateway works for you, not the other way around. If you're deploying LiteLLM or any AI API gateway, do a quick audit: If any of these questions made you nervous, you might want to check out the ** AI API Gateway Pitfall Map** — a one-page production survival guide I put together that covers these traps (and a few more) in a format you can print and pin above your desk. It's the checklist I wish I'd had before I learned these lessons the expensive way. *Have you hit any of these traps in production? Or found others I missed? Drop a comment — I'm collecting war stories for a follow-up post.* *Tags: #litellm #ai #devops #costoptimization* Before you ship, run through this 43-point checklist covering auth, cost control, caching, fallbacks, security, monitoring, and production readiness. It's free — grab it here: **👉 Free Pre-Deployment Checklist (PDF)** And if you want the full pitfall map with detailed fixes for each trap above, that's here: [AI API Gateway Pitfall Map ($9)](https://payhip.com/b/S96bB)