Surviving the 429 Storm: Building Resilient LLM Fallbacks in Production

How to eliminate cascading failures from LLM rate limits using exponential backoff, circuit breakers, and tiered fallback chains.

Most production LLM integrations start with a direct SDK call wrapped in a naive try/except

block. During quiet periods, it works fine. But when traffic spikes, your application hits provider Token-Per-Minute (TPM) or Request-Per-Minute (RPM) limits, throwing HTTP 429 errors.

The immediate reaction is often an uncontrolled retry loop. That is an anti-pattern:

for _ in range(5):
    try:
        return openai_client.chat.completions.create(...)
    except Exception:
        time.sleep(0.5)  # Thrashes the API and worsens rate limits

When 100 concurrent workers hammer an already-throttled provider with instant retries, you trigger a cascading failure. Your API workers block, thread pools exhaust, upstream quotas remain locked, and raw JSON errors leak to your users.

Rate limits and transient outages are not anomalies; they are environmental constraints. A production-grade backend requires a layered defensive strategy rather than raw retries.

A resilient LLM architecture relies on a 3-layer safety net:

[Incoming User Request]
         │
         ▼
 ┌──────────────┐    HTTP 429/5xx     ┌───────────────────────┐
 │ Primary LLM  │ ──────────────────► │ Exponential Backoff   │
 └───────┬──────┘   (Retries Exhaust) └───────────┬───────────┘
         │                                        │
         │ Success                                ▼
         │                             ┌───────────────────────┐
         │                             │ Secondary / Fast Model│
         │                             └───────────┬───────────┘
         │                                         │
         ▼                                         ▼ Failed
  [Final Response] ◄─────────────────── ┌───────────────────────┐
                                        │ Cache / Static Helper │
                                        └───────────────────────┘

Using tenacity

, we can implement exponential backoff targeting specific HTTP status codes, paired with an automated fallback handler.

Here is a minimal, robust implementation:

import openai
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception

def is_retryable_error(exception: BaseException) -> bool:
    status_code = getattr(exception, "status_code", None)
    return status_code in {429, 500, 502, 503, 504}

def fallback_completion(prompt: str) -> str:
    client = openai.OpenAI()
    fallback = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return fallback.choices[0].message.content

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=8),
    retry=retry_if_exception(is_retryable_error),
    reraise=False,
)
def generate_response(prompt: str) -> str:
    try:
        client = openai.OpenAI()
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
        )
        return response.choices[0].message.content
    except Exception:
        return fallback_completion(prompt)

This pattern isolates failures. If the primary model encounters a 429, it backs off cleanly without monopolizing compute. If the rate limit persists past three attempts, the execution transparently transfers to the fallback model without failing the upstream pipeline.