{"slug": "your-fallback-provider-is-the-thing-that-breaks-first", "title": "Your fallback provider is the thing that breaks first", "summary": "An engineer argues that LLM fallback providers fail first during outages because failover traffic arrives as a correlated burst rather than a trickle. The writeup offers a burst/peak sizing formula and three mitigations: honoring Retry-After with jittered exponential backoff, reserving headroom or lower concurrency ceilings on the secondary, and instrumenting per-provider failover rate rather than error rate alone.", "body_md": "Anyone running an LLM feature for more than a few months has a fallback path: a second provider, a router, or a forty-line wrapper that tries the next model when the first one returns `529` or `503`. The design work usually stops at \"if the primary fails, use the secondary.\" That is the easy half.\n\nThe hard half is what happens to the secondary at the exact moment the primary trips.\n\nWhen the primary starts returning overload errors, every client fails over within the same few seconds. The secondary does not receive a trickle; it receives its normal traffic plus the entire primary's traffic, arriving as one burst, with retry loops attached.\n\nIf the secondary was sized for its own load plus a small margin, it was sized for a number that never occurs during an incident. The result is a familiar pattern: the primary degrades, the fallback immediately degrades, and the incident looks like \"both providers are down\" when in fact the second one is being DDoS'd by its own customer.\n\nTwo numbers make this concrete:\n\n```\nburst = normal_secondary_load + primary_load x failover_fraction\npeak  = burst x (1 + retry_multiplier)\n```\n\nWith a secondary at 30% of primary capacity and a 2x retry multiplier, the fallback sees roughly 2.6x what it handles on a good day, in the first minute.\n\n**1. Treat `Retry-After` as authoritative, and jitter everything else.** `503` and `529` carry the header inconsistently, so read it when present, and when absent use exponential backoff *with jitter*. Unjittered backoff turns a synchronised burst into a synchronised second burst, one interval later.\n\n**2. Reserve headroom on the secondary instead of treating it as free capacity.** Either keep it deliberately under-subscribed, or give fallback traffic a lower concurrency ceiling than primary traffic so it cannot consume the whole pool. A fallback you can oversubscribe is a shared-fate design, not a resilience one.\n\n**3. Instrument failover rate per provider, not just error rate.** Error rate tells you what is failing now. Failover rate tells you how much of your traffic has already moved, which is the number that predicts whether the *fallback* is about to become the incident. In my experience it leads the user-visible symptom by minutes, which is the only window where you can act.\n\nA fallback is not a spare tire. It is a second system that has to absorb a correlated spike at exactly the moment its sibling is unhealthy. Designing it as identical-but-idle guarantees it will be the first thing to fall over, and it will do so while every dashboard still shows the primary as the problem.\n\nThe cheap version of all this is a single line in the runbook: *when the primary trips, cap fallback concurrency before you touch anything else.* The expensive version is discovering it during the incident.", "url": "https://wpnews.pro/news/your-fallback-provider-is-the-thing-that-breaks-first", "canonical_source": "https://dev.to/aiapikey/your-fallback-provider-is-the-thing-that-breaks-first-5149", "published_at": "2026-11-13 21:43:51+00:00", "updated_at": "2026-11-13 21:53:18.776340+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "large-language-models", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-fallback-provider-is-the-thing-that-breaks-first", "markdown": "https://wpnews.pro/news/your-fallback-provider-is-the-thing-that-breaks-first.md", "text": "https://wpnews.pro/news/your-fallback-provider-is-the-thing-that-breaks-first.txt", "jsonld": "https://wpnews.pro/news/your-fallback-provider-is-the-thing-that-breaks-first.jsonld"}}