Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

A developer built SentioGate, a Python LLM router that normalizes provider rate-limit headers into a shared CapacityState, scores candidates on remaining capacity, and treats 429 responses as circuit-breaker events rather than simple retries. The router combines server-supplied Retry-After values with exponential backoff using max() rather than addition, and enforces a single wall-clock deadline across all retries so a failing provider cannot hang the application.

Most "LLM router" projects treat rate limits as an afterthought. They don't. This post walks through the pieces that make a router actually survive a provider's 429s in production. If you've ever had your app die because OpenAI hit a 429 Too Many Requests mid-request and your only fallback was "try again in 1 second in a tight loop," you know the pain. That pattern burns you twice: you hammer the provider faster, and you have no idea how long to actually wait. The fix isn't a longer sleep. It's reading the provider's own headers and building a router that's capacity-aware . That's the core engineering problem this post tackles, using SentioGate https://sentiogate.app as the reference implementation. A rate-limit-aware LLM router is three things glued together: Retry-After , x-ratelimit-limit- , x-ratelimit-remaining- from each response Each one is small. Together they're the difference between "occasionally 429s" and "never surfaces a 429 to the user." Providers are inconsistent. OpenAI sends x-ratelimit-limit-tokens , x-ratelimit-remaining-tokens , retry-after . Anthropic sends anthropic-ratelimit-requests-remaining , anthropic-ratelimit-requests-reset . Gemini sends retry-after + its own quota headers. Groq, Mistral, DeepSeek — each has its own flavor. Your router's provider adapter layer normalizes them: @dataclass class CapacityState: remaining requests: int requests left in current window reset at: datetime when the window refreshes remaining tokens: int | None if the provider reports token quotas reset tokens at: datetime | None is rate limited: bool last response was a 429 Each provider adapter's job is to fill in one of these from its response. The router core never knows which provider it's talking to — it only sees CapacityState . The router's scoring layer already ranks candidates on cost, latency, reliability, quality, and — crucially — capacity . The capacity score is derived from CapacityState : is rate limited = True This is where most naive implementations go wrong. They treat a 429 as "retry in 1s" instead of "this provider is unavailable until time T, rank it last until then." The circuit-breaker framing is the difference between a router that survives and one that just shuffles the same failing candidate around. When the router must retry you're out of alternatives, or the failure was transient , it uses the server's Retry-After as the base wait: python def next wait retry after s: int, attempt: int, backoff base: float = 0.5 : Respect the server; floor at the exponential backoff so you don't trust a 0 from a flaky provider. server wait = max retry after s, 0 backoff wait = backoff base 2 attempt - 1 return max server wait, backoff wait Key detail: max , not + . If the server says "retry in 30s" and your backoff says "wait 2s," you wait 30s. If the server says "retry in 0" a bug or a 500, not a 429 , your backoff is the floor. You never blind-loop. The router caps total retries max retries and enforces a wall-clock deadline on the whole request, so a pathological provider can't hang your app. This is the part people forget. Retries + backoffs add up. A request that's "30s timeout, 2 retries, 0.5s base backoff" can actually run for over 70s before giving up. Your front-end is long gone. So the router carries a single wall-clock deadline for the entire logical request all retries included . Each candidate attempt gets min candidate timeout, time until deadline . When the deadline passes, the router stops and returns what it has or a structured error — it never runs past the deadline. This is what "hardened transport" means in the intro post https://dev.to/sentiogate/stop-juggling-7-llm-api-keys-one-byok-router-that-ranks-streams-compacts-for-you-41e . With all three pieces in, the router's behavior under pressure is: model: "auto" That last line is the goal: the user never knows a provider failed. The router absorbs it. Here's the skeleton abridged of the core routing loop you'd build around these pieces: python def route request, providers, deadline : scored = rank providers, request cost+latency+reliability+quality+capacity for attempt, provider in enumerate scored : if time.time deadline: return DeadlineExceeded if provider.capacity.is rate limited: continue circuit breaker: skip for now try: return provider.invoke request, timeout=min provider.timeout, deadline - time.time except TransientError as e: update capacity provider, e.headers decode Retry-After / x-ratelimit if attempt + 1 = max retries: raise time.sleep next wait e.retry after s, attempt + 1 return NoProviderAvailable The whole system is ~200 lines once the provider adapters exist. The adapters are the real work — and they're the part SentioGate's SDKs https://www.npmjs.com/package/@hushamdev/ai-auto-router-sdk handle for you. httpx retries? httpx / requests retry middleware is connection-level — it retries the same URL with the same body. It has no idea that a 429 means "this provider is capped, rank another one." A router is decision-level — it picks a different provider on the next attempt. That's the entire value proposition, and it's why a plain retry loop isn't enough. This is the engineering post of the series. The companion posts: intro https://dev.to/sentiogate/stop-juggling-7-llm-api-keys-one-byok-router-that-ranks-streams-compacts-for-you-41e , semantic compaction https://dev.to/sentiogate/how-i-cut-long-chat-token-bills-by-85-95-with-semantic-co , and custom providers https://dev.to/sentiogate/ollama-vllm-private-llms-routing-self-hosted-models . Tell me below what I should cover next.