# Building a Rate-Limit-Aware LLM Router in Python: Retries, Backoff & Capacity Scoring

> Source: <https://dev.to/sentiogate/building-a-rate-limit-aware-llm-router-in-python-retries-backoff-capacity-scoring-37bl>
> Published: 2026-10-20 15:40:34+00:00

Most "LLM router" projects treat rate limits as an afterthought. They don't. This post walks through the pieces that make a router actually *survive* a provider's 429s in production.

If you've ever had your app die because OpenAI hit a `429 Too Many Requests` mid-request and your only fallback was "try again in 1 second in a tight loop," you know the pain. That pattern burns you twice: you hammer the provider faster, and you have no idea *how long* to actually wait.

The fix isn't a longer sleep. It's **reading the provider's own headers** and building a router that's *capacity-aware*. That's the core engineering problem this post tackles, using [**SentioGate**](https://sentiogate.app) as the reference implementation.

A rate-limit-aware LLM router is three things glued together:

`Retry-After`, `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` from each response
Each one is small. Together they're the difference between "occasionally 429s" and "never surfaces a 429 to the user."

Providers are inconsistent. OpenAI sends `x-ratelimit-limit-tokens`, `x-ratelimit-remaining-tokens`, `retry-after`. Anthropic sends `anthropic-ratelimit-requests-remaining`, `anthropic-ratelimit-requests-reset`. Gemini sends `retry-after` + its own quota headers. Groq, Mistral, DeepSeek — each has its own flavor.

Your router's provider adapter layer normalizes them:

```
@dataclass
class CapacityState:
    remaining_requests: int      # requests left in current window
    reset_at: datetime           # when the window refreshes
    remaining_tokens: int | None # if the provider reports token quotas
    reset_tokens_at: datetime | None
    is_rate_limited: bool        # last response was a 429
```

Each provider adapter's job is to fill in one of these from its response. The router core never knows which provider it's talking to — it only sees `CapacityState`.

The router's scoring layer already ranks candidates on cost, latency, reliability, quality, and — crucially — **capacity**. The capacity score is derived from `CapacityState`:

`is_rate_limited = True`
This is where most naive implementations go wrong. They treat a 429 as "retry in 1s" instead of "this provider is *unavailable until time T, rank it last until then." The circuit-breaker framing is the difference between a router that survives and one that just shuffles the same failing candidate around.

When the router *must* retry (you're out of alternatives, or the failure was transient), it uses the server's `Retry-After` as the base wait:

``` python
def next_wait(retry_after_s: int, attempt: int, backoff_base: float = 0.5):
    # Respect the server; floor at the exponential backoff so you don't
    # trust a 0 from a flaky provider.
    server_wait = max(retry_after_s, 0)
    backoff_wait = backoff_base * (2 ** (attempt - 1))
    return max(server_wait, backoff_wait)
```

Key detail: **`max()`, not `+`**. If the server says "retry in 30s" and your backoff says "wait 2s," you wait 30s. If the server says "retry in 0" (a bug or a 500, not a 429), your backoff is the floor. You never blind-loop.

The router caps total retries (`max_retries`) and enforces a wall-clock deadline on the whole request, so a pathological provider can't hang your app.

This is the part people forget. Retries + backoffs add up. A request that's "30s timeout, 2 retries, 0.5s base backoff" can actually run for over 70s before giving up. Your front-end is long gone.

So the router carries a **single wall-clock deadline** for the entire logical request (all retries included). Each candidate attempt gets `min(candidate_timeout, time_until_deadline)`. When the deadline passes, the router stops and returns what it has (or a structured error) — it never runs past the deadline.

This is what "hardened transport" means in [the intro post](https://dev.to/sentiogate/stop-juggling-7-llm-api-keys-one-byok-router-that-ranks-streams-compacts-for-you-41e).

With all three pieces in, the router's behavior under pressure is:

`model: "auto"`
That last line is the goal: **the user never knows a provider failed.** The router absorbs it.

Here's the skeleton (abridged) of the core routing loop you'd build around these pieces:

``` python
def route(request, providers, deadline):
    scored = rank(providers, request)          # cost+latency+reliability+quality+capacity
    for attempt, provider in enumerate(scored):
        if time.time() > deadline:
            return DeadlineExceeded()
        if provider.capacity.is_rate_limited:
            continue                            # circuit breaker: skip for now
        try:
            return provider.invoke(request, timeout=min(provider.timeout, deadline - time.time()))
        except TransientError as e:
            update_capacity(provider, e.headers)  # decode Retry-After / x-ratelimit
            if attempt + 1 >= max_retries:
                raise
            time.sleep(next_wait(e.retry_after_s, attempt + 1))
    return NoProviderAvailable()
```

The whole system is ~200 lines once the provider adapters exist. The adapters are the real work — and they're the part [SentioGate's SDKs](https://www.npmjs.com/package/@hushamdev/ai-auto-router-sdk) handle for you.

`httpx` retries?
`httpx` / `requests` retry middleware is *connection-level* — it retries the same URL with the same body. It has no idea that a 429 means "this provider is capped, rank another one." A router is *decision-level* — it picks a *different provider* on the next attempt. That's the entire value proposition, and it's why a plain retry loop isn't enough.

*This is the engineering post of the series. The companion posts: [intro](https://dev.to/sentiogate/stop-juggling-7-llm-api-keys-one-byok-router-that-ranks-streams-compacts-for-you-41e), [semantic compaction](https://dev.to/sentiogate/how-i-cut-long-chat-token-bills-by-85-95-with-semantic-co), and [custom providers](https://dev.to/sentiogate/ollama-vllm-private-llms-routing-self-hosted-models). Tell me below what I should cover next.*