# How to Handle AI API 429 Errors: Retry-After and Backoff in Python

> Source: <https://dev.to/_fde76ff2d01974642930b/how-to-handle-ai-api-429-errors-retry-after-and-backoff-in-python-mn1>
> Published: 2026-11-10 01:33:37+00:00

A batch of 50 AI requests hits a rate limit. Every worker waits exactly one second, then sends the same request again.

The next second brings another batch of 50 requests. If the original queue is still accepting work, the service now has new traffic and retries competing for the same capacity.

This is how a small burst becomes a retry storm.

Handling an AI API 429 error takes more than wrapping a request in a loop. The client needs to identify the kind of rejection, respect the server's timing, and reduce the pressure that caused it.

HTTP 429 means Too Many Requests. It does not tell you which limit you reached, and some APIs also use it for exhausted quota or billing limits.

Read the documented error code and message, along with the response headers. The exact codes vary between providers and gateways; do not classify errors by guessing a substring in an English message.

| What the response indicates | What to do | 
|---|---|
| A temporary request or token rate limit | Wait, reduce the request rate, and retry within a budget | 
| Too many simultaneous generations | Reduce active requests; use a bounded queue | 
| Exhausted credits or a hard account quota | Fix the account or quota; waiting in a loop will not help | 
| An unknown 429 | Keep the diagnostic information and check the API contract before retrying automatically | 

A 401, 403, invalid model name, or malformed request needs its own handling. A generic “retry every error” function hides the actual problem.

When a gateway sits between your application and a model provider, the rejection might originate at either layer. Correlate timestamps and request IDs where available. Do not assume that a 429 proves the model provider rejected the request.

A server can send:

```
HTTP/1.1 429 Too Many Requests
Retry-After: 5
```

Here, wait at least five seconds before another attempt. The header can also contain an HTTP date:

```
Retry-After: Tue, 10 Nov 2026 09:00:00 GMT
```

Your client should understand both formats. Date-based delays also depend on a reasonably synchronized clock.

Treat the server's delay as a minimum. If your local backoff cap is eight seconds but Retry-After asks for 60, retrying after eight seconds ignores the server's instruction. Either wait long enough or stop because the operation's deadline cannot accommodate it.

Provider-specific reset headers sometimes use different units or timestamps. Parse them only according to that provider's documentation; they are not automatically interchangeable with Retry-After.

An exponential schedule might use windows of 0.5, 1, 2, and 4 seconds. Randomizing the delay within each window spreads retries across time.

When Retry-After exists, add that random delay after the required minimum. This avoids sending every worker back at the exact reset boundary.

Here is a small timing helper for Python 3.10+. It makes no network requests and deliberately does not decide whether an error is retryable:

``` python
import math
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_delay(retry_number: int, retry_after: str | None) -> float:
    """retry_number starts at 0; call only for a retryable response."""
    minimum_wait = 0.0
    if retry_after:
        value = retry_after.strip()
        if value.isascii() and value.isdecimal():
            minimum_wait = float(value)
        else:
            try:
                retry_at = parsedate_to_datetime(value)
                if retry_at.tzinfo is None:
                    retry_at = retry_at.replace(tzinfo=timezone.utc)
                minimum_wait = max(
                    0.0,
                    (retry_at - datetime.now(timezone.utc)).total_seconds(),
                )
            except (TypeError, ValueError, OverflowError):
                pass  # Invalid header: use local backoff.

    if not math.isfinite(minimum_wait):
        return math.inf  # Caller should stop rather than sleep forever.

    window = min(8.0, 0.5 * 2 ** min(retry_number, 10))
    return minimum_wait + random.uniform(0.0, window)
```

For example, retry number 2 uses a two-second jitter window. With Retry-After: 5, the resulting delay is between five and seven seconds. Without a valid header, it is between zero and two seconds.

The eight-second cap applies to the local jitter window, not to the server's minimum wait. A malformed header falls back to local backoff; an unreasonably large numeric delay results in infinity so the caller can reject another attempt.

This helper is only the timing part. Put it behind a classifier that recognizes a documented temporary limit. It should never turn a quota error into a retryable error.

A request timeout limits one attempt. It does not necessarily limit the total time spent across attempts and sleeps.

For an interactive request, a starting policy could be:

These numbers are examples, not universal defaults. A background job and a chat message usually need different budgets.

Use a monotonic clock for the elapsed-time budget. Before sleeping, compare the computed delay with the remaining time. If the server asks for a longer wait, return the rate-limit result or reschedule the background job; do not shorten the server's wait to squeeze in one more attempt.

In async Python, use a cancellable async sleep and enforce the overall deadline around the operation. A blocking time.sleep inside an async request handler can stall unrelated work.

Imagine this configuration:

```
Application: up to 3 attempts
    SDK: up to 3 attempts per application attempt
        Gateway: up to 3 upstream attempts per SDK attempt
```

In a worst case where every layer retries the same failure, one logical operation can generate 27 upstream attempts. Real behavior depends on which errors each layer retries, but the multiplication is easy to miss.

Choose which layer owns the end-to-end retry budget. Check SDK defaults and any gateway failover behavior before adding another outer loop. If multiple layers must retry, coordinate their maximum attempts, deadlines, and observability.

A “successful request” metric can also conceal the first two failed attempts. Track both logical operations and physical attempts, then inspect attempts per successful operation and total latency.

A semaphore limits simultaneous requests. A rate limiter controls how often requests start. A bounded queue limits how much work can accumulate. They solve different problems.

Consider a hypothetical limit of 60 requests per minute. Starting 60 requests at once may still violate a shorter burst window. Pacing the starts and keeping some headroom is safer than assuming the minute is an empty bucket you can drain instantly.

Token limits are a separate constraint. A few large prompts can exhaust a token budget even while request count stays low. Use the provider's documented accounting rules, monitor actual usage, and reserve capacity conservatively for work that is about to start.

If the service publishes no capacity limit, begin with low concurrency and increase gradually while measuring queue time, 429 frequency, and end-to-end latency. Do not discover the limit by repeatedly flooding it.

This 429 policy applies to a rejected request with a known temporary limit. A connection timeout or a broken stream is a different situation: the server may already have performed work.

Once a user has received part of a generated answer, silently replaying the entire request can produce duplicate or contradictory output. Report an interrupted stream explicitly, and offer a deliberate restart unless the protocol supports a documented resume mechanism.

Tool executions need their own deduplication. Repeating an agent turn must not send the same email, place the same order, or write the same record twice. If an API supports idempotency keys, follow its documented scope and retention rules; inventing a header does not create idempotency.

Also check whether repeating inference can incur another charge. A client-side timeout does not prove the upstream request was never processed.

For each attempt, record the logical operation ID, attempt number, status, documented error code, chosen delay, remaining deadline, and request ID when available. Add queue time and whether any streamed output had already reached the user.

Redact credentials. Avoid logging complete prompts, generated content, or raw request bodies by default. A useful retry log should explain the failure without becoming a second store of private user data.

Before shipping, exercise a few deterministic cases: a temporary 429 followed by success; a quota error that stops immediately; a Retry-After longer than the deadline; cancellation while waiting; and a stream that disconnects after producing output.

A retry policy is working when temporary failures recover within the user's budget and the system sends fewer unnecessary requests during congestion. Counting eventual successes alone misses half the story.

I work on [YW Code API (ywcode)](https://ywcode.top/), an AI API gateway. These are client-side design considerations for direct provider integrations and gateway integrations alike; a gateway does not remove upstream capacity limits.

Related: [AI API Streaming Not Working? Debug SSE and Nginx Buffering](https://dev.to/_fde76ff2d01974642930b/ai-api-streaming-not-working-debug-sse-and-nginx-buffering-3ina).

HTTP references: [429 Too Many Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429) and [Retry-After](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After).