Fixing "TooManyRequests" From Azure OpenAI Under Load

An engineer explains that HTTP 429 errors from Azure OpenAI represent four distinct problems sharing one status code, and that response headers can distinguish them in about a line of code. The post details how to differentiate allocation limits from Azure capacity issues, the impact of max_tokens and short RPM windows, and the importance of reading the correct retry-after headers. It also highlights that quota increases may not resolve throttling if the TPM is not allocated to the specific deployment.

HTTP 429 from Azure OpenAI is four different problems sharing one status code. Three of them are fixed by backing off and one is not, and the response headers distinguish them in about a line of code. Most teams skip that line and file a quota increase for a condition that would have cleared on its own. The SDK surfaces it as a rate-limit error — openai.RateLimitError in Python, a RequestFailedException with Status == 429 in .NET. The message text is the first discriminator, and Microsoft documents the indicator phrases rather than a single fixed string: "Requests to … have been limited" or "Rate limit is exceeded" "The service is temporarily unable to process your request" or "System is experiencing high demand" Those two groups mean opposite things. The first is your allocation; the second is Azure’s capacity. Log the message body on every 429 — without it you are guessing. Microsoft, Manage Azure OpenAI quota https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota . max tokens and the prompt estimate, not the tokens actually generated. A request with a large max tokens spends that budget whether or not it uses it.Two more mechanics explain 429s that look impossible. RPM is enforced over short windows — Microsoft documents evaluation typically at 1 or 10 seconds — so a 600-RPM deployment throttles if more than about 10 requests arrive in one second, even though the minute total is fine. And unsuccessful requests still count toward your per-minute rate limit, which is why an aggressive retry loop makes throttling worse rather than better. Azure OpenAI returns rate-limit information on every call. The documented headers: x-ratelimit-limit-requests e.g. 60 requests/min for this deployment x-ratelimit-limit-tokens e.g. 150000 tokens/min for this deployment x-ratelimit-remaining-requests e.g. 59 x-ratelimit-remaining-tokens e.g. 149984 x-ratelimit-reset-requests e.g. 10 until the request limit resets x-ratelimit-reset-tokens e.g. 300 until the token limit resets retry-after-ms e.g. 2000 on 429s: recommended wait, in ms Note the unit on the last one. The header Microsoft documents for Azure OpenAI is retry-after-ms and its value is milliseconds . Client code written against the more familiar seconds-valued Retry-After convention will sleep 2,000 seconds where it meant to sleep two, or two milliseconds where it meant two seconds — read whichever header is present and convert explicitly rather than assuming. And here is the diagnostic that resolves cause three. Compare x-ratelimit-limit-tokens against the TPM you configured on the deployment. If the header is lower , a temporary protective reduction is active — that is the documented way to detect it. Retry with backoff and wait it out; a quota request will not shorten it. Monitoring x-ratelimit-remaining-tokens in normal operation lets you throttle before you are throttled, which is strictly better than discovering the limit by hitting it. Microsoft’s guidance is explicit on what each signal is for: use token usage metrics in Azure Monitor to understand billed consumption, and use the HTTP status codes and these response headers to detect and respond to rate-limit enforcement in real time. They will not reconcile, and expecting them to is the root of most of the confusion in this area. One more documented asymmetry explains 429s that survive a quota increase. Approved quota is a subscription-and-region pool; the rate limit that rejected your request is the TPM assigned to the specific deployment receiving traffic. Quota approved but never allocated changes nothing. TPM moves between deployments of the same model freely, in increments of 1,000, so rebalancing from an idle deployment is usually faster than any request form. The simplest correct answer is the SDK’s own retry. The OpenAI Python SDK from v1.0 has built-in automatic retry with exponential backoff for 429 and transient errors, and Microsoft documents the default as two retries: python from openai import AzureOpenAI client = AzureOpenAI azure endpoint="https://my-aoai.openai.azure.com/", api key="<key ", api version="2024-10-21", max retries=5, default is 2 If you need custom behaviour — logging, a circuit breaker, selective handling — reach for a retry library, and then heed the documented warning that catches almost everybody: client = AzureOpenAI azure endpoint="https://my-aoai.openai.azure.com/", api key="<key ", api version="2024-10-21", max retries=0, REQUIRED when wrapping with tenacity/Polly @retry wait=wait random exponential min=1, max=60 , stop=stop after attempt 6 , retry=retry if exception type openai.RateLimitError , reraise=True, def chat with backoff kwargs : return client.chat.completions.create kwargs Without max retries=0 , each of your six outer attempts triggers up to two more SDK retries underneath. Six becomes eighteen, every one of them counts against the per-minute limit, and you have built an amplifier for the condition you were trying to survive. The rest of the documented backoff recipe: honour retry-after-ms where present, otherwise exponential backoff with random jitter so clients do not resynchronise, and always a maximum attempt count. Before adding retries, remove the cause: max tokens to the smallest value that serves the case. best of to 1 unless you truly need it — each increment multiplies the counted tokens.Spreading one workload across two deployments is straightforward until you look at the details: each deployment has its own key or its own token audience, its own rate-limit header values, and possibly its own model version. Something has to hold the per-deployment budget, read the headers from whichever one answered, and decide where the next request goes. That routing and normalisation layer is what an LLM gateway is; building it inside your application means every service that calls a model gets its own copy of the logic and its own bugs in it. Header names, the SDK retry defaults and the documented root-cause taxonomy are Microsoft’s current published guidance. Verify against the quota management article before you encode a header name.