{"slug": "fixing-toomanyrequests-from-azure-openai-under-load", "title": "Fixing \"TooManyRequests\" From Azure OpenAI Under Load", "summary": "An engineer explains that HTTP 429 errors from Azure OpenAI represent four distinct problems sharing one status code, and that response headers can distinguish them in about a line of code. The post details how to differentiate allocation limits from Azure capacity issues, the impact of max_tokens and short RPM windows, and the importance of reading the correct retry-after headers. It also highlights that quota increases may not resolve throttling if the TPM is not allocated to the specific deployment.", "body_md": "HTTP 429 from Azure OpenAI is four different problems sharing one status code. Three of them are fixed by backing off and one is not, and the response headers distinguish them in about a line of code. Most teams skip that line and file a quota increase for a condition that would have cleared on its own.\n\nThe SDK surfaces it as a rate-limit error — `openai.RateLimitError`\n\nin Python, a `RequestFailedException`\n\nwith `Status == 429`\n\nin .NET. The message text is the first discriminator, and Microsoft documents the indicator phrases rather than a single fixed string:\n\n`\"Requests to … have been limited\"`\n\nor `\"Rate limit is exceeded\"`\n\n`\"The service is temporarily unable to process your request\"`\n\nor `\"System is experiencing high demand\"`\n\nThose two groups mean opposite things. The first is your allocation; the second is Azure’s capacity. Log the message body on every 429 — without it you are guessing. [Microsoft, Manage Azure OpenAI quota](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota).\n\n`max_tokens`\n\nand the prompt estimate, not the tokens actually generated. A request with a large `max_tokens`\n\nspends that budget whether or not it uses it.Two more mechanics explain 429s that look impossible. RPM is enforced over short windows — Microsoft documents evaluation typically at 1 or 10 seconds — so a 600-RPM deployment throttles if more than about 10 requests arrive in one second, even though the minute total is fine. And unsuccessful requests still count toward your per-minute rate limit, which is why an aggressive retry loop makes throttling worse rather than better.\n\nAzure OpenAI returns rate-limit information on every call. The documented headers:\n\n```\nx-ratelimit-limit-requests      # e.g. 60      requests/min for this deployment\nx-ratelimit-limit-tokens        # e.g. 150000  tokens/min for this deployment\nx-ratelimit-remaining-requests  # e.g. 59\nx-ratelimit-remaining-tokens    # e.g. 149984\nx-ratelimit-reset-requests      # e.g. 10      until the request limit resets\nx-ratelimit-reset-tokens        # e.g. 300     until the token limit resets\nretry-after-ms                  # e.g. 2000    on 429s: recommended wait, in ms\n```\n\nNote the unit on the last one. The header Microsoft documents for Azure OpenAI is `retry-after-ms`\n\nand its value is **milliseconds**. Client code written against the more familiar seconds-valued `Retry-After`\n\nconvention will sleep 2,000 seconds where it meant to sleep two, or two milliseconds where it meant two seconds — read whichever header is present and convert explicitly rather than assuming.\n\nAnd here is the diagnostic that resolves cause three. Compare `x-ratelimit-limit-tokens`\n\nagainst the TPM you configured on the deployment. If the header is *lower*, a temporary protective reduction is active — that is the documented way to detect it. Retry with backoff and wait it out; a quota request will not shorten it.\n\nMonitoring `x-ratelimit-remaining-tokens`\n\nin normal operation lets you throttle before you are throttled, which is strictly better than discovering the limit by hitting it. Microsoft’s guidance is explicit on what each signal is for: use token usage metrics in Azure Monitor to understand billed consumption, and use the HTTP status codes and these response headers to detect and respond to rate-limit enforcement in real time. They will not reconcile, and expecting them to is the root of most of the confusion in this area.\n\nOne more documented asymmetry explains 429s that survive a quota increase. Approved quota is a subscription-and-region pool; the rate limit that rejected your request is the TPM assigned to the specific deployment receiving traffic. Quota approved but never allocated changes nothing. TPM moves between deployments of the same model freely, in increments of 1,000, so rebalancing from an idle deployment is usually faster than any request form.\n\nThe simplest correct answer is the SDK’s own retry. The OpenAI Python SDK from v1.0 has built-in automatic retry with exponential backoff for 429 and transient errors, and Microsoft documents the default as two retries:\n\n``` python\nfrom openai import AzureOpenAI\n\nclient = AzureOpenAI(\n    azure_endpoint=\"https://my-aoai.openai.azure.com/\",\n    api_key=\"<key>\",\n    api_version=\"2024-10-21\",\n    max_retries=5,   # default is 2\n)\n```\n\nIf you need custom behaviour — logging, a circuit breaker, selective handling — reach for a retry library, and then heed the documented warning that catches almost everybody:\n\n```\nclient = AzureOpenAI(\n    azure_endpoint=\"https://my-aoai.openai.azure.com/\",\n    api_key=\"<key>\",\n    api_version=\"2024-10-21\",\n    max_retries=0,   # REQUIRED when wrapping with tenacity/Polly\n)\n\n@retry(\n    wait=wait_random_exponential(min=1, max=60),\n    stop=stop_after_attempt(6),\n    retry=retry_if_exception_type(openai.RateLimitError),\n    reraise=True,\n)\ndef chat_with_backoff(**kwargs):\n    return client.chat.completions.create(**kwargs)\n```\n\nWithout `max_retries=0`\n\n, each of your six outer attempts triggers up to two more SDK retries underneath. Six becomes eighteen, every one of them counts against the per-minute limit, and you have built an amplifier for the condition you were trying to survive.\n\nThe rest of the documented backoff recipe: honour `retry-after-ms`\n\nwhere present, otherwise exponential backoff with random jitter so clients do not resynchronise, and always a maximum attempt count.\n\nBefore adding retries, remove the cause:\n\n`max_tokens`\n\nto the smallest value that serves the case.`best_of`\n\nto 1 unless you truly need it — each increment multiplies the counted tokens.Spreading one workload across two deployments is straightforward until you look at the details: each deployment has its own key or its own token audience, its own rate-limit header values, and possibly its own model version. Something has to hold the per-deployment budget, read the headers from whichever one answered, and decide where the next request goes. That routing and normalisation layer is what an LLM gateway is; building it inside your application means every service that calls a model gets its own copy of the logic and its own bugs in it.\n\nHeader names, the SDK retry defaults and the documented root-cause taxonomy are Microsoft’s current published guidance. Verify against the quota management article before you encode a header name.", "url": "https://wpnews.pro/news/fixing-toomanyrequests-from-azure-openai-under-load", "canonical_source": "https://dev.to/multigrid/fixing-toomanyrequests-from-azure-openai-under-load-34bn", "published_at": "2026-08-12 21:35:26+00:00", "updated_at": "2026-08-12 21:45:35.827861+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Azure OpenAI", "Microsoft", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/fixing-toomanyrequests-from-azure-openai-under-load", "markdown": "https://wpnews.pro/news/fixing-toomanyrequests-from-azure-openai-under-load.md", "text": "https://wpnews.pro/news/fixing-toomanyrequests-from-azure-openai-under-load.txt", "jsonld": "https://wpnews.pro/news/fixing-toomanyrequests-from-azure-openai-under-load.jsonld"}}