Round-Robin Broke My Agent

A round-robin load balancer across three Azure OpenAI resources caused a 400 error on the Responses API because the API is stateful and requires requests to hit the same resource that created the items, with one failed call carrying a successful payload. The bug, which also degrades prompt caching hit rates, was identified by grouping gateway telemetry by operation, showing 64% of operations landing in one bucket.

The first sign was a trace that made no sense. One inbound request to the gateway, three outgoing calls to three different Azure OpenAI resources. Two returned 400. The third returned 200. The client got its answer, 19.6 seconds later, and nobody filed a bug — the system was working , just expensively. What made it strange was the body attached to one of those 400s. It wasn’t an error. It was a complete, successful response object: "status": "completed", a created at, a completed at, "error": null. A failed call carrying a successful payload. That contradiction took a day to unwind, and the answer turned out to be a design assumption baked into almost every LLM gateway tutorial on the internet: that an LLM endpoint is a stateless HTTP endpoint, and you can therefore round-robin it. For chat completions, that’s true. For the API you are probably migrating to right now, it isn’t. The instinct behind an LLM load balancer is sound. A single Azure OpenAI resource has a per-deployment quota — tokens per minute, requests per minute. You have three resources. Put a gateway in front, round-robin across them, and you have tripled your ceiling. Every reference architecture shows this, and for the Chat Completions API it works perfectly, because a chat completion is a pure function: you send the entire conversation, you get an answer, the server keeps nothing. The Responses API is not a pure function. It is a stateful API, and the state lives on the resource that served the call. With the default store=true, every item the model produces is persisted server-side and handed back to you with an id: fc … for a function call, rs … for a reasoning item, msg … for a message, resp … for the response itself. Agent frameworks then replay those items as the next turn's input — that is how the model sees its own previous tool calls. The OpenAI Agents SDK does it on every turn of every run. Some code does it explicitly: the next turn carries the previous turn's items — ids and allfollow up = result.to input list + {"role": "user", "content": nudge} await Runner.run agent, follow up, max turns=8 Send those ids to a resource that didn’t mint them, and you get this — the actual response body, worth committing to memory: 400 — “The requested item was created under a different Azure OpenAI resource. Use the same resource that created the item to access it.” There it is. A round-robin pool guarantees that a two-resource-out-of-three miss happens on every turn after the first. The first turn has no ids, so it lands anywhere. Every turn after it is pinned to a resource the load balancer has already forgotten about. A 400 that succeeds when you retry it on a different host is not a transient error. It is a routing bug wearing a client error’s clothes. The useful generalization is not “the Responses API is special.” It’s that every LLM provider now ships a mix of stateless and resource-scoped endpoints, and the resource-scoped ones share a property: the API hands you an opaque id and expects you to hand it back. That last row is the one people miss, because nothing fails. Prompt caching discounts the repeated prefix of a request — the system prompt, the tool schemas, the retrieved context — and that cache lives on the resource that saw the prefix. Round-robin across N resources and a warm prefix has roughly a 1/N chance of landing where it's warm. Your correctness is fine; your cache hit rate quietly drops to a third and your bill doesn't. A load balancer built to save money on quota can cost you more than it saves. This class of bug has a shape you can query for, and the shape is what convinced me before I had a reproduction. Group your gateway telemetry by operation and count attempts: A transient fault — throttling, a bad node, a cold start — produces a binomial spread across those buckets. What I had was 64% of operations landing in exactly one bucket: N−1 failures, then a success, with the failures distributed evenly across all three backends. That is not randomness. That is “only one specific host can serve this request, and the balancer is finding it by exhaustion.” If you can express that query, you can detect this in ten minutes: // per-operation attempt shape — the tell is a spike at N-1 fails, 1 success dependencies| where timestamp ago 2d and target has "openai"| summarize fails=countif resultCode=="400" , oks=countif resultCode=="200" by operation Id| summarize operations=count by fails, oks| order by operations desc Remember that 400 carrying a successful payload? Every one of the 302 failures in my first sample had a body that began "status": "completed", and every body was truncated at exactly 8192 characters. The gateway was capturing the response of the attempt that eventually succeeded and attributing it to each attempt in the retry group. Two facts fell out of that: the diagnostic body limit was hiding the actual error, and a 2.5-second call cannot contain the seven seconds of generation its own log claimed. The fix for that is not a better query. It’s reproducing the call outside the gateway — hit each backend directly, with the same payload, and see what it really says. That took fifteen minutes and ended the speculation. The routing bug wasted two calls per turn. The retry policy in front of it was prepared to waste thirty. Here is what was actually deployed: <retry condition="@ context.Response.StatusCode = 400 " count="30" interval="2" max-interval="60" delta="2" first-fast-retry="false" Read that condition: = 400 already covers everything. This policy retries every client error — a malformed request, a bad api-key, a content-filter block, a nonexistent deployment — up to thirty times, with backoff climbing to sixty seconds. One genuinely bad request becomes 31 attempts and several minutes of latency, and if the failure happens after generation, 31 billed generations. Two rules keep retry policies honest: Retry only what a different host can fix. That’s 429 and 5xx, plus 408 if you like. A 400 means the request is wrong; identical requests to identical backends will be identically wrong. The only reason retrying 400 "worked" for me is that my backends were not identical — the retry was accidentally functioning as a resource-affinity search, at triple the cost. Size the budget to the pool, not to your optimism. With three backends, the most a cross-backend retry can ever explore is three attempts — count="2". Anything beyond that is re-asking hosts that already said no. Pair it with first-fast-retry="true" so a 429 moves to the next resource immediately rather than sleeping two seconds first; the whole point of the pool is that another resource is free right now. The best option when it’s available. Send the conversation in full, take no server-side handles, and the pool becomes legal again. In the OpenAI Agents SDK, it’s a run-level setting rather than a per-agent one, so it also covers agents reached through handoffs: python from agents import ModelSettings, RunConfig, Runner STATELESS = RunConfig model settings=ModelSettings store=False result = await Runner.run agent, user input, max turns=30, run config=STATELESS RunConfig.model settings merges over each agent's own settings — only non-null fields override — so per-agent temperature and tool choice survive. Verify it reached the wire before you believe it; wrapping client.responses.create and printing the store kwarg per turn takes two minutes and settles the question. The cost is bandwidth: you now upload the entire conversation every turn, which for a tool-heavy agent with a large system prompt is not nothing. The benefit is that any host can serve any turn, forever. When the state is genuinely required — Assistants threads, async video jobs, anything where the id is the product — the pool must stop being round-robin and start being a consistent hash. Route on a key the client controls, and every turn of one conversation lands on one host: php < -- gateway-side affinity: same session ⇒ same backend -- <set-variable name="slot" value="@ Math.Abs context.Request.Headers.GetValueOrDefault "x-session-id","" .GetHashCode % 3 " / <choose <when condition="@ context.Variables.GetValueOrDefault<int "slot" == 0 " <set-backend-service backend-id="res-a" / </when < -- … -- </choose Affinity buys correctness and costs you balance: one heavy session can hot-spot a single resource, and a host going down takes its sessions with it unless you add fallback. If your client can’t send a stable key, you don’t have this option — which is worth knowing before you design the gateway rather than after. The unglamorous option, and often correct. Split by workload instead of by request: send the stateful agent traffic to one resource with quota sized for it, and keep the pool for the embarrassingly parallel work — embeddings, batch classification, image generation — where round-robin is free. A load balancer in front of a stateful API isn’t a performance optimization; it’s a correctness hazard you’re choosing to manage. If you’re on a reasoning deployment, store=false raises a fair question: reasoning items are part of the conversation the model needs, and if nothing is stored server-side, where do they go? They come back inline, as encrypted content on the reasoning item — an opaque blob any resource in the family can decrypt. I expected to have to ask for it via include: "reasoning.encrypted content" , and tested both ways. With store=false the encrypted content was returned automatically , and replaying that item to a different resource was accepted. So on this deployment, statelessness costs you nothing in reasoning continuity. Test it on yours rather than trusting mine. The check is small: one turn with reasoning.effort high enough to emit a reasoning item, then replay that item's output to a different resource and see whether you get a 200. If your provider doesn't return the encrypted blob, statelessness and reasoning continuity are in genuine conflict, and you're back to affinity. The affinity bug was the headline, but the investigation surfaced a quieter problem worth checking in any pool you run. My three “identical” backends were provisioned independently, in three different subscriptions, and they had drifted: A pool is a claim that its members are interchangeable. If that claim isn’t enforced — same model version, same filter policy, same quota class — you haven’t built a load balancer, you’ve built a randomizer that occasionally changes your app’s behavior. Two days of gateway telemetry, after the diagnosis and before the fix: 867 calls to the Responses API, 443 of them failed — 51%. Meanwhile the image endpoints on the same gateway, same pool, ran 18 calls with zero failures, which is the whole thesis in one line: the stateless surfaces were fine, the stateful one was on fire. The same two-turn agent run, measured through the gateway before and after: Note what was not recovered: those failed calls were rejected during input validation, before generation, so they weren’t burning output tokens. The loss was latency, request quota, and the upload of a six-figure-byte payload three times per turn. Worth fixing, but it’s the kind of waste that hides from a cost dashboard — which is exactly why it ran for months. Numbers and error strings here come from a production incident on Azure OpenAI behind API Management, with an agent built on the OpenAI Agents SDK. Resource names have been genericized. The failure mode is not Azure-specific: any provider offering a stateful conversation API behind a multi-tenant pool has the same shape waiting in it. Round-Robin Broke My Agent https://pub.towardsai.net/round-robin-broke-my-agent-dda6885c0a25 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.