{"slug": "eliminate-gpu-waste-by-cutting-the-retry-tax", "title": "Eliminate GPU Waste by Cutting the Retry Tax", "summary": "A latent retry bug amplified GitHub Copilot traffic from roughly 7-9K requests per second to 70-100K RPS during an outage, and Microsoft reported that a single failed request during its May 29 Azure OpenAI outage could generate up to 48 additional retry attempts, highlighting the 'GPU retry tax' that wastes expensive compute. Aquifer, an agent-native load balancer, aims to mitigate retry amplification by queueing and dynamically pacing requests against downstream capacity, potentially reducing wasted GPU capacity and improving utilization.", "body_md": "In my previous article, [I proposed an agent-native load balancer](/content/github-outage-reactive-scaling/) built\naround the unique traffic patterns of agents.\n\nGitHub recently showed why this matters. A latent retry bug amplified Copilot traffic from\nroughly 7-9K requests per second to 70-100K RPS, delaying recovery during an outage.[1](#fn1)\n\nAquifer is designed to mitigate this kind of retry amplification. Instead of immediately rejecting excess work and forcing every client to retry independently, it queues requests and dynamically paces them against downstream capacity.\n\nDynamic pacing turns excess demand into waiting time instead of excess attempts.\n\nThat becomes especially interesting with GPUs.\n\nAt an illustrative $5 per GPU-hour, a fleet of 100,000 GPUs represents **$500,000 of\ncompute every hour**. Even a small percentage of wasted capacity caused by retries,\nduplicate inference, or abandoned work can become expensive quickly.\n\nWe don't currently have good public numbers showing how much GPU capacity the industry wastes\non retries. **Inference providers should measure it.**\n\nMicrosoft recently demonstrated how extreme retry amplification can become. During its May 29\nAzure OpenAI outage, which lasted 7 hours and 26 minutes, Microsoft reported that a single\nfailed request could generate **up to 48 additional retry attempts**.[2](#fn2)\n\nThose retries lacked sufficient backoff and jitter, and both remain important defenses. AWS\nhas shown that exponential backoff with jitter dramatically reduces wasted work under\ncontention.[3](#fn3)\n\nBut retries still aren't free.\n\nAgents make this more important because they operate at machine speed, fan out into parallel\nrequests, and retry automatically. Cockroach Labs has described this as a new thundering herd\nproblem for agent infrastructure. 4 Datadog also found that 60% of\nthe LLM-call span errors it observed in February 2026 were rate-limit errors.\n\n[5](#fn5)\nFor GPU inference, repeated work can be particularly expensive. NVIDIA explicitly describes\nKV-cache recomputation as wasted compute and designs its inference infrastructure to avoid\nunnecessary prefill computation.[6](#fn6)\n\nPutting the burden of retrying on the client creates a tax for both the consumer and the provider. Consumers pay in latency and repeated work. Providers pay in capacity, congestion, slower recovery, and potentially wasted inference.\n\nSo I think inference operators should start measuring a simple metric:\n\nHow many physical inference attempts does it take to produce one successful logical request?\n\nAnd ultimately:\n\nHow many GPU-seconds are being spent on retries, duplicate requests, abandoned inference, and recomputation?\n\nThat's the **GPU retry tax**.\n\nTraditional load balancers are good at deciding **where** a request should go.\n\nFor expensive workloads that can tolerate waiting, we also need infrastructure that decides\n**when** that request should run.\n\nAquifer and EZThrottle Local explore that model by queueing excess work and dynamically pacing\nexecution instead of relying entirely on failure followed by client retries.[7](#fn7)[8](#fn8)\n\nThis won't fit every workload. Real-time voice or interactive chat may require extremely low latency.\n\nBut a huge class of AI work does not.\n\nImage generation, video generation, music generation, research agents, document processing, batch inference, and long-running background agents can often tolerate waiting.\n\nFor these workloads, waiting may be cheaper than retrying.\n\nInstead of provisioning GPUs around every instantaneous spike, we can queue work, pace execution against available capacity, and scale when sustained useful demand actually requires it.\n\nThat could mean fewer retry storms, higher GPU utilization, less wasted compute, and potentially a greener inference stack.\n\nBefore buying another GPU, calculate your retry tax.", "url": "https://wpnews.pro/news/eliminate-gpu-waste-by-cutting-the-retry-tax", "canonical_source": "https://rahmipruitt.me/content/gpu-retry-tax/", "published_at": "2026-08-22 23:07:58+00:00", "updated_at": "2026-08-22 23:43:52.373539+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "ai-products"], "entities": ["GitHub", "Microsoft", "Azure OpenAI", "Aquifer", "NVIDIA", "Cockroach Labs", "Datadog", "AWS"], "alternates": {"html": "https://wpnews.pro/news/eliminate-gpu-waste-by-cutting-the-retry-tax", "markdown": "https://wpnews.pro/news/eliminate-gpu-waste-by-cutting-the-retry-tax.md", "text": "https://wpnews.pro/news/eliminate-gpu-waste-by-cutting-the-retry-tax.txt", "jsonld": "https://wpnews.pro/news/eliminate-gpu-waste-by-cutting-the-retry-tax.jsonld"}}