cd /news/ai-agents/how-to-handle-llm-api-rate-limits-in… · home › topics › ai-agents › article
[ARTICLE · art-171983] src=startupfortune.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

How to handle LLM API rate limits in production before retries burn your quota

Production teams should budget agent fleets at roughly 80% of a provider's tokens-per-minute limit and cap retries at about 10% of total requests, the per-client budget Google's SRE book recommends, because retry loops rather than 429 errors drain quota. The guidance cites OpenAI's rate limit cookbook, which warns that unsuccessful requests still count toward the per-minute limit, and notes that a single agent step sending 14,000 tokens and retrying four times consumes 56,000 tokens of quota with nothing produced. Anthropic's documentation describes its limit as a continuously refilling token bucket split into input and output tokens, and the official Python clients retry a couple of times by default, so three retries in a wrapper on top of two in the SDK can reach up to twelve attempts for one logical call.

recommendation

Set agent spend at 80% of your tokens-per-minute limit, read the retry-after header, cap retries at 10% of requests and four or five attempts, and halve concurrency on a 429 while tracking tokens spent on failed attempts.

more in playbook: Controlling LLM cost
by read11 min views2 publishedNov 18, 2026
How to handle LLM API rate limits in production before retries burn your quota
Image: Startupfortune (auto-discovered)

Your agents' 429 errors aren't what drains the quota. The retry loop you wrote to fix them does, by sending the same heavy prompt again and again while everything else waits.

  • Set the agent fleet's spend at about 80% of your provider's tokens-per-minute limit and keep the other 20% free as headroom.
  • Split that budget into lanes, for example 30% for user-facing calls and the rest for background agents, so a batch job can't starve a customer.
  • Read the retry-after header first and fall back to full-jitter exponential backoff with a hard cap of four or five attempts.
  • Cap retries at about 10% of total requests, the per-client budget Google's SRE book recommends, so an outage can't turn into a retry storm.
  • Cut agent concurrency in half on a 429 and raise it by one on each success, and track the share of tokens spent on failed attempts as your headline metric.

Picture one agent step that sends 14,000 tokens of instructions, tool results and history to a model, gets a 429 back and retries four times in a row. On OpenAI's API, where the company's own rate limit cookbook warns that unsuccessful requests still count toward your per-minute limit, that single stuck step has just eaten 56,000 tokens of quota and produced nothing. Providers differ on whether a rejected call is billed against the limit, so check yours. Either way, the arithmetic is why how to handle LLM API rate limits in production starts with a budget and ends with a retry policy, not the other way round.

Most teams do it backwards. They add a retry decorator, watch the errors disappear in staging, and ship. Then traffic doubles and the errors come back worse.

An HTTP 429 means you asked for more than your allowance. That's all it says. It doesn't say the provider is broken, and it doesn't say the same request will work in two seconds. Treating it as a transient network blip is the root mistake, because a blip clears by itself and a spent budget doesn't.

The limits are also more than one number. Both OpenAI and Anthropic meter requests per minute and tokens per minute, and Anthropic splits the token side into input and output tokens. Anthropic's documentation describes the limit as a token bucket that refills continuously, not a counter that resets on the clock minute. That detail matters for planning: a burst can drain the bucket in a few seconds, and you then wait on the refill rate, not for a fresh minute to start.

How to Build an AI Agent Usage Dashboard Customers Will Actually Trust The piece points to Anthropic's own Console, which breaks token usage out by model and by API key so a developer can trace spend to its source. It also flags a concrete failure case: an agent looping 12,000 times overnight before anyone notices. - how to build transparent AI agent usage dashboard - why customers dispute AI agent billing charges

Agents make all of this worse in three ways. A single task can chain thirty model calls, and each call resends a growing context, so token cost per step climbs as the run goes on. Several agents usually share one API key and therefore one organisation-level limit, so one runaway loop starves the rest. And agents retry at two layers at once: your code and, very often, the provider's SDK, which in the official Python clients retries a couple of times by default. Three retries in your wrapper on top of two in the SDK is up to twelve attempts for one logical call.

Then comes the cascade. Twenty agents hit the ceiling at the same moment, all of them back off for the same one or two seconds, and all of them come back together. The second wave hits a bucket that's only partly refilled, so some fail again and sleep longer. Meanwhile the requests that did succeed are slower, which keeps more agents running at once. Nothing in that loop is a bug. It's what synchronised retries do.

How to size a tokens per minute limit for AI agents #

Start with the real number. Your dashboard shows the tier limit for each model, and the response headers show what's left at that instant: OpenAI sends x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens, and Anthropic sends anthropic-ratelimit-input-tokens-remaining along with a retry-after value on a 429. Log those headers on every call for a week before you design anything. Limits are often per model, so a fleet that uses a large model for planning and a small one for tool calls has two budgets, not one.

Next, measure what one step actually costs. Every response reports its usage, so you don't need to guess. Pull the median and the 95th percentile of input plus output tokens per step for each agent type. The 95th percentile is the number to plan with, since agents that run long are the ones that cause the trouble.

Watch the max_tokens setting while you're there. OpenAI's rate limit documentation says the limiter estimates your usage partly from the max_tokens you request, not only from what the model ends up writing, and Anthropic's output limit is estimated from max_tokens at the start of a request too. An agent that asks for 8,000 output tokens to produce a 200-token tool call reserves forty times more budget than it uses. Setting max_tokens close to what each step really needs is the cheapest capacity you'll ever buy.

Now do the sums. Say your limit is 400,000 tokens per minute. Hold back 20% as headroom and you plan against 320,000. That 20% is what absorbs a burst, a retry wave or a bad deploy, and it's the first thing teams talk themselves out of when finance asks why the limit isn't fully used. Frankly, leave it alone. A fleet run at 100% of its limit spends its life being throttled.

Split the 320,000 into lanes. Give 30%, or 96,000, to traffic where a human is waiting on the answer. The remaining 224,000 goes to background agents: research jobs, enrichment, nightly summaries. If a background agent step costs about 18,000 tokens at the 95th percentile, that lane supports roughly twelve steps a minute. That sounds low until you notice it's an honest number, and it tells you at once whether twenty parallel agents are possible. They aren't, and it's better to learn that from a spreadsheet than from an incident.

How to Calculate AI Agent Cost Per Task Before It Wrecks Your Margin Anthropic's Claude Opus 4.5 pricing runs $5 per million input tokens and $25 per million output, and retries on failed runs can cost more than the original attempt, turning a 70% success rate into roughly triple the cost of a 95% rate. - how to calculate AI agent cost per task - AI agent pricing model for startup margins

Priority lanes should be soft where it's safe and hard where it isn't. Let background work borrow unused interactive capacity, and take it back the moment a user request arrives. Never let the reverse happen without a ceiling, or a spike in chat traffic will quietly starve every scheduled job.

If your agents run in more than one process, the limiter can't live inside each one. Put the token bucket in shared storage. Redis with an atomic script is the usual choice, because every worker has to draw from the same bucket before it sends a request. Per-process limiters multiply your real rate by the number of workers, which is how a fleet scaled from three pods to nine suddenly starts throwing 429s without a single line of code changing.

LLM API 429 error exponential backoff, done with jitter and a ceiling #

When a 429 does get through, the order of operations matters. First, read retry-after. If the provider tells you how long to wait, wait that long, plus a small random amount. It beats any formula you'll write, because it's based on the provider's actual bucket state.

If there's no header, use exponential backoff with full jitter: sleep for a random time between zero and the smaller of your cap and base times two to the power of the attempt number. Marc Brooker's AWS Architecture Blog post on exponential backoff and jitter simulated this against plain exponential backoff and found that the jittered versions finished the same contended work with far fewer calls, because the clients stopped arriving in synchronised waves. Plain exponential backoff without jitter still makes everyone wait the same time. It just makes them wait longer. Then put limits on the retrying itself, because that's where the money goes. Cap attempts at four or five. Cap total elapsed time per logical call, say 60 seconds for a background step and 8 seconds for an interactive one, since a user who has waited that long has already left. And add a retry budget. Google's SRE book recommends a per-client limit where retries may make up no more than about 10% of requests, on top of a per-request cap of three attempts. When the retry ratio goes over that line, new retries are dropped and the call fails fast. During a real provider incident, this one rule is the difference between a degraded service and a self-inflicted outage.

Not every error deserves the same treatment. A 429 means you're over your limit. A 529 from Anthropic means the service itself is overloaded, and 500-series errors mean something broke on the other side. Count them separately. If you retry a 529 with the same aggressive logic as a 429, you're adding load to a service that's already struggling, and your own budget accounting becomes useless because two different problems are in one counter.

The agent-specific part is what you do after the retries run out. A thirty-step run that fails on step twenty-six shouldn't start from step one. Save the state after each completed step, including tool outputs, so a retry resumes from the failed call and doesn't replay twenty-five steps of token spend. Make any tool with side effects idempotent too, or a resumed run will send the same email twice.

If your provider offers prompt caching, use it as rate limit relief, not just a cost saver. Anthropic's documentation says cached input tokens don't count toward the input-tokens-per-minute limit on most current models. An agent that resends a long, stable system prompt and tool list on every step can cut its metered input dramatically by caching that prefix, and that frees budget without touching your quota tier.

Queues, priority lanes and shedding load #

Put a queue between your agents and the client. Agents submit a request with a priority, an estimated token cost and a deadline, and a single dispatcher releases requests only when the shared bucket has room. This inverts the usual design. Instead of sending first and finding out about the limit from an error, the request waits its turn and the 429 becomes rare.

Deadlines make the queue honest. A background step that has been waiting longer than its deadline should be dropped or rescheduled, not sent late into a bucket that other work is already competing for. A queue with no shedding rule just stores your outage in memory and releases it all at once.

Concurrency is the other dial, and a simple rule works well. Start each agent pool at a fixed number of parallel runs. On a 429, cut that number in half. On every success, raise it by one. Networking engineers will recognise this as additive increase, multiplicative decrease, the logic behind TCP congestion control. It finds your real capacity without anyone needing to know the exact limit, and it backs off on its own when the provider changes your tier or has a bad afternoon.

Retrying isn't the only response to saturation. When the bucket is low, you can route low-stakes steps such as classification, formatting and routing to a smaller, cheaper model that has its own separate limit. That's capacity you already own and aren't using.

What to put on the dashboard #

Four numbers tell you whether the budget is working: tokens used per minute as a share of your limit, the 429 rate, queue wait at the 95th percentile, and the share of tokens spent on attempts that failed or were retried. The last one is the metric most teams don't have, and it's the one that connects a rate limit problem to the invoice. Alert when sustained usage passes 70% of the limit, since that's already inside your headroom.

Run a drill before you need one. Drop your limiter's ceiling to half its normal value in staging and watch what the fleet does: whether interactive latency holds, whether background work drains in order, whether the retry ratio stays under 10%. If a half-sized quota breaks something you can see, a real throttle from your provider will break it too, only at a worse hour.

Pull last week's logs and calculate how many tokens went to calls that returned a 429 or were retried. If that figure is above 10% of your total, your agents are paying for the same work twice, and the fix is a limiter and a budget, not a longer timeout.

Also read: How to Write AI Agent Tool Descriptions That Stop Wrong Tool Calls • How to Compare LLM API Cost Per Million Tokens Before You Pick a Model • How to Calculate Cost of Goods Sold for AI Startups Before Diligence Does

This article is posted in AI News, check it out for more related stories.

Join the discussion #

Open in the community → Almost there. Sign in and your reply posts straight away.

── more in #ai-agents 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-handle-llm-ap…] indexed:0 read:11min 2026-11-18 · —