A Field Guide to Beating Amazon Bedrock Throttling

A developer has published a field guide to mitigating Amazon Bedrock throttling in production LLM workloads, recommending a layered defense built on knowing per-model, per-region quotas, exponential backoff with full jitter, concurrency caps, circuit breakers, separate timeout handling, cross-region inference profiles, and provisioned throughput as a last resort. The guide notes that Bedrock reserves token-per-minute quota at request start based on input tokens plus maxTokens, so leaving maxTokens unset can cause earlier throttling than actual usage warrants.

If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS. This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place. TL;DR - Know your quota first: check Service Quotas, learn which limits are adjustable soft and which are not hard . - Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling. - Cap concurrency so you stop generating load the quota cannot absorb. - Add a circuit breaker so a throttle storm backs off instead of hammering. - Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts. - Use cross-region inference profiles to spread token demand across regions. - Reserve provisioned throughput only when on-demand plus the above is not enough. Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute TPM , requests per minute RPM , and for some models separate input/output token quotas. This distinction decides your whole strategy. Adjustable: Yes . You can raise them. Request an increase through Adjustable: No . No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern. The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture. One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens . If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling. The baseline defense. Catch the throttling error codes ThrottlingException , TooManyRequestsException , wait, and try again. Two details separate a good retry from a harmful one: base 2^attempt , capped at a ceiling. This gives the quota window time to refill. A compact pattern: python import random, time from botocore.exceptions import ClientError THROTTLE = {"ThrottlingException", "TooManyRequestsException"} def call with backoff fn, , base=2.0, cap=15.0, max retries=5 : for attempt in range max retries + 1 : try: return fn except ClientError as exc: code = exc.response.get "Error", {} .get "Code", "" if code not in THROTTLE or attempt == max retries: raise delay = min base 2 attempt, cap time.sleep random.uniform 0, delay full jitter A note on the boto3 client's own retries. Boto3 has a built-in retry mode standard or adaptive you set via botocore.config.Config retries=... . You can lean on it, or disable it max attempts=0 and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other. If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code. "InvokeBedrock": { "Type": "Task", "Resource": "arn:aws:states:::bedrock:invokeModel", "Retry": { "ErrorEquals": "ThrottlingException", "TooManyRequestsException" , "IntervalSeconds": 2, "MaxAttempts": 15, "BackoffRate": 2.0, "MaxDelaySeconds": 30, "JitterStrategy": "FULL" } , "Catch": { "ErrorEquals": "States.ALL" , "Next": "HumanReview" } , "Next": "NextStep" } The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution. Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is. Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it: python import asyncio semaphore = asyncio.Semaphore 10 max concurrent Bedrock calls async def bounded call fn : async with semaphore: return await fn Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum concurrency . Work waits in the queue until a slot frees, with no polling and no custom counters: resource "aws lambda event source mapping" "sqs to worker" { event source arn = aws sqs queue.work.arn function name = aws lambda function.worker.arn batch size = 1 scaling config { maximum concurrency = 5 } } Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit. Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput. The state machine: This converts a feedback loop that makes throttling worse retry, throttle, retry, throttle into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up. Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes. A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller. python async def call shrinking fn, items, , max retries=3 : batch = list items for attempt in range max retries + 1 : try: return await fn batch except Exception as exc: code = getattr exc, "response", {} .get "Error", {} .get "Code", "" if code = "ModelTimeoutException" or attempt == max retries: raise batch = batch : max 1, len batch // 2 halve and retry A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client. Two moves: read timeout python from botocore.config import Config client = boto3.client "bedrock-runtime", config=Config read timeout=300, retries={"max attempts": 0} , RETRYABLE = {"ThrottlingException", "TooManyRequestsException"} also retry ReadTimeoutError / ConnectTimeoutError by exception class name Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all. Quotas are per region. A cross-region inference profile https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles.html lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools. It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room. If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html : reserved capacity model units with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours. Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics. The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly. Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload. Two caveats that are easy to get wrong: A short decision order when you see throttling: Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.