{"slug": "a-field-guide-to-beating-amazon-bedrock-throttling", "title": "A Field Guide to Beating Amazon Bedrock Throttling", "summary": "A developer has published a field guide to mitigating Amazon Bedrock throttling in production LLM workloads, recommending a layered defense built on knowing per-model, per-region quotas, exponential backoff with full jitter, concurrency caps, circuit breakers, separate timeout handling, cross-region inference profiles, and provisioned throughput as a last resort. The guide notes that Bedrock reserves token-per-minute quota at request start based on input tokens plus maxTokens, so leaving maxTokens unset can cause earlier throttling than actual usage warrants.", "body_md": "If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A `ThrottlingException` or `TooManyRequestsException` is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.\n\nThis is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.\n\n**TL;DR**\n\n- Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).\n- Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.\n- Cap concurrency so you stop generating load the quota cannot absorb.\n- Add a circuit breaker so a throttle storm backs off instead of hammering.\n- Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.\n- Use cross-region inference profiles to spread token demand across regions.\n- Reserve provisioned throughput only when on-demand plus the above is not enough.\n\nBefore writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.\n\nThis distinction decides your whole strategy.\n\n`Adjustable: Yes`. You can raise them. Request an increase through `Adjustable: No`. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.\nThe practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.\n\nOne often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus `maxTokens`. If you leave `maxTokens` unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set `maxTokens` to a realistic ceiling.\n\nThe baseline defense. Catch the throttling error codes (`ThrottlingException`, `TooManyRequestsException`), wait, and try again. Two details separate a good retry from a harmful one:\n\n`base * 2^attempt`, capped at a ceiling. This gives the quota window time to refill.\nA compact pattern:\n\n``` python\nimport random, time\nfrom botocore.exceptions import ClientError\n\nTHROTTLE = {\"ThrottlingException\", \"TooManyRequestsException\"}\n\ndef call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):\n    for attempt in range(max_retries + 1):\n        try:\n            return fn()\n        except ClientError as exc:\n            code = exc.response.get(\"Error\", {}).get(\"Code\", \"\")\n            if code not in THROTTLE or attempt == max_retries:\n                raise\n            delay = min(base * 2 ** attempt, cap)\n            time.sleep(random.uniform(0, delay))  # full jitter\n```\n\nA note on the boto3 client's own retries. Boto3 has a built-in retry mode (`standard` or `adaptive`) you set via `botocore.config.Config(retries=...)`. You can lean on it, or disable it (` max_attempts=0`) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.\n\nIf your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's `Retry` block and the service applies exponential backoff with jitter for you. [Step Functions error handling](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html) added a `JitterStrategy` field so wait times are spread instead of synchronized, the same reason full jitter matters in code.\n\n```\n\"InvokeBedrock\": {\n  \"Type\": \"Task\",\n  \"Resource\": \"arn:aws:states:::bedrock:invokeModel\",\n  \"Retry\": [\n    {\n      \"ErrorEquals\": [\"ThrottlingException\", \"TooManyRequestsException\"],\n      \"IntervalSeconds\": 2,\n      \"MaxAttempts\": 15,\n      \"BackoffRate\": 2.0,\n      \"MaxDelaySeconds\": 30,\n      \"JitterStrategy\": \"FULL\"\n    }\n  ],\n  \"Catch\": [\n    { \"ErrorEquals\": [\"States.ALL\"], \"Next\": \"HumanReview\" }\n  ],\n  \"Next\": \"NextStep\"\n}\n```\n\nThe fields map onto the code version: `IntervalSeconds` is the base delay, `BackoffRate` is the exponential multiplier, `MaxDelaySeconds` is the cap, `MaxAttempts` is the retry count, and `JitterStrategy: FULL` randomizes each wait. The `Catch` routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.\n\nRetries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.\n\nPut a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:\n\n``` python\nimport asyncio\n\nsemaphore = asyncio.Semaphore(10)  # max concurrent Bedrock calls\n\nasync def bounded_call(fn):\n    async with semaphore:\n        return await fn()\n```\n\nUpstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a `maximum_concurrency`. Work waits in the queue until a slot frees, with no polling and no custom counters:\n\n```\nresource \"aws_lambda_event_source_mapping\" \"sqs_to_worker\" {\n  event_source_arn = aws_sqs_queue.work.arn\n  function_name    = aws_lambda_function.worker.arn\n  batch_size       = 1\n  scaling_config {\n    maximum_concurrency = 5\n  }\n}\n```\n\nMatch the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.\n\nBackoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, pauses all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.\n\nThe state machine:\n\nThis converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.\n\nThrottling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.\n\nA `ModelTimeoutException` means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.\n\n``` python\nasync def call_shrinking(fn, items, *, max_retries=3):\n    batch = list(items)\n    for attempt in range(max_retries + 1):\n        try:\n            return await fn(batch)\n        except Exception as exc:\n            code = getattr(exc, \"response\", {}).get(\"Error\", {}).get(\"Code\", \"\")\n            if code != \"ModelTimeoutException\" or attempt == max_retries:\n                raise\n            batch = batch[: max(1, len(batch) // 2)]  # halve and retry\n```\n\nA client-side `ReadTimeoutError` or `ConnectTimeoutError` is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.\n\nTwo moves:\n\n`read_timeout`\n\n``` python\nfrom botocore.config import Config\n\nclient = boto3.client(\n    \"bedrock-runtime\",\n    config=Config(read_timeout=300, retries={\"max_attempts\": 0}),\n)\n\nRETRYABLE = {\"ThrottlingException\", \"TooManyRequestsException\"}\n# also retry ReadTimeoutError / ConnectTimeoutError by exception class name\n```\n\nClassify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a `ValidationException` wants no retry at all.\n\nQuotas are per region. A [cross-region inference profile](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles.html) lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.\n\nIt is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.\n\nIf on-demand plus everything above still throttles under predictable heavy load, Bedrock offers [Provisioned Throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html): reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.\n\nTreat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.\n\nThe layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.\n\nMatch the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.\n\nTwo caveats that are easy to get wrong:\n\nA short decision order when you see throttling:\n\nThrottling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.", "url": "https://wpnews.pro/news/a-field-guide-to-beating-amazon-bedrock-throttling", "canonical_source": "https://dev.to/avneet_bansal_a65b3f31fc4/a-field-guide-to-beating-amazon-bedrock-throttling-1b9g", "published_at": "2026-10-20 20:45:01+00:00", "updated_at": "2026-10-20 20:48:13.094948+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["Amazon Bedrock", "AWS", "boto3", "Step Functions"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-field-guide-to-beating-amazon-bedrock-throttling", "markdown": "https://wpnews.pro/news/a-field-guide-to-beating-amazon-bedrock-throttling.md", "text": "https://wpnews.pro/news/a-field-guide-to-beating-amazon-bedrock-throttling.txt", "jsonld": "https://wpnews.pro/news/a-field-guide-to-beating-amazon-bedrock-throttling.jsonld"}}