If you run real LLM traffic through Amazon Bedrock, throttling is not an edge case. It is the normal operating condition at scale. A ThrottlingException or TooManyRequestsException is Bedrock telling you that your request rate or token rate crossed a quota for that model in that region. The fix is rarely one setting. It is a layered defense: understand the quota, retry intelligently, cap concurrency, degrade gracefully, handle timeouts, and know what to escalate to AWS.
This is the set of strategies I run across production LLM workloads, written as a generic guide. No workload specifics, just the patterns and when each one earns its place.
TL;DR
- Know your quota first: check Service Quotas, learn which limits are adjustable (soft) and which are not (hard).
- Retry with exponential backoff and full jitter on throttling errors. This is the floor, not the ceiling.
- Cap concurrency so you stop generating load the quota cannot absorb.
- Add a circuit breaker so a throttle storm backs off instead of hammering.
- Handle timeouts separately: shrink the batch for model timeouts, tune and retry for client read timeouts.
- Use cross-region inference profiles to spread token demand across regions.
- Reserve provisioned throughput only when on-demand plus the above is not enough.
Before writing a line of retry code, find the actual limit. Bedrock quotas are per model, per region, and split by type: tokens per minute (TPM), requests per minute (RPM), and for some models separate input/output token quotas.
This distinction decides your whole strategy.
Adjustable: Yes. You can raise them. Request an increase through Adjustable: No. No amount of asking moves these. Your only lever is architectural: reduce demand, spread it across regions, or change the access pattern.
The practical rule: if the limit is soft and you are consistently near it, file the increase and also build the defenses below, because the increase takes time and never makes throttling impossible. If the limit is hard, skip the request entirely and go straight to architecture.
One often-missed detail: Bedrock reserves TPM quota at the start of a request based on input tokens plus maxTokens. If you leave maxTokens unset, it can default to the model maximum and reserve far more quota than your call actually needs, so you throttle earlier than your real token use suggests. Set maxTokens to a realistic ceiling.
The baseline defense. Catch the throttling error codes (ThrottlingException, TooManyRequestsException), wait, and try again. Two details separate a good retry from a harmful one:
base * 2^attempt, capped at a ceiling. This gives the quota window time to refill.
A compact pattern:
import random, time
from botocore.exceptions import ClientError
THROTTLE = {"ThrottlingException", "TooManyRequestsException"}
def call_with_backoff(fn, *, base=2.0, cap=15.0, max_retries=5):
for attempt in range(max_retries + 1):
try:
return fn()
except ClientError as exc:
code = exc.response.get("Error", {}).get("Code", "")
if code not in THROTTLE or attempt == max_retries:
raise
delay = min(base * 2 ** attempt, cap)
time.sleep(random.uniform(0, delay)) # full jitter
A note on the boto3 client's own retries. Boto3 has a built-in retry mode (standard or adaptive) you set via botocore.config.Config(retries=...). You can lean on it, or disable it ( max_attempts=0) and own the retry loop yourself. Owning it is worth the effort when you want the backoff, jitter, and error classification to match the rest of your system rather than two retry layers fighting each other.
If your Bedrock calls run inside a Step Functions state machine, you do not write the backoff loop by hand. You declare it on the task's Retry block and the service applies exponential backoff with jitter for you. Step Functions error handling added a JitterStrategy field so wait times are spread instead of synchronized, the same reason full jitter matters in code.
"InvokeBedrock": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Retry": [
{
"ErrorEquals": ["ThrottlingException", "TooManyRequestsException"],
"IntervalSeconds": 2,
"MaxAttempts": 15,
"BackoffRate": 2.0,
"MaxDelaySeconds": 30,
"JitterStrategy": "FULL"
}
],
"Catch": [
{ "ErrorEquals": ["States.ALL"], "Next": "HumanReview" }
],
"Next": "NextStep"
}
The fields map onto the code version: IntervalSeconds is the base delay, BackoffRate is the exponential multiplier, MaxDelaySeconds is the cap, MaxAttempts is the retry count, and JitterStrategy: FULL randomizes each wait. The Catch routes to a fallback once retries are exhausted, for example a human-review step, so an exhausted retry degrades gracefully instead of failing the whole execution.
Retries handle the request that already failed. Concurrency caps stop you from generating more load than the quota can absorb in the first place. If 50 workers all call the same model at once, you will throttle no matter how clever the backoff is.
Put a ceiling on in-flight Bedrock calls. In async Python a semaphore does it:
import asyncio
semaphore = asyncio.Semaphore(10) # max concurrent Bedrock calls
async def bounded_call(fn):
async with semaphore:
return await fn()
Upstream of the application, cap concurrency at the infrastructure layer too. If documents arrive as events, put them on an SQS queue and let the SQS-to-Lambda event source mapping enforce a maximum_concurrency. Work waits in the queue until a slot frees, with no polling and no custom counters:
resource "aws_lambda_event_source_mapping" "sqs_to_worker" {
event_source_arn = aws_sqs_queue.work.arn
function_name = aws_lambda_function.worker.arn
batch_size = 1
scaling_config {
maximum_concurrency = 5
}
}
Match the concurrency ceiling to the quota, not to how many workers you could theoretically run. The right number is the one that keeps steady-state token rate under the TPM limit.
Backoff and concurrency caps handle normal pressure. A sustained throttle storm needs something that steps back harder. A circuit breaker watches the throttle rate and, once it crosses a threshold inside a time window, s all calls for a cooldown, then resumes at reduced concurrency before restoring full throughput.
The state machine:
This converts a feedback loop that makes throttling worse (retry, throttle, retry, throttle) into one that actively sheds load until Bedrock recovers. It pairs with the concurrency limiter: the breaker is what turns the limit down and back up.
Throttling rarely travels alone, and the loudest neighbor is the timeout. There are two distinct kinds, and they want opposite fixes.
A ModelTimeoutException means the request itself was too large to finish in the model's time budget. It is not a rate problem, so retrying the same payload will time it out again. The fix is to reduce the work: halve the batch or chunk the input, then retry smaller.
async def call_shrinking(fn, items, *, max_retries=3):
batch = list(items)
for attempt in range(max_retries + 1):
try:
return await fn(batch)
except Exception as exc:
code = getattr(exc, "response", {}).get("Error", {}).get("Code", "")
if code != "ModelTimeoutException" or attempt == max_retries:
raise
batch = batch[: max(1, len(batch) // 2)] # halve and retry
A client-side ReadTimeoutError or ConnectTimeoutError is different again: the SDK gave up waiting before the model answered. Large prompts and long generations routinely exceed the boto3 default read timeout, which produces timeouts that look like failures but are really an impatient client.
Two moves:
read_timeout
from botocore.config import Config
client = boto3.client(
"bedrock-runtime",
config=Config(read_timeout=300, retries={"max_attempts": 0}),
)
RETRYABLE = {"ThrottlingException", "TooManyRequestsException"}
Classify before reacting. A throttle wants backoff, a model timeout wants a smaller batch, a client timeout wants a longer budget plus a retry, and a ValidationException wants no retry at all.
Quotas are per region. A cross-region inference profile lets a single on-demand call be served from one of several regions, so your token demand is spread across each region's quota instead of piling onto one. For on-demand workloads this is often the highest-leverage change: no capacity to buy, and the effective ceiling goes up because you are drawing from multiple regional pools.
It is not infinite headroom, and it does not change a hard per-model limit in a single region, but for spiky on-demand traffic it buys real room.
If on-demand plus everything above still throttles under predictable heavy load, Bedrock offers Provisioned Throughput: reserved capacity (model units) with a guaranteed token rate, bought by commitment. It removes throttling for that reserved slice because the capacity is yours.
Treat it as the last layer, not the first. It costs whether or not you use it, so it only makes sense for steady, predictable volume. For bursty or unpredictable traffic, retries, concurrency caps, and cross-region profiles on on-demand are usually the better economics.
The layers above are the toolbox. How hard you turn each knob depends on the workload, and a couple of patterns are worth stating plainly.
Match the aggressiveness of retries to the cost of a lost call. Heavy, expensive calls justify more retry attempts, a higher backoff ceiling, and tighter concurrency, because losing one is costly. High-volume, cheaper work is better served by an upstream queue that caps concurrency and a workflow-level self-retry that ends in a manual-review fallback, keeping the per-call retries light. There is no single correct number; the right setting is the one that keeps steady-state token rate under the quota for that workload.
Two caveats that are easy to get wrong:
A short decision order when you see throttling:
Throttling is not a wall you hit once and clear. It is a pressure you manage continuously. The systems that stay up are the ones that treat the quota as a first-class design constraint, not an exception to catch.