{"slug": "rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that", "title": "Rekognition ProvisionedThroughputExceededException: TPS limits and backoff that holds up", "summary": "A developer documented how AWS Rekognition's per-account, per-region TPS quotas (5 TPS for IndexFaces, SearchFaces and SearchFacesByImage in most regions, 50 TPS in us-east-1, us-west-2 and eu-west-1) cause ProvisionedThroughputExceededException during large face-indexing backfills, and outlined a fix combining a shared client-side token bucket set just under the quota, the SDK's adaptive retry mode and a dead-letter list for failed photos. The writeup notes that retries alone worsen throttling because synchronized worker retries arrive in over-quota waves, and that a 200,000-photo backfill takes roughly 11 hours at 5 TPS versus about an hour at 50 TPS.", "body_md": "**Editor note for Ashish (delete before publishing):** this draft was written with AI. Rewrite it in your own voice and check every claim. If any AI-written text remains, add a line saying the post was written with AI assistance (DEV.to and Medium both require this).\n\n**TL;DR** Rekognition's default quota for `IndexFaces`, `SearchFacesByImage` and `SearchFaces` is **5 transactions per second per account in most regions**, and 50 in us-east-1, us-west-2 and eu-west-1. A naive parallel backfill blows through that in under a second and gets `ProvisionedThroughputExceededException` (or `ThrottlingException`). Retries alone don't fix it. Rate-limit on the client to just under the quota, share one limiter across every worker in the account and region, set the SDK to `adaptive` retry mode, and request a quota increase before a big job, not during it.\n\nThe usual story: a face-search feature works fine in development, then someone kicks off a backfill of 200,000 existing photos with a thread pool of 32 workers. Within seconds the logs fill with this:\n\n```\nbotocore.errorfactory.ProvisionedThroughputExceededException: An error occurred\n(ProvisionedThroughputExceededException) when calling the IndexFaces operation (reached max retries: 2):\nProvisioned rate exceeded\n```\n\n\"reached max retries: 2\" is the SDK telling you it already retried twice with backoff and gave up. The problem isn't the retry policy. The job is asking for more throughput than the account has.\n\nFrom the [Rekognition service quotas page](https://docs.aws.amazon.com/general/latest/gr/rekognition.html). These are **per account, per region, per operation**, so `IndexFaces` and `SearchFacesByImage` each have their own budget.\n\n*Default TPS quotas as listed by AWS on 2026-10-31. Mumbai, Frankfurt, Tokyo, Sydney, Singapore, London, Ohio and others are in the right-hand column.*\n\n| Operations | us-east-1, us-west-2, eu-west-1 | Most other regions | \n|---|---|---|\n| IndexFaces, SearchFaces, SearchFacesByImage, ListFaces, SearchUsers, SearchUsersByImage | 50 TPS | 5 TPS | \n| CompareFaces, DetectFaces | 100 TPS | 25 TPS | \n| AssociateFaces, DisassociateFaces | 5 TPS | 5 TPS | \n| CreateCollection, DeleteCollection, DeleteFaces, DescribeCollection, ListCollections | 5 TPS | 5 TPS | \n\nDo the arithmetic for that backfill. 200,000 photos at 5 TPS is 40,000 seconds, about 11 hours, and that's at a perfect 5 per second with zero throttles. At 50 TPS it's a bit over an hour. Search has its own quota, so live searches aren't affected by the backfill. Live uploads are: `IndexFaces` from your upload path and from the backfill share one 5 TPS budget, so new uploads start failing while the backfill runs.\n\n**Note:** Throttling is per account. Two services, two environments or two developers' scripts in the same AWS account and region all draw on the same quota. A staging backfill can throttle production.\n\nEach worker that gets throttled backs off and tries again. With 32 workers retrying on similar schedules, the retries arrive in waves that are themselves over quota. Raising `max_attempts` to 10 just means each photo fails more slowly. In the SDK's default `standard` mode, failed retries also drain a client-side retry quota (a token bucket), so a sustained overload ends with the SDK refusing to retry at all.\n\nThe [AWS SDK retry docs](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html) say the default is `standard` mode with `max_attempts` of 3 (one request plus two retries). `adaptive` mode adds a client-side rate limiter that slows the initial requests too once it sees throttling, which is what a bulk job needs. AWS's own guidance is that adaptive is for \"single-resource, throttling-heavy, latency-tolerant\" work. A backfill fits that description. A user-facing search request doesn't.\n\nThree pieces: a token bucket set slightly below the quota, shared by all workers; `adaptive` retries as a safety net; and a dead-letter list for photos that still fail, so the job finishes and you retry the remainder later.\n\n*One process, many threads, one shared limiter. For several processes, put the bucket in Redis.*\n\n``` python\nimport threading, time\nimport boto3\nfrom botocore.config import Config\nfrom botocore.exceptions import ClientError\nfrom concurrent.futures import ThreadPoolExecutor\n\nREGION = \"ap-south-1\"\nQUOTA_TPS = 5          # check Service Quotas for your account; it may have been raised\nTARGET_TPS = QUOTA_TPS * 0.8   # headroom for live traffic and clock jitter\n\nrek = boto3.client(\n    \"rekognition\",\n    region_name=REGION,\n    config=Config(retries={\"mode\": \"adaptive\", \"max_attempts\": 6}),\n)\n\nclass TokenBucket:\n    def __init__(self, rate: float, burst: int = 1):\n        self.rate, self.capacity = rate, burst\n        self.tokens, self.updated = burst, time.monotonic()\n        self.lock = threading.Lock()\n\n    def take(self) -> None:\n        while True:\n            with self.lock:\n                now = time.monotonic()\n                self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)\n                self.updated = now\n                if self.tokens >= 1:\n                    self.tokens -= 1\n                    return\n                wait = (1 - self.tokens) / self.rate\n            time.sleep(wait)\n\nbucket = TokenBucket(TARGET_TPS)\nfailed: list[tuple[str, str]] = []\n\ndef index_one(key: str) -> None:\n    bucket.take()\n    try:\n        rek.index_faces(\n            CollectionId=\"gallery-2026\",\n            Image={\"S3Object\": {\"Bucket\": \"my-photos\", \"Name\": key}},\n            ExternalImageId=key.replace(\"/\", \"_\"),  # ExternalImageId allows [a-zA-Z0-9_.\\-:]+\n            MaxFaces=100,\n        )\n    except ClientError as e:\n        failed.append((key, e.response[\"Error\"][\"Code\"]))\n\nkeys = [...]  # your S3 keys\nwith ThreadPoolExecutor(max_workers=8) as pool:   # more threads than TPS buys nothing\n    list(pool.map(index_one, keys))\nprint(f\"done, {len(failed)} to retry later\")\n```\n\nA few notes on the numbers. The pool size only needs to cover latency: if an `IndexFaces` call takes around 400 ms, four threads already sustain 10 calls a second, so eight is plenty for a 5 TPS budget. Burst is 1 on purpose, so the job never sends a spike on startup. And 80% of quota leaves room for whatever else in the account calls the same operation.\n\n`standard` mode with the default 3 attempts, and return a \"busy, try again\" response after that.`SearchFacesByImage` has its own quota, but live `IndexFaces` (new uploads) shares one with any backfill. Pause or slow the backfill during peak upload hours.`ThrottledCount` metric for Rekognition. An alarm on it tells you you're near the ceiling before users notice.\nTPS quotas are adjustable through Service Quotas or the AWS Support Center, per operation and per region. Ask with a specific number and a reason (\"IndexFaces to 50 TPS in ap-south-1 for a one-time backfill of 2M photos\"), and ask days before you need it, since approval isn't instant. Ask for each operation separately: a raise for `IndexFaces` does nothing for `SearchFacesByImage`.\n\nWe build [SightRadar](https://sightradar.com/?utm_source=devto&utm_medium=referral&utm_campaign=launch-2026q4&utm_content=rekognition-throttling-provisionedthroughputexceededexception), a face search API that uses Rekognition's collection model, and we went the other way for bulk work. `POST /v1/batches` takes up to 1,000 photo URLs in one call, queues them, and sends one signed webhook per photo as results arrive. Your code never meets a per-second ceiling during a backfill, and batch photos are billed at $0.00062 each instead of the $0.00093 real-time rate. See [batch and webhooks](https://sightradar.com/docs/guides/batch-and-webhooks?utm_source=devto&utm_medium=referral&utm_campaign=launch-2026q4&utm_content=rekognition-throttling-provisionedthroughputexceededexception).\n\n``` python\nimport os, requests\n\nphotos = [{\"url\": f\"https://cdn.example.com/{k}\", \"external_id\": k} for k in keys[:1000]]\nr = requests.post(\n    \"https://api.sightradar.com/v1/batches\",\n    headers={\"Authorization\": f\"Bearer {os.environ['SIGHTRADAR_API_KEY']}\"},\n    json={\"collection_id\": \"gallery-2026\", \"photos\": photos},\n    timeout=30,\n)\nprint(r.json()[\"batch_id\"])  # results arrive per photo via webhook or GET /v1/batches/{id}\n```\n\nReal-time calls do have limits: 50 requests per second per API key and 10 in-flight requests per account by default, returning `429`. So the token-bucket advice above still applies to a synchronous loop. The difference is that bulk jobs don't need one. If your existing code is boto3, the [Rekognition shim](https://sightradar.com/docs/sdks/rekognition-shim?utm_source=devto&utm_medium=referral&utm_campaign=launch-2026q4&utm_content=rekognition-throttling-provisionedthroughputexceededexception) keeps your `index_faces` calls as they are.\n\nEstimate a backfill at batch rates before you start one. [Open the pricing calculator](https://sightradar.com/pricing?utm_source=devto&utm_medium=referral&utm_campaign=launch-2026q4&utm_content=rekognition-throttling-provisionedthroughputexceededexception)\n\n5 transactions per second per account in most regions, and 50 in US East (N. Virginia), US West (Oregon) and Europe (Ireland). SearchFaces and SearchFacesByImage have the same defaults, each as a separate quota. The limits are adjustable through Service Quotas.\n\nBoth mean you sent requests faster than the account's quota allows. Handle them the same way: slow down on the client, retry with backoff and jitter, and request a quota increase if the sustained rate you need is above the default.\n\nFor bulk, latency-tolerant jobs such as backfills, yes. Adaptive mode adds a client-side rate limiter that slows new requests after throttling. For user-facing requests, standard mode with few attempts is better, so a user is not kept waiting.\n\nNo. They apply per AWS account, per region, per operation. Every collection, service and script in the same account and region shares the same quota for a given operation.", "url": "https://wpnews.pro/news/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that", "canonical_source": "https://dev.to/ashishsikka/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that-holds-up-399g", "published_at": "2026-10-31 03:10:57+00:00", "updated_at": "2026-10-31 03:19:24.684323+00:00", "lang": "en", "topics": ["ai-tools", "computer-vision", "developer-tools"], "entities": ["AWS", "Amazon Rekognition", "botocore"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that", "markdown": "https://wpnews.pro/news/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that.md", "text": "https://wpnews.pro/news/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that.txt", "jsonld": "https://wpnews.pro/news/rekognition-provisionedthroughputexceededexception-tps-limits-and-backoff-that.jsonld"}}