Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order A developer published a tested Python pattern for bounding concurrent LLM API calls using asyncio.Semaphore combined with asyncio.timeout(), showing that the timeout must be placed inside the semaphore rather than outside it. In a test of 10 tasks with 2 slots and a 50 ms timeout, placing the timeout outside caused 8 of 10 tasks to time out, 6 of which never began their work, because queue wait time counted against the clock. The correct ordering, paired with gather's return_exceptions=True, kept peak concurrency at the configured limit, preserved input order, and returned individual failures as exceptions while the remaining calls succeeded. asyncio.gather call x for x in items starts every call at once : with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with asyncio.Semaphore n : each task waits for a slot before it starts. Give each call a timeout with asyncio.timeout , and pass return exceptions=True so one failure doesn't cancel the rest. The order matters: the timeout must go inside the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks time out without ever running ; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later. python title="bounded.py" import asyncio async def map limit items, limit: int, worker, timeout: float | None = None : """Run worker item for every item, at most limit at once, each with its own timeout. Returns results in input order; failures come back as exceptions instead of cancelling the rest.""" sem = asyncio.Semaphore limit python async def run item : async with sem: wait for a free slot BEFORE starting the timer async with asyncio.timeout timeout : return await worker item return await asyncio.gather run i for i in items , return exceptions=True asyncio.Semaphore limit lets limit tasks hold it at once; the rest wait in line. asyncio.timeout Python 3.11+ cancels the work inside it when time runs out and raises TimeoutError . On older versions, asyncio.wait for worker item , timeout does the same job. return exceptions=True returns each failure in its place in the results list. Without it, the first exception propagates from gather , and you lose every result that did succeed. All the coroutines are still created up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an asyncio.Queue instead, so you don't hold that many coroutine objects in memory. Does it work? The test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises ValueError : python title="Results" results = await map limit range 20 , 4, fake llm, timeout=0.2 peak concurrency: 4 0, 1, 4, 9, 16, 25, 36, TimeoutError, 64, 81, 100, 121, 144, ValueError, 196, 225, 256, 289, 324, 361 Concurrency never exceeds 4, results stay in input order, the slow call becomes a TimeoutError and the bad one a ValueError , and the other 18 succeed. The whole batch takes about 0.23 seconds. This version looks almost identical: python title="Timeout outside the semaphore wrong " async def wrong n : async with asyncio.timeout 0.05 : async with sem: Semaphore 2 await asyncio.sleep 0.03 the actual work: 30 ms return n Each task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms waiting for a slot , and the timeout counts that wait. In the test, 8 of 10 timed out , and 6 of those 8 never started their work at all. The same work through map limit with the same limit and timeout completes all 10. This matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer. Choosing the numbers The limit bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in rate limiting LLM API calls https://aliahmed.si/blog/llm-rate-limiter-token-bucket-nodejs/ translates directly to Python. The timeout should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one. Retries go inside worker , so a retried call keeps its slot. Use backoff with jitter https://aliahmed.si/blog/llm-api-retries-backoff-fallbacks/ , and don't retry timeouts on requests that change state unless they're idempotent. The same pattern in JavaScript is a 15-line promise pool https://aliahmed.si/blog/limit-concurrent-promises-nodejs/ . Summary gather alone starts everything at once; a Semaphore caps how many run. Put asyncio.timeout inside async with sem , so queueing time doesn't count. Use return exceptions=True to keep every result and see every failure. Pair the concurrency limit with a rate limiter for APIs with per-minute limits. --- Written by Ali Ahmed ยท Original article https://aliahmed.si/blog/asyncio-semaphore-timeout-concurrency/