asyncio.gather(*(call(x) for x in items)) starts every call at once: with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with asyncio.Semaphore(n): each task waits for a slot before it starts. Give each call a timeout with asyncio.timeout(), and pass return_exceptions=True so one failure doesn't cancel the rest. The order matters: the timeout must go inside the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks time out without ever running; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later.
import asyncio
async def map_limit(items, limit: int, worker, timeout: float | None = None):
"""Run worker(item) for every item, at most `limit` at once, each with its own timeout.
Returns results in input order; failures come back as exceptions instead of cancelling the rest."""
sem = asyncio.Semaphore(limit)
``` python
async def run(item):
async with sem: # wait for a free slot BEFORE starting the timer
async with asyncio.timeout(timeout):
return await worker(item)
return await asyncio.gather(*(run(i) for i in items), return_exceptions=True)
* **`asyncio.Semaphore(limit)`** lets `limit` tasks hold it at once; the rest wait in line.
* **`asyncio.timeout()`** (Python 3.11+) cancels the work inside it when time runs out and raises `TimeoutError`. On older versions, `asyncio.wait_for(worker(item), timeout)` does the same job.
* **`return_exceptions=True`** returns each failure in its place in the results list. Without it, the first exception propagates from `gather`, and you lose every result that did succeed.
All the coroutines are still **created** up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an `asyncio.Queue` instead, so you don't hold that many coroutine objects in memory.
## Does it work?
The test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises `ValueError`:
``` python title="Results"
results = await map_limit(range(20), 4, fake_llm, timeout=0.2)
Concurrency never exceeds 4, results stay in input order, the slow call becomes a TimeoutError and the bad one a ValueError, and the other 18 succeed. The whole batch takes about 0.23 seconds.
This version looks almost identical:
async def wrong(n):
async with asyncio.timeout(0.05):
async with sem: # Semaphore(2)
await asyncio.sleep(0.03) # the actual work: 30 ms
return n
Each task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms waiting for a slot, and the timeout counts that wait. In the test, 8 of 10 timed out, and 6 of those 8 never started their work at all. The same work through map_limit with the same limit and timeout completes all 10.
This matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer.
Choosing the numbers #
- The limit bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in rate limiting LLM API calls translates directly to Python.
- The timeout should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one.
- Retries go inside
worker, so a retried call keeps its slot. Use backoff with jitter, and don't retry timeouts on requests that change state unless they're idempotent.
The same pattern in JavaScript is a 15-line promise pool.
Summary #
gatheralone starts everything at once; aSemaphorecaps how many run.- Put
asyncio.timeout()insideasync with sem, so queueing time doesn't count. - Use
return_exceptions=Trueto keep every result and see every failure. - Pair the concurrency limit with a rate limiter for APIs with per-minute limits.
Written by Ali Ahmed · Original article