{"slug": "limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in", "title": "Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order", "summary": "A developer published a tested Python pattern for bounding concurrent LLM API calls using asyncio.Semaphore combined with asyncio.timeout(), showing that the timeout must be placed inside the semaphore rather than outside it. In a test of 10 tasks with 2 slots and a 50 ms timeout, placing the timeout outside caused 8 of 10 tasks to time out, 6 of which never began their work, because queue wait time counted against the clock. The correct ordering, paired with gather's return_exceptions=True, kept peak concurrency at the configured limit, preserved input order, and returned individual failures as exceptions while the remaining calls succeeded.", "body_md": "`asyncio.gather(*(call(x) for x in items))` starts **every call at once**: with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with **`asyncio.Semaphore(n)`**: each task waits for a slot before it starts. Give each call a **timeout with `asyncio.timeout()`**, and pass **` return_exceptions=True`** so one failure doesn't cancel the rest. The order matters: the timeout must go **inside** the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks **time out without ever running**; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later.\n\n``` python title=\"bounded.py\"\n\nimport asyncio\n\nasync def map_limit(items, limit: int, worker, timeout: float | None = None):\n\n    \"\"\"Run worker(item) for every item, at most `limit` at once, each with its own timeout.\n\n    Returns results in input order; failures come back as exceptions instead of cancelling the rest.\"\"\"\n\n    sem = asyncio.Semaphore(limit)\n\n``` python\nasync def run(item):\n    async with sem:  # wait for a free slot BEFORE starting the timer\n        async with asyncio.timeout(timeout):\n            return await worker(item)\n\nreturn await asyncio.gather(*(run(i) for i in items), return_exceptions=True)\n* **`asyncio.Semaphore(limit)`** lets `limit` tasks hold it at once; the rest wait in line.\n* **`asyncio.timeout()`** (Python 3.11+) cancels the work inside it when time runs out and raises `TimeoutError`. On older versions, `asyncio.wait_for(worker(item), timeout)` does the same job.\n* **`return_exceptions=True`** returns each failure in its place in the results list. Without it, the first exception propagates from `gather`, and you lose every result that did succeed.\n\nAll the coroutines are still **created** up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an `asyncio.Queue` instead, so you don't hold that many coroutine objects in memory.\n\n## Does it work?\n\nThe test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises `ValueError`:\n\n``` python title=\"Results\"\nresults = await map_limit(range(20), 4, fake_llm, timeout=0.2)\n# peak concurrency: 4\n# [0, 1, 4, 9, 16, 25, 36, TimeoutError, 64, 81, 100, 121, 144, ValueError, 196, 225, 256, 289, 324, 361]\n```\n\nConcurrency never exceeds 4, results stay in input order, the slow call becomes a `TimeoutError` and the bad one a `ValueError`, and the other 18 succeed. The whole batch takes about 0.23 seconds.\n\nThis version looks almost identical:\n\n``` python title=\"Timeout outside the semaphore (wrong)\"\n\nasync def wrong(n):\n\n    async with asyncio.timeout(0.05):\n\n        async with sem:  # Semaphore(2)\n\n            await asyncio.sleep(0.03)  # the actual work: 30 ms\n\n            return n\n\n```\nEach task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms **waiting for a slot**, and the timeout counts that wait. In the test, **8 of 10 timed out**, and 6 of those 8 never started their work at all. The same work through `map_limit` with the same limit and timeout completes all 10.\n\nThis matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer.\n\n## Choosing the numbers\n\n* **The limit** bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in [rate limiting LLM API calls](https://aliahmed.si/blog/llm-rate-limiter-token-bucket-nodejs/) translates directly to Python.\n* **The timeout** should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one.\n* **Retries** go inside `worker`, so a retried call keeps its slot. Use [backoff with jitter](https://aliahmed.si/blog/llm-api-retries-backoff-fallbacks/), and don't retry timeouts on requests that change state unless they're idempotent.\n\nThe same pattern in JavaScript is [a 15-line promise pool](https://aliahmed.si/blog/limit-concurrent-promises-nodejs/).\n\n## Summary\n\n* `gather` alone starts everything at once; a `Semaphore` caps how many run.\n* Put `asyncio.timeout()` **inside** `async with sem`, so queueing time doesn't count.\n* Use `return_exceptions=True` to keep every result and see every failure.\n* Pair the concurrency limit with a rate limiter for APIs with per-minute limits.\n\n---\n\nWritten by Ali Ahmed · [Original article](https://aliahmed.si/blog/asyncio-semaphore-timeout-concurrency/)\n```\n\n", "url": "https://wpnews.pro/news/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in", "canonical_source": "https://dev.to/quantumbreakz/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in-the-right-order-2hc0", "published_at": "2026-11-20 14:11:32+00:00", "updated_at": "2026-11-20 14:13:57.700088+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": [], "also_reported_by": [], "takeaway": "Place asyncio.timeout() inside the semaphore so queue wait time isn't counted against the call's deadline; with the timeout outside, 8 of 10 tasks timed out and 6 never ran.", "recommendation": {"advice": "Wrap each LLM call in asyncio.Semaphore(limit) with asyncio.timeout() nested inside it and gather with return_exceptions=True, so queued tasks don't burn their timeout before starting.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in", "markdown": "https://wpnews.pro/news/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in.md", "text": "https://wpnews.pro/news/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in.txt", "jsonld": "https://wpnews.pro/news/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in.jsonld"}}