# Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order

> Source: <https://dev.to/quantumbreakz/limiting-concurrency-in-python-asyncio-with-a-semaphore-and-per-task-timeouts-in-the-right-order-2hc0>
> Published: 2026-11-20 14:11:32+00:00

**Recommendation:** Wrap each LLM call in asyncio.Semaphore(limit) with asyncio.timeout() nested inside it and gather with return_exceptions=True, so queued tasks don't burn their timeout before starting.
More in the playbook "Controlling LLM cost": <https://wpnews.pro/playbooks/llm-cost>

`asyncio.gather(*(call(x) for x in items))` starts **every call at once**: with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with **`asyncio.Semaphore(n)`**: each task waits for a slot before it starts. Give each call a **timeout with `asyncio.timeout()`**, and pass **` return_exceptions=True`** so one failure doesn't cancel the rest. The order matters: the timeout must go **inside** the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks **time out without ever running**; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later.

``` python title="bounded.py"

import asyncio

async def map_limit(items, limit: int, worker, timeout: float | None = None):

    """Run worker(item) for every item, at most `limit` at once, each with its own timeout.

    Returns results in input order; failures come back as exceptions instead of cancelling the rest."""

    sem = asyncio.Semaphore(limit)

``` python
async def run(item):
    async with sem:  # wait for a free slot BEFORE starting the timer
        async with asyncio.timeout(timeout):
            return await worker(item)

return await asyncio.gather(*(run(i) for i in items), return_exceptions=True)
* **`asyncio.Semaphore(limit)`** lets `limit` tasks hold it at once; the rest wait in line.
* **`asyncio.timeout()`** (Python 3.11+) cancels the work inside it when time runs out and raises `TimeoutError`. On older versions, `asyncio.wait_for(worker(item), timeout)` does the same job.
* **`return_exceptions=True`** returns each failure in its place in the results list. Without it, the first exception propagates from `gather`, and you lose every result that did succeed.

All the coroutines are still **created** up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an `asyncio.Queue` instead, so you don't hold that many coroutine objects in memory.

## Does it work?

The test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises `ValueError`:

``` python title="Results"
results = await map_limit(range(20), 4, fake_llm, timeout=0.2)
# peak concurrency: 4
# [0, 1, 4, 9, 16, 25, 36, TimeoutError, 64, 81, 100, 121, 144, ValueError, 196, 225, 256, 289, 324, 361]
```

Concurrency never exceeds 4, results stay in input order, the slow call becomes a `TimeoutError` and the bad one a `ValueError`, and the other 18 succeed. The whole batch takes about 0.23 seconds.

This version looks almost identical:

``` python title="Timeout outside the semaphore (wrong)"

async def wrong(n):

    async with asyncio.timeout(0.05):

        async with sem:  # Semaphore(2)

            await asyncio.sleep(0.03)  # the actual work: 30 ms

            return n

```
Each task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms **waiting for a slot**, and the timeout counts that wait. In the test, **8 of 10 timed out**, and 6 of those 8 never started their work at all. The same work through `map_limit` with the same limit and timeout completes all 10.

This matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer.

## Choosing the numbers

* **The limit** bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in [rate limiting LLM API calls](https://aliahmed.si/blog/llm-rate-limiter-token-bucket-nodejs/) translates directly to Python.
* **The timeout** should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one.
* **Retries** go inside `worker`, so a retried call keeps its slot. Use [backoff with jitter](https://aliahmed.si/blog/llm-api-retries-backoff-fallbacks/), and don't retry timeouts on requests that change state unless they're idempotent.

The same pattern in JavaScript is [a 15-line promise pool](https://aliahmed.si/blog/limit-concurrent-promises-nodejs/).

## Summary

* `gather` alone starts everything at once; a `Semaphore` caps how many run.
* Put `asyncio.timeout()` **inside** `async with sem`, so queueing time doesn't count.
* Use `return_exceptions=True` to keep every result and see every failure.
* Pair the concurrency limit with a rate limiter for APIs with per-minute limits.

---

Written by Ali Ahmed · [Original article](https://aliahmed.si/blog/asyncio-semaphore-timeout-concurrency/)
```


