cd /news/large-language-models/limiting-concurrency-in-python-async… · home › topics › large-language-models › article
[ARTICLE · art-173260] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order

A developer published a tested Python pattern for bounding concurrent LLM API calls using asyncio.Semaphore combined with asyncio.timeout(), showing that the timeout must be placed inside the semaphore rather than outside it. In a test of 10 tasks with 2 slots and a 50 ms timeout, placing the timeout outside caused 8 of 10 tasks to time out, 6 of which never began their work, because queue wait time counted against the clock. The correct ordering, paired with gather's return_exceptions=True, kept peak concurrency at the configured limit, preserved input order, and returned individual failures as exceptions while the remaining calls succeeded.

recommendation

Wrap each LLM call in asyncio.Semaphore(limit) with asyncio.timeout() nested inside it and gather with return_exceptions=True, so queued tasks don't burn their timeout before starting.

more in playbook: Controlling LLM cost
by read4 min views3 publishedNov 20, 2026

asyncio.gather(*(call(x) for x in items)) starts every call at once: with a thousand prompts, that's a thousand simultaneous requests to an LLM API, and an immediate wall of 429s. Cap it with asyncio.Semaphore(n): each task waits for a slot before it starts. Give each call a timeout with asyncio.timeout(), and pass return_exceptions=True so one failure doesn't cancel the rest. The order matters: the timeout must go inside the semaphore. Put it outside, and the clock starts while the task is still queueing, so tasks time out without ever running; in my test, 8 of 10 did. Here's the correct version in a dozen lines, tested on Python 3.11 and later.


import asyncio

async def map_limit(items, limit: int, worker, timeout: float | None = None):

    """Run worker(item) for every item, at most `limit` at once, each with its own timeout.

    Returns results in input order; failures come back as exceptions instead of cancelling the rest."""

    sem = asyncio.Semaphore(limit)

``` python
async def run(item):
    async with sem:  # wait for a free slot BEFORE starting the timer
        async with asyncio.timeout(timeout):
            return await worker(item)

return await asyncio.gather(*(run(i) for i in items), return_exceptions=True)
* **`asyncio.Semaphore(limit)`** lets `limit` tasks hold it at once; the rest wait in line.
* **`asyncio.timeout()`** (Python 3.11+) cancels the work inside it when time runs out and raises `TimeoutError`. On older versions, `asyncio.wait_for(worker(item), timeout)` does the same job.
* **`return_exceptions=True`** returns each failure in its place in the results list. Without it, the first exception propagates from `gather`, and you lose every result that did succeed.

All the coroutines are still **created** up front; only their execution is limited. For hundreds of thousands of items, feed a fixed number of worker tasks from an `asyncio.Queue` instead, so you don't hold that many coroutine objects in memory.

## Does it work?

The test runs 20 fake LLM calls with a limit of 4 and a 0.2-second timeout. Call 7 takes 0.5 seconds, and call 13 raises `ValueError`:

``` python title="Results"
results = await map_limit(range(20), 4, fake_llm, timeout=0.2)

Concurrency never exceeds 4, results stay in input order, the slow call becomes a TimeoutError and the bad one a ValueError, and the other 18 succeed. The whole batch takes about 0.23 seconds.

This version looks almost identical:


async def wrong(n):

    async with asyncio.timeout(0.05):

        async with sem:  # Semaphore(2)

            await asyncio.sleep(0.03)  # the actual work: 30 ms

            return n

Each task does 30 ms of work and has a 50 ms timeout, which should be plenty. But with 10 tasks and 2 slots, most tasks spend longer than 50 ms waiting for a slot, and the timeout counts that wait. In the test, 8 of 10 timed out, and 6 of those 8 never started their work at all. The same work through map_limit with the same limit and timeout completes all 10.

This matters most when the timeout is your signal that a provider is slow. With the timeout outside, a long queue on your side looks like a slow provider, which can trigger retries and fallbacks that make the queue longer.

Choosing the numbers #

  • The limit bounds concurrent requests, not their rate. LLM APIs limit requests and tokens per minute, so pair the semaphore with a rate limiter; the token-bucket design in rate limiting LLM API calls translates directly to Python.
  • The timeout should cover your slowest normal call. For streaming LLM responses, a time-to-first-token timeout is more useful than a total one.
  • Retries go inside worker, so a retried call keeps its slot. Use backoff with jitter, and don't retry timeouts on requests that change state unless they're idempotent.

The same pattern in JavaScript is a 15-line promise pool.

Summary #

  • gather alone starts everything at once; a Semaphore caps how many run.
  • Put asyncio.timeout() inside async with sem, so queueing time doesn't count.
  • Use return_exceptions=True to keep every result and see every failure.
  • Pair the concurrency limit with a rate limiter for APIs with per-minute limits.

Written by Ali Ahmed · Original article


── more in #large-language-models 4 stories · sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/limiting-concurrency…] indexed:0 read:4min 2026-11-20 · —