{"slug": "concurrency-limits-arent-enough-for-llm-apis", "title": "Concurrency Limits Aren’t Enough for LLM APIs", "summary": "A developer has released async-bulkhead-llm, an open-source TypeScript library that adds token-aware admission control to LLM APIs. Unlike traditional concurrency limiters that treat all requests equally, the library bounds both concurrent requests and estimated in-flight tokens, preventing large batch requests from consuming disproportionate provider capacity. The project highlights that concurrency is only one dimension of load for LLM systems.", "body_md": "A concurrency limit treats every request as roughly equal.\n\nFor LLM workloads, that assumption breaks down.\n\nA request with a 500-token budget and one with a 30,000-token budget might both occupy a single concurrency slot, but they can represent very different amounts of provider capacity.\n\nThat’s the problem I built async-bulkhead-llm to address.\n\nInstead of limiting only concurrent requests, it can bound two things at once:\n\nconcurrent requests\n\n+\n\nestimated in-flight tokens\n\nA request is admitted only when both budgets have enough capacity.\n\nConceptually:\n\n``` js\nconst bulkhead = new AsyncBulkhead({\n  maxConcurrent: 8,\n  maxInFlightTokens: 40_000,\n});\n```\n\nThat gives you a bulkhead around an LLM provider that accounts for workload size rather than simply request count.\n\nThis becomes especially useful when interactive and large batch requests share the same upstream model. A conventional concurrency limiter can still allow a few very large requests to consume most of the useful capacity.\n\nToken-aware admission gives you another control surface.\n\nasync-bulkhead-llm is an open-source TypeScript/npm library I’ve been developing specifically around this problem.\n\nThe broader lesson has been simple: for LLM systems, concurrency is only one dimension of load.\n\nIf you're building production LLM infrastructure, I'd be interested in how you're handling admission control and overload protection.", "url": "https://wpnews.pro/news/concurrency-limits-arent-enough-for-llm-apis", "canonical_source": "https://dev.to/janbalangue/concurrency-limits-arent-enough-for-llm-apis-4lgm", "published_at": "2026-08-25 03:27:59+00:00", "updated_at": "2026-08-25 04:43:33.859004+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["async-bulkhead-llm"], "alternates": {"html": "https://wpnews.pro/news/concurrency-limits-arent-enough-for-llm-apis", "markdown": "https://wpnews.pro/news/concurrency-limits-arent-enough-for-llm-apis.md", "text": "https://wpnews.pro/news/concurrency-limits-arent-enough-for-llm-apis.txt", "jsonld": "https://wpnews.pro/news/concurrency-limits-arent-enough-for-llm-apis.jsonld"}}