{"slug": "when-batch-traffic-competes-with-interactive-llm-requests", "title": "When batch traffic competes with interactive LLM requests", "summary": "A developer explains how batch LLM traffic can degrade interactive request latency when both share the same infrastructure, and proposes admission control with protected capacity and token-aware reservations to prevent starvation. The post highlights that concurrency limits alone are insufficient and that workload isolation is key to maintaining responsiveness.", "body_md": "Not all LLM traffic is equally urgent.\n\nA user waiting for a response cares about the next few seconds, whereas a batch task summarizing thousands of documents usually doesn't.\n\nIf both types of work share the same capacity, your infrastructure may treat them the same.\n\nHere's how that can become a problem.\n\nImagine your application handles two kinds of workloads: interactive (user waiting for a response) and batch (background summarization, classification, extraction, or enrichment jobs). Both call the same LLM provider.\n\nWhen batch processing ramps up, suddenly your interactive requests have to compete with batch traffic that nobody is waiting for.\n\nOne obvious solution is to limit the LLM concurrency. However, a concurrency limit knows how many requests are running. It has no idea which requests matter most. So if 20 batch requests acquire all 20 slots, latency-sensitive interactive traffic has nowhere to go.\n\nWhat about a queue for excess requests? In that case, the interactive request waits behind the batch traffic. Technically, the system is behaving correctly, but the user experience is terrible.\n\nThis is one way overload can be deceptive. No crashes, but requests spend more time waiting for capacity. TTFT rises, tail latency worsens, callers time out or retry, retries create more work.\n\nResource contention can turn into an overload feedback loop.\n\nBut it's not just concurrency. Let's consider two requests:\n\nInteractive:\n\n1,000 input tokens\n\n500 maximum output tokens\n\nBatch:\n\n30,000 input tokens\n\n4,000 maximum output tokens\n\nEach consumes one concurrency slot, but they aren't anywhere near the same amount of work.\n\nThis is why it's useful to consider both concurrent requests and in-flight token commitments in making admission decisions. Before starting a request, estimate its potential token footprint and reserve that amount. It doesn't have to be exact; a reasonable approximation can still prevent a small number of very large requests from consuming a disproportionate amount of capacity.\n\nYou could also prioritize interactive requests, which helps when multiple requests are waiting. However, priority cannot recover capacity already tied up in running requests. If batch work occupies every slot, a high-priority request would still be stuck waiting.\n\nIt's important not only to ask which request should run next, but also whether a request should be allowed to consume capacity now. That's why admission control matters.\n\nOne approach is to reserve part of the available capacity for different workload classes. If you have 20 available concurrency slots, try something like the following:\n\nInteractive protected capacity: 8\n\nBatch protected capacity: 2\n\nShared capacity: 10\n\nThese numbers can vary depending on workload requirements. The point is that protected capacity can prevent either workload from starving the other. Interactive does not always win.\n\nThe goal is to protect latency-sensitive work, ensure background processes run, and share the remaining capacity efficiently. This is workload isolation.\n\nWhen capacity isn't available, there are often two choices: wait or reject. Waiting sounds friendlier, but if requests accumulate faster than the system can complete them, queueing converts overload into latency.\n\nSometimes the more reliable behavior is to reject work immediately.\n\nThe broader lesson is that overload management encompasses more than limiting how much work enters the system. It's also about deciding which work is allowed to proceed when demand exceeds system capacity.", "url": "https://wpnews.pro/news/when-batch-traffic-competes-with-interactive-llm-requests", "canonical_source": "https://dev.to/janbalangue/when-batch-traffic-competes-with-interactive-llm-requests-5684", "published_at": "2026-08-27 22:12:00+00:00", "updated_at": "2026-08-27 22:48:45.560400+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/when-batch-traffic-competes-with-interactive-llm-requests", "markdown": "https://wpnews.pro/news/when-batch-traffic-competes-with-interactive-llm-requests.md", "text": "https://wpnews.pro/news/when-batch-traffic-competes-with-interactive-llm-requests.txt", "jsonld": "https://wpnews.pro/news/when-batch-traffic-competes-with-interactive-llm-requests.jsonld"}}