Not all LLM traffic is equally urgent.
A user waiting for a response cares about the next few seconds, whereas a batch task summarizing thousands of documents usually doesn't.
If both types of work share the same capacity, your infrastructure may treat them the same. Here's how that can become a problem.
Imagine your application handles two kinds of workloads: interactive (user waiting for a response) and batch (background summarization, classification, extraction, or enrichment jobs). Both call the same LLM provider.
When batch processing ramps up, suddenly your interactive requests have to compete with batch traffic that nobody is waiting for.
One obvious solution is to limit the LLM concurrency. However, a concurrency limit knows how many requests are running. It has no idea which requests matter most. So if 20 batch requests acquire all 20 slots, latency-sensitive interactive traffic has nowhere to go.
What about a queue for excess requests? In that case, the interactive request waits behind the batch traffic. Technically, the system is behaving correctly, but the user experience is terrible.
This is one way overload can be deceptive. No crashes, but requests spend more time waiting for capacity. TTFT rises, tail latency worsens, callers time out or retry, retries create more work.
Resource contention can turn into an overload feedback loop.
But it's not just concurrency. Let's consider two requests:
Interactive:
1,000 input tokens
500 maximum output tokens
Batch:
30,000 input tokens
4,000 maximum output tokens
Each consumes one concurrency slot, but they aren't anywhere near the same amount of work.
This is why it's useful to consider both concurrent requests and in-flight token commitments in making admission decisions. Before starting a request, estimate its potential token footprint and reserve that amount. It doesn't have to be exact; a reasonable approximation can still prevent a small number of very large requests from consuming a disproportionate amount of capacity.
You could also prioritize interactive requests, which helps when multiple requests are waiting. However, priority cannot recover capacity already tied up in running requests. If batch work occupies every slot, a high-priority request would still be stuck waiting.
It's important not only to ask which request should run next, but also whether a request should be allowed to consume capacity now. That's why admission control matters.
One approach is to reserve part of the available capacity for different workload classes. If you have 20 available concurrency slots, try something like the following:
Interactive protected capacity: 8
Batch protected capacity: 2
Shared capacity: 10
These numbers can vary depending on workload requirements. The point is that protected capacity can prevent either workload from starving the other. Interactive does not always win.
The goal is to protect latency-sensitive work, ensure background processes run, and share the remaining capacity efficiently. This is workload isolation.
When capacity isn't available, there are often two choices: wait or reject. Waiting sounds friendlier, but if requests accumulate faster than the system can complete them, queueing converts overload into latency.
Sometimes the more reliable behavior is to reject work immediately.
The broader lesson is that overload management encompasses more than limiting how much work enters the system. It's also about deciding which work is allowed to proceed when demand exceeds system capacity.