A single GPU serving an LLM does not process one request at a time. It processes one step at a time, and a step can belong to any request that happens to be ready. Here’s how it goes …
Generating a response is not one big computation; it is a loop. Each iteration of that loop (a step) takes the current sequence, runs one forward pass, and produces exactly one new token. So, a response of 500 tokens is 500 separate steps.
This is where continuous batching kicks in.
Continuous batching decouples the step from the request. After every single token generation step, the scheduler checks the batch. Any sequence that has finished (hit an EOS token) is evicted immediately, and any new request waiting in the queue is slotted into that now-open slot, right in the middle of everyone else’s generation.
So at any given moment, the GPU’s batch might contain token 3 of a brand new request sitting right next to token 400 of a long-running one. The GPU does not know or care; it just runs a forward pass over whatever sequences are currently assigned to its batch slots.
Now, this is only possible because attention and the KV cache are computed per-sequence.
Each request’s keys and values live in their own memory region (this is where PagedAttention comes in, to manage that memory efficiently), so batching different requests together at different stages of generation does not corrupt anyone’s context.
Thus, GPU utilization stays high and average latency drops, because no one is stuck waiting behind a single slow request that happened to start first.