Static vs. Dynamic vs. Continuous Batching in LLM Inference

IBM's article explains that static, dynamic, and continuous batching are methods to improve GPU utilization in large language model inference, with continuous batching scheduling at the token level to handle variable request lengths. Static batching waits for a full batch, causing latency; dynamic batching adds a timeout to bound wait times; continuous batching is essential for production-scale LLM serving.

In this article, you will learn how static, dynamic, and continuous batching work in LLM inference, and why the differences between them matter at production scale. Topics we will cover include: - Static batching, and why waiting for a full batch is simple but costly under real traffic - Dynamic batching, and how a timeout window fixes the worst of that cost - Continuous batching, and why large language models need scheduling at the token level instead of the request level Introduction Most GPUs serving AI models spend most of their time doing nothing. A request comes in, the model runs it, and the GPU sits idle waiting for the next one while it could have handled several at once for close to the same cost. This gets worse with large language models https://www.ibm.com/think/topics/large-language-models specifically, since one request might finish in a few tokens and another might run for a thousand, so whatever handles the traffic has to deal with highly uneven work, not identical jobs arriving one after another. Batching is how you fix this. Instead of running the model once per request, you group several requests together and run them through the same loaded weights in one pass, turning idle GPU cycles into throughput you’re already paying for. The part that actually matters is how you form those groups, because a batching scheme built for uniform workloads breaks down fast once request lengths stop being predictable. How Static Batching Works Static batching is the most literal version of the idea: wait until a fixed number of requests have arrived, then run them all through the model together as one batch. Nothing starts until the batch is full. If you’ve set a batch size of eight, the seventh request that shows up sits and waits for an eighth to arrive before any of the eight get processed. The mechanism is straightforward. Requests accumulate in a queue, and once the count hits the configured batch size, the server runs a single forward pass across all of them, sharing one weight load across the whole group. This is where the benefit comes from: loading model weights from GPU memory is expensive, and doing it once for k requests instead of k separate times is a large efficiency win. This is why static batching works well for scheduled, latency-tolerant jobs https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/ like running inference over a large stored dataset, where there is no waiting on an individual response and the whole job needs to finish quickly in aggregate. The same design that makes static batching efficient in bulk makes it a poor fit for live traffic: - The first request to arrive has to wait for every other slot in the batch to fill, so latency depends on how fast the rest of the batch shows up, not on how fast that request could have run alone. - Once the batch starts, every request in it is held until the slowest one finishes, so five short requests sitting in a batch with one long one all wait on that single long request. - There is no way to bound how long a request waits before a batch even starts, which makes static batching unsuitable for almost anything with a latency requirement. This last limitation is what dynamic batching solves. How Dynamic Batching Works Dynamic batching keeps the core idea of static batching — grouping requests to share one weight load — but removes the requirement that the group be complete before anything can start. Instead of waiting indefinitely for a batch to fill, the server sets two limits: a maximum batch size and a timeout window . Whichever limit is hit first triggers the batch to run. In practice, this means the server starts a timer the moment the first request in a new batch arrives. If enough requests show up to fill the batch before the timer expires, it runs immediately, the same way static batching would. If the timer runs out first, the server runs whatever has accumulated so far, even a partial batch. A tuned dynamic batching configuration on a Triton inference benchmark https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user guide/performance tuning.html significantly improved throughput while introducing a moderate increase in tail latency. This is a common trade-off in dynamic batching, where higher hardware utilization and request throughput come at the cost of slightly increased response times https://docs.nvidia.com/deeplearning/triton-inference-server/archives/triton inference server 1140/user-guide/docs/model configuration.html section-dynamic-batcher . Dynamic batching therefore gives you increased throughput, and the timeout window decides how much latency you pay for it. - Setting a shorter timeout protects latency at the cost of running smaller, less efficient batches. - Setting a longer timeout improves the odds of a full batch but increases how long early requests sit in the queue. - The maximum batch size still caps how much work can share one weight load, regardless of the timeout setting. This bounds the worst-case wait before a batch starts. It does nothing for the second problem, though. Once a batch begins running, every request in it is still stuck until the slowest one in that batch finishes. For a model like an image generator, where every output takes roughly the same number of steps, that’s rarely an issue. For an large language model, where one request might need five tokens and another might need five hundred, it means short requests routinely wait behind long ones with no way around it. How Continuous Batching Works Continuous batching drops the request as the unit of scheduling and replaces it with the individual decoding step. Rather than waiting for every sequence in a batch to finish before starting the next batch, the server tracks each sequence in the batch independently, one token at a time. Here’s how this plays out during serving. At each decoding iteration, the server runs one forward pass that produces the next token for every active sequence at once. The moment a sequence emits an end-of-sequence token, it is removed from the batch immediately, and a new request from the queue is inserted into that freed slot on the very next iteration. In continuous batching, there is no fixed batch that needs to complete. There is, instead, a rolling set of active sequences whose composition changes on nearly every step, so a GPU running continuous batching is rarely waiting on anything. - A sequence that finishes early frees its slot right away instead of holding up the rest of the batch until the whole group is done. - A new request only has to wait a single iteration to be considered for an open slot, not until an entire batch cycle completes. - Long prompts still create a cost, since a new request’s initial prefill pass is compute-heavy and can delay the decode step for every other active sequence that iteration, which is why chunked prefill https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching/ chunked-prefill splits long prompts into smaller pieces processed across multiple steps instead of all at once. Many inference frameworks built for LLM serving https://handbook.modular.com/getting-started/choosing-the-right-inference-framework/ , including vLLM, TensorRT-LLM under the name in-flight batching https://developer.nvidia.com/blog/nvidia-tensorrt-llm-now-accelerates-encoder-decoder-models-with-in-flight-batching/ , and TGI, default to continuous batching rather than the request-level dynamic batching used for other model types. Continuous batching generally delivers substantially higher throughput than request-level dynamic batching under heavy concurrent workloads. In contrast, request-level dynamic batching can provide a faster time to first token under light workloads, where request traffic is low and processing resources experience minimal contention. Summary The batching techniques we’ve discussed solve the same underlying problem at increasingly finer granularity. Static batching shares one weight load across a group but makes every request in that group wait for the slowest one and for the batch to fill in the first place. Dynamic batching bounds the wait before a batch starts by adding a timeout, but a batch still can’t return early once it begins running. Continuous batching removes the batch as a fixed unit entirely, scheduling at the level of individual decode steps, which is what makes it the standard choice for serving large language models at scale. Here is a review: | Strategy | Scheduling Unit | GPU Idle Time | Latency Behavior | Best Fit | |---|---|---|---|---| | Static batching | Whole batch | High between batches | High, bounded by the slowest request in the batch | Offline jobs with no latency requirement | | Dynamic batching | Whole batch with timeout | Moderate | Bounded by the maximum batch size or timeout window | Fixed-length outputs such as image generation | | Continuous batching | Individual decode step | Low | Variable per request with high overall throughput | Production autoregressive LLM serving | Here are some useful resources you can refer to next: Static, dynamic and continuous batching | LLM Inference Handbook https://handbook.modular.com/inference-optimization/static-dynamic-continuous-batching/ Dynamic Batching & Concurrent Model Execution | NVIDIA Triton Inference Server https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Conceptual Guide/Part 2-improving resource utilization/README.html Continuous vs dynamic batching for AI inference | Baseten https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/ How continuous batching enables 23x throughput in LLM inference while reducing p50 latency | Anyscale https://www.anyscale.com/blog/continuous-batching-llm-inference If you’d like an article exploring the different LLM inference frameworks and the features and optimizations they offer, do let us know in the comments