18:01
2026-06-18
pub.towardsai.net
large-language-models
Continuous Batching: How to Keep Your GPU Actually Busy
Continuous batching, introduced in the 2022 Orca paper, improves GPU utilization during LLM inference by dynamically updating the batch at each iteration, freeing slots as requests finish and immediat…