TurboPrefill

A new scheduling technique called TurboPrefill achieves up to 3.27× prefill speedup in llama.cpp by pipelining prompt microbatches across multiple GPUs, reducing inter-GPU communication bottlenecks. D…

TurboPrefill introduces intra-prompt pipeline scheduling for multi-GPU prefill, achieving up to 2.7× faster performance than llama.cpp on Llama-3-70B by overlapping GPU stage execution. The PoC shows …

A new scheduling technique called TurboPrefill reduces waiting time for Vision Language Models by nearly half, from 9.0 to 4.6 seconds, without changing model weights or architecture. The optimization…