Inspiration #
Large language models running across multiple GPUs are often limited not only by computation, but by communication between GPUs. This becomes especially visible on systems without NVLink or with slow PCIe connections.
TurboPrefill by Trykhlieb was created to reduce these bottlenecks during the prefill stage.
What it does #
TurboPrefill changes the scheduling of prompt microbatches across multiple GPUs.
Instead of processing each microbatch through all GPUs sequentially, it pipelines microbatches so that several GPUs can perform useful work at the same time. Decode behavior remains unchanged.
The original implementation was developed for llama.cpp and tested on different GPU generations, large models, and systems with very limited inter-GPU bandwidth.
What I built during OpenAI Build Week #
During OpenAI Build Week, I created an experimental port of TurboPrefill to ik_llama.cpp by Ivan Kawrakow.
The port is based on ik_llama.cpp commit 6198a356
.
The published materials include:
- modified source files
- clean and diagnostic versions
- benchmark scripts
- test contexts
- baseline and TurboPrefill benchmark results
How Codex and GPT-5.6 were used #
Codex and GPT-5.6 assisted with the initial port, code adaptation, diagnostic logging, execution-flow analysis, error localization, cleanup, and preparation of reusable diagnostic files.
Codex did not complete the entire task automatically, but it significantly accelerated the development and debugging process.
Results #
The experimental ik_llama.cpp port shows higher prefill performance across all tested context sizes.
For GPT-OSS-20B, at a context length of 1,024 tokens, prefill performance increased from approximately 184 to 602 tokens per second. The main TurboPrefill implementation for llama.cpp has also demonstrated substantial improvements on multiple GPU configurations, especially where inter-GPU communication is the primary bottleneck.
Challenges #
The main challenge was preserving correct execution while changing microbatch scheduling inside a complex multi-GPU inference engine.
Extended logging was used to trace execution, identify porting errors, and verify the modified implementation.
What I learned #
The work confirmed that pipeline scheduling can substantially improve multi-GPU prefill performance, particularly on heterogeneous or bandwidth-limited systems.
It also demonstrated how Codex can accelerate low-level systems work when combined with direct testing, benchmarking, and manual engineering analysis.