{"slug": "turboprefill-3-27x-prefill-speedup-in-llama-cpp", "title": "TurboPrefill: 3.27× Prefill Speedup in Llama.cpp", "summary": "A new scheduling technique called TurboPrefill achieves up to 3.27× prefill speedup in llama.cpp by pipelining prompt microbatches across multiple GPUs, reducing inter-GPU communication bottlenecks. Developer Trykhlieb created the original implementation for llama.cpp, and during OpenAI Build Week ported it to ik_llama.cpp, where GPT-OSS-20B prefill performance increased from 184 to 602 tokens per second at 1,024-token context. The work demonstrates that pipeline scheduling can substantially improve multi-GPU prefill performance on bandwidth-limited systems.", "body_md": "## Inspiration\n\nLarge language models running across multiple GPUs are often limited not only by computation, but by communication between GPUs. This becomes especially visible on systems without NVLink or with slow PCIe connections.\n\nTurboPrefill by Trykhlieb was created to reduce these bottlenecks during the prefill stage.\n\n## What it does\n\nTurboPrefill changes the scheduling of prompt microbatches across multiple GPUs.\n\nInstead of processing each microbatch through all GPUs sequentially, it pipelines microbatches so that several GPUs can perform useful work at the same time. Decode behavior remains unchanged.\n\nThe original implementation was developed for llama.cpp and tested on different GPU generations, large models, and systems with very limited inter-GPU bandwidth.\n\n## What I built during OpenAI Build Week\n\nDuring OpenAI Build Week, I created an experimental port of TurboPrefill to ik_llama.cpp by Ivan Kawrakow.\n\nThe port is based on ik_llama.cpp commit `6198a356`\n\n.\n\nThe published materials include:\n\n- modified source files\n- clean and diagnostic versions\n- benchmark scripts\n- test contexts\n- baseline and TurboPrefill benchmark results\n\n## How Codex and GPT-5.6 were used\n\nCodex and GPT-5.6 assisted with the initial port, code adaptation, diagnostic logging, execution-flow analysis, error localization, cleanup, and preparation of reusable diagnostic files.\n\nCodex did not complete the entire task automatically, but it significantly accelerated the development and debugging process.\n\n## Results\n\nThe experimental ik_llama.cpp port shows higher prefill performance across all tested context sizes.\n\nFor GPT-OSS-20B, at a context length of 1,024 tokens, prefill performance increased from approximately 184 to 602 tokens per second.\n\nThe main TurboPrefill implementation for llama.cpp has also demonstrated substantial improvements on multiple GPU configurations, especially where inter-GPU communication is the primary bottleneck.\n\n## Challenges\n\nThe main challenge was preserving correct execution while changing microbatch scheduling inside a complex multi-GPU inference engine.\n\nExtended logging was used to trace execution, identify porting errors, and verify the modified implementation.\n\n## What I learned\n\nThe work confirmed that pipeline scheduling can substantially improve multi-GPU prefill performance, particularly on heterogeneous or bandwidth-limited systems.\n\nIt also demonstrated how Codex can accelerate low-level systems work when combined with direct testing, benchmarking, and manual engineering analysis.", "url": "https://wpnews.pro/news/turboprefill-3-27x-prefill-speedup-in-llama-cpp", "canonical_source": "https://devpost.com/software/turboprefill", "published_at": "2026-07-25 07:32:37+00:00", "updated_at": "2026-07-25 07:52:28.241864+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["TurboPrefill", "Trykhlieb", "llama.cpp", "ik_llama.cpp", "Ivan Kawrakow", "OpenAI Build Week", "Codex", "GPT-5.6"], "alternates": {"html": "https://wpnews.pro/news/turboprefill-3-27x-prefill-speedup-in-llama-cpp", "markdown": "https://wpnews.pro/news/turboprefill-3-27x-prefill-speedup-in-llama-cpp.md", "text": "https://wpnews.pro/news/turboprefill-3-27x-prefill-speedup-in-llama-cpp.txt", "jsonld": "https://wpnews.pro/news/turboprefill-3-27x-prefill-speedup-in-llama-cpp.jsonld"}}