cd /news/ai-infrastructure/wavepp-high-throughput-pipeline-para… · home › topics › ai-infrastructure › article
[ARTICLE · art-173148] src=arxiv.org ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

WavePP: High-Throughput Pipeline Parallel LLM Prefill Under Prefix Reuse

A September 28, 2026 arXiv paper introduces WavePP, a prefill runtime built on TensorRT-LLM that overlaps request admission with pipeline execution to reuse prefixes across all pipeline stages. WavePP improved TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7, reaching 2.91x and 2.02x throughput at concurrency 128 with high cache reuse, and posted the highest measured throughput in all 18 Kimi K3 settings at concurrency eight or higher against tensor/expert-parallel and pipeline-parallel baselines from TRT-LLM, SGLang, and vLLM.

recommendation

When serving prefill with pipeline parallelism and prefix caching, overlap cross-stage prefix lookup and cache reservation with in-flight requests instead of serializing admission, which raised throughput up to 2.91x at concurrency 128.

more in playbook: Inference and self-hosting
read2 min views1 publishedNov 20, 2026
WavePP: High-Throughput Pipeline Parallel LLM Prefill Under Prefix Reuse
Image: source
  [Submitted on 28 Sep 2026]


[View PDF](https://arxiv.org/pdf/2609.35263)

[HTML (experimental)](https://arxiv.org/html/2609.35263v1)

Abstract:Pipeline parallelism can improve prefill throughput by processing multiple request chunks concurrently across different stages of the model. However, keeping the pipeline fully utilized requires efficient scheduling and request preparation. In systems where stages retain and evict cache state independently, a local cache hit does not guarantee that the same prefix can be reused across the pipeline. Here, coordination overhead can impede request admission cadence and thus reduce overall throughput. In this paper, we present WavePP, a prefill runtime built on top of TensorRT-LLM that addresses these challenges by overlapping request admission with pipeline execution. WavePP asynchronously finds a prefix that can be reused across all stages, protects the cached state, and reserves space for the remaining input while earlier requests continue to execute. It subsequently plans the chunk sizes of each request dynamically to maximize pipeline fill. Each stage then completes the local preparation before executing the request. In the same system and pipeline topology, WavePP improves TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7. At concurrency 128 with high cache reuse, these changes increase throughput by factors of 2.91 and 2.02, respectively. Across 28 Kimi K3 settings, WavePP also has the highest measured throughput in all 18 settings at concurrency eight or higher, compared with tensor/expert-parallel and pipeline-parallel baselines from TRT-LLM, SGLang, and vLLM.

Additional Features

References & Citations

...

Bibliographic Explorer

(What is the Explorer?) Connected Papers

(What is Connected Papers?) Litmaps

(What is Litmaps?) scite Smart Citations

(What are Smart Citations?) alphaXiv

(What is alphaXiv?) CatalyzeX Code Finder for Papers

(What is CatalyzeX?) DagsHub

(What is DagsHub?) Gotit.pub

(What is GotitPub?) Hugging Face

(What is Huggingface?) ScienceCast

(What is ScienceCast?) Influence Flower

(What are Influence Flowers?) CORE Recommender

(What is CORE?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @wavepp 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/wavepp-high-throughp…] indexed:0 read:2min 2026-11-20 · —