One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

A paper on arXiv (2611.13188v1) presents NVLinkOneSided, a one-sided expert-parallel all-to-all for Mixture-of-Experts inference within a single NVLink domain that completes dispatch and combine 1.2-3.0× faster than DeepEP on Blackwell GPUs, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4× faster than DeepEP with DeepGEMM and outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, the authors report. The design pushes tokens without first exchanging metadata into rank-major buffers on target GPUs, where grouped GEMMs read them in place, and offers Fence and Sentinel synchronization protocols for high-throughput and low-latency scenarios respectively.

arXiv:2611.13188v1 Announce Type: cross Abstract: Expert parallelism EP shards the experts of a Mixture-of-Experts MoE layer across GPUs, so every MoE layer requires two all-to-all communications, namely dispatch and combine. Compared with training, inference processes relatively few tokens per GPU at a time, and at small batch sizes fixed costs that do not grow with the number of tokens, rather than data movement, dominate the latency. We present NVLinkOneSided, a one-sided EP all-to-all for GPUs within a single NVLink domain. Each source rank pushes its tokens, without first exchanging metadata, into rank-major buffers on the target GPUs, and the grouped GEMMs read them in place instead of from a per-expert copy. We provide two protocols for the synchronization that the communication requires, namely Fence and Sentinel, for high-throughput and low-latency scenarios respectively. On Blackwell GPUs, NVLinkOneSided completes dispatch and combine 1.2-3.0$\times$ faster than DeepEP, with the largest speedups at small batch sizes. The resulting MoE layer is up to 2.4$\times$ faster than DeepEP with DeepGEMM. It even outperforms the fused DeepGEMM MegaMoE at up to 128 tokens per GPU, making it still competitive for decoding. We expose NVLinkOneSided through a modular interface, so that it is agnostic to the activation quantization format, decoupled from the MoE compute backend and easy to integrate into inference frameworks.