{"slug": "the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler", "title": "The Hidden Cost of MoE Diffusion Decoding: Profiling and Fixing the Sampler", "summary": "Profiling a 26B-parameter diffusion Mixture-of-Experts language model on an NVIDIA GB10 edge accelerator, researchers found that after weight quantization the per-token sampling and stopping pipeline over the full canvas-by-vocabulary logits tensor, not the expert GEMMs, is the largest single consumer of decode time, because diffusion decoding processes S times as many logit entries per output token as autoregressive decoding. Removing the redundant pipeline in two lossless tiers — a bit-exact tier producing byte-identical output and an accuracy-parity tier using a fused single-pass entropy kernel with Gumbel-max sampling — combined with a W4A16 expert kernel reached 2.4-2.7 times end-to-end speedup over bf16 at unchanged accuracy on GSM8K, HumanEval and MATH on a single 273 GB/s device. The finding held on a second, independently built diffusion MoE (LLaDA2.0-mini: pipeline 28.8% of decode, fused W4A16 3.3 times at accuracy parity) but disappeared on a dense diffusion model (3.6%), indicating the cost is specific to MoE diffusion, and the code and quantized weights were released.", "body_md": "arXiv:2611.21852v1 Announce Type: new \nAbstract: Masked diffusion language models (dLLMs) decode by iteratively denoising a full canvas of tokens, so every forward pass runs a per-token sampling and stopping pipeline over the entire canvas-by-vocabulary logits tensor. We profile single-user decoding of a 26B-parameter diffusion Mixture-of-Experts model on a bandwidth-bound edge accelerator (NVIDIA GB10) and report a result that is easy to miss: after weight quantisation, this logits pipeline, not the expert GEMMs, is the single largest consumer of decode time. Unlike autoregressive decoding, where sampling touches one position's logits per committed token, diffusion decoding processes S times as many logit entries per output token, where S is the number of denoising steps. We show this pipeline is largely redundant or fusable and remove it in two lossless tiers: a bit-exact tier whose output is byte-identical to the reference decoder, and an accuracy-parity tier using a fused single-pass entropy kernel and Gumbel-max sampling. Combined with a W4A16 expert kernel, the full stack reaches 2.4-2.7 times end-to-end speedup over bf16 at unchanged benchmark accuracy on GSM8K, HumanEval, and MATH, on a single 273 GB/s device. We confirm the finding on a second, independently built diffusion MoE (LLaDA2.0-mini: pipeline 28.8% of decode, fused W4A16 3.3 times at accuracy parity) and show it disappears on a dense diffusion model (3.6%). The cost is specific to MoE diffusion, where quantisation makes the backbone cheap. Code and quantised weights are released.", "url": "https://wpnews.pro/news/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler", "canonical_source": "https://arxiv.org/abs/2611.21852", "published_at": "2026-11-18 05:00:00+00:00", "updated_at": "2026-11-18 05:17:11.623856+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure", "mlops"], "entities": ["NVIDIA GB10", "LLaDA2.0-mini", "GSM8K", "HumanEval", "MATH"], "also_reported_by": [], "takeaway": "On MoE diffusion models, the per-token sampling and stopping pipeline over the full logits canvas, not the expert GEMMs, dominates decode time after quantization; replacing it with a fused single-pass entropy kernel plus Gumbel-max sampling and a W4A16 expert kernel yields 2.4-2.7x speedup over bf16", "recommendation": {"advice": "Profile the logits sampling and stopping pipeline before optimizing expert GEMMs in MoE diffusion decoding; fusing it with a single-pass entropy kernel and Gumbel-max sampling plus W4A16 experts gave 2.4-2.7x speedup at accuracy parity.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler", "markdown": "https://wpnews.pro/news/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler.md", "text": "https://wpnews.pro/news/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler.txt", "jsonld": "https://wpnews.pro/news/the-hidden-cost-of-moe-diffusion-decoding-profiling-and-fixing-the-sampler.jsonld"}}