Expert-locality-aware decode routing reduces MoE serving latency

Researchers introduced ELDR, an expert-locality-aware decode routing method for mixture-of-experts (MoE) models, which reduces median time-per-output-token by 5.9–13.9% across three MoE models and two workloads. Implemented in vLLM and evaluated on deployments up to 40 GPUs, ELDR leverages prefill-time expert activations to predict decode-time usage, improving serving latency without changing model outputs.

Current MoE serving pipelines treat every decode worker as interchangeable, assuming that equalizing request counts suffices to minimize latency. In practice, the weight‑loading cost of each request varies wildly because different inputs activate disjoint expert subsets, and ignoring this fact inflates inference time. Prefill‑decode PD disaggregated serving has historically relied on routers that balance only the number of pending tokens per worker, without considering which experts will be needed later. This load‑centric view leaves a hidden dimension—expert locality—that standard schedulers never exploit. ELDR cuts median time‑per‑output token by 5.9–13.9 % across three MoE models and two workloads while leaving generated images unchanged. “Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9–13.9% over the strongest of four load‑balancing baselines across three MoE models and two workloads, with model outputs unchanged.” 1 https://arxiv.org/abs/2607.00466 The router’s premise holds because prefill‑time expert activations predict decode‑time usage. “The correlation is strong for Qwen3-30B-A3B and Gemma-4‑26B‑A4B, and substantial for GPT‑OSS‑120B.” 1 https://arxiv.org/abs/2607.00466 This predictive signal lets the system assign each request to a worker whose cached experts already match its signature. The improvement is confined to PD‑disaggregated scenarios; ELDR depends on offline K‑means partitions of the signature space and assumes a homogeneous GPU pool up to 40 devices. Outside this regime the routing gains may diminish or require re‑training of the partitioning step. This suggests that future serving stacks should explore online clustering or hierarchical signatures to accommodate larger, heterogeneous clusters and very short prompts where cache warm‑up overhead dominates. Practitioners can swap the default vLLM load balancer for ELDR’s locality‑band routing and may achieve up to a double‑digit latency reduction 5.9–13.9% without retraining any MoE weights. The change is a drop‑in configuration tweak that immediately translates into faster image generation pipelines. 1 https://arxiv.org/abs/2607.00466 If expert activation patterns become first‑class scheduling metadata, will the next generation of serving frameworks render brute‑force scaling obsolete?