{"slug": "when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of", "title": "When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models", "summary": "A new arXiv preprint (arXiv:2609.04453v1) finds that expert pruning in Mixture-of-Experts (MoE) models fails under over-dispersed routing, where aggressive load-balancing makes router probabilities unreliable importance signals. On gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning while the highest-perplexity configuration preserves it, and no single scoring metric dominates across domains. The authors propose Minimax Expert Score Allocation (MESA), a domain-aware method that at 25% expert pruning achieves the smallest worst-case degradation across domains and outperforms activation-aware baselines on 7 of 11 benchmarks, generalizing to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B.", "body_md": "arXiv:2609.04453v1 Announce Type: new \nAbstract: Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that this assumption breaks down under over-dispersed routing, a regime associated with aggressive load-balancing during training, in which tokens are distributed nearly uniformly across experts and importance signals collapse. In this regime, perplexity does not predict downstream task accuracy: on gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning, while the highest-perplexity configuration preserves it. This does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together. Pruning under over-dispersed routing also exposes a capability trade-off in which no single scoring metric dominates: activation-aware scoring preserves mathematical reasoning but severely degrades knowledge-intensive science (an 18-point gap on GPQA), whereas frequency-based scoring exhibits the reverse. We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected, minimizing worst-case domain degradation rather than average accuracy. At 25% expert pruning MESA achieves the smallest worst-case degradation across domains, outperforming activation-aware baselines on 7 of 11 benchmarks at a correspondingly reduced memory footprint, and it generalizes to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. Our results indicate that over-dispersed routing is a qualitatively distinct pruning regime in which standard assumptions fail, and that recognizing it is a prerequisite for principled expert pruning of load-balanced MoE models.", "url": "https://wpnews.pro/news/when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of", "canonical_source": "https://arxiv.org/abs/2609.04453", "published_at": "2026-09-07 04:00:00+00:00", "updated_at": "2026-09-07 04:27:19.438128+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv", "gpt-oss-20B", "Mixtral-8x7B-Instruct", "MESA", "gpt-oss-120B", "Gemma-4-26B-A4B", "OLMoE-1B-7B", "GPQA"], "alternates": {"html": "https://wpnews.pro/news/when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of", "markdown": "https://wpnews.pro/news/when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of.md", "text": "https://wpnews.pro/news/when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of.txt", "jsonld": "https://wpnews.pro/news/when-load-balancing-goes-too-far-expert-pruning-in-over-dispersed-mixture-of.jsonld"}}