MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

A new arXiv paper (2610.30206v1) proposes MoEZip, a routing-aware KV cache compression method for sparse Mixture-of-Experts language models that scores contexts using a Fisher information matrix-based metric of expert-routing sensitivity combined with attention weights. At a 10% KV retention ratio, MoEZip beats the state-of-the-art baseline by +24 points on average on LongBench and +20 points on RULER, corresponding to 2.3x and 4.9x higher scores respectively. The authors argue existing KV cache compression methods designed for dense transformers overlook that eviction in MoE models can change which experts are selected during answer generation, degrading output quality.

arXiv:2610.30206v1 Announce Type: new Abstract: Mixture-of-Experts MoE language models scale model capacity efficiently by activating a small subset of parameters per token. However, in long-context inference, memory remains a major bottleneck because the KV cache grows linearly with sequence length. Existing KV cache compression methods reduce this cost for dense transformers, but overlook a problem specific to MoE: KV eviction can change which experts are selected during answer generation, degrading the generation quality. We propose MoEZip, a routing-aware KV cache compression for sparse MoE language models. To estimate the sensitivity of expert routing to perturbation, we propose a novel metric based on the Fisher information matrix of routing distributions. MoEZip combines this metric with attention weights for routing-aware scoring of contexts. Finally, we compose a complementary scoring system that combines eviction methods for dense and sparse-MoE architectures. Experiments show that MoEZip achieves robust performance under aggressive cache compression on various benchmarks. At a KV retention ratio of 10\%, MoEZip improves over the SoTA baseline by +24 and +20 points on average on LongBench and RULER, corresponding to 2.3$\times$ and 4.9$\times$ higher scores, respectively.