SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

Researchers propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded sparse Mixture-of-Experts (MoE) inference that uses a shared lightweight adapter to predict next-layer expert candidates for asynchronous transfer, reducing expert-loading latency without altering pretrained routing semantics. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with fewer trainable parameters than baselines, and on a Snapdragon 8 Elite device improves decoding throughput by up to 20% over a compute-optimized offloading runtime.

arXiv:2607.24787v1 Announce Type: new Abstract: Sparse Mixture-of-Experts MoE models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory. Although expert offloading alleviates memory pressure by moving inactive experts to host memory or storage, it introduces a routing-dependent transfer bottleneck: required experts are known only after native top-\ K\ routing, which serializes routing, expert loading, and expert execution during inference. To address this bottleneck, we propose SpecPrefetch, a parameter-efficient prefetching framework for offloaded MoE inference. SpecPrefetch uses a shared lightweight adapter to predict next-layer expert candidates only for asynchronous transfer, while the frozen native router still determines the final executed experts. By separating transfer prediction from execution routing, SpecPrefetch reduces exposed expert-loading latency without changing pretrained routing semantics, so prediction errors affect transfer efficiency rather than model outputs. In addition, a window-aware scheduler prioritizes feasible transfers under cache and bandwidth constraints. Across Qwen3-VL-30B-A3B and DeepSeek-VL2-Tiny, SpecPrefetch achieves the best average expert recall in 9 out of 10 model-benchmark settings with substantially fewer trainable parameters than learned predictor baselines. On a Snapdragon 8 Elite device, SpecPrefetch further improves decoding throughput by up to \ 20\%\ over a compute-optimized offloading runtime, demonstrating practical benefits for storage-constrained MoE deployment. The code and model weights are available at https://github.com/wei390/SpecPrefetch.