Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Researchers present Specter, an inference system that uses quantization to combine multi-step speculative offloading for mixture-of-experts (MoE) models that exceed a single accelerator's memory capacity. Specter targets the PCIe I/O bottleneck caused by offloading experts to host memory, where data-dependent routing keeps transfers on the critical path.

Mixture-of-experts MoE models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine s