Researchers present Specter, an inference system that uses quantization to combine multi-step speculative offloading for mixture-of-experts (MoE) models that exceed a single accelerator's memory capacity. Specter targets the PCIe I/O bottleneck caused by offloading experts to host memory, where data-dependent routing keeps transfers on the critical path.
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Off experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine s
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.