InferSSD streams mixture-of-experts weights straight off NVMe, letting a single consumer machine run 700B+ GGUF models with double-digit speedups over llama.cpp.
- InferSSD streams MoE expert weights from NVMe, running 700B+ models on consumer hardware.
- Built on llama.cpp, uses standard GGUF files with an OpenAI-compatible API server.
- Supports DeepSeek-V4.1-Flash-1017 (763B) and Muse-Spark-Open (816B) today.
- Hits 164.5 tok/s prefill on RTX 5090, a 20x speedup over llama.cpp.
- DGX Spark shows the biggest gain: 48x faster prefill on Muse-Spark-Open.
- macOS Metal and Linux CUDA supported, Windows in progress, Apache-2.0 licensed.
InferSSD is an open-source inference runtime that uses NVMe storage as another memory tier, allowing a laptop with 48 GB of unified memory to execute a 763-billion-parameter mixture-of-experts model. Built on llama.cpp, it streams selected expert weights from SSD, reads GGUF model files, and provides terminal chat, browser chat, and an OpenAI-compatible API.
The project gives developers local access to models that would otherwise require hundreds of gigabytes of RAM or VRAM. Its reported results favor prompt processing over token generation, making the current release more useful for experiments and long-input workloads than latency-sensitive or concurrent serving.
MoE routing shrinks the working set #
Mixture-of-experts models contain many specialized subnetworks, called experts, at each layer. A router selects only a small number for each token, so inference touches a fraction of the stored weights during any forward pass. InferSSD keeps inactive experts on NVMe and loads selected experts as inference proceeds.
Total parameter count describes every weight stored in the checkpoint, while active parameter count approximates the weights used for one token. That distinction explains how a 763B model with 8B to 16B active parameters can run on hardware that cannot hold the full checkpoint in memory.
| Model | Total parameters | Active parameters | Distribution |
|---|---|---|---|
| DeepSeek-V4.1-Flash-1017 | 763B | 8B to 16B | Community GGUF conversion |
| Muse-Spark-Open | 816B | 42B | Community GGUF conversion |
Current compatibility remains model-specific despite the use of standard GGUF files. Both supported checkpoints are community conversions hosted on Hugging Face and can be downloaded directly through InferSSD, without a separate conversion step.
DeepSeek’s Q8_0
This story is for Pro members #
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.