InferSSD Runs a 763B Open-Source Model on a Laptop Using NVMe Storage

Open-source inference runtime InferSSD runs a 763-billion-parameter mixture-of-experts model on a laptop with 48 GB of unified memory by streaming selected expert weights from NVMe storage, according to the project's GitHub repository. Built on llama.cpp and reading standard GGUF files, InferSSD reports 164.5 tokens per second prefill on an RTX 5090 — a 20x speedup over llama.cpp — and a 48x prefill gain on DGX Spark with the 816B Muse-Spark-Open model. The Apache-2.0-licensed project currently supports DeepSeek-V4.1-Flash-1017 (763B, 8B to 16B active parameters) and Muse-Spark-Open (816B, 42B active), with macOS Metal and Linux CUDA supported and Windows in progress.

InferSSD Runs a 763B Open-Source Model on a Laptop Using NVMe Storage InferSSD streams mixture-of-experts weights straight off NVMe, letting a single consumer machine run 700B+ GGUF models with double-digit speedups over llama.cpp. - InferSSD https://github.com/wici-ai/inferssd?utm source=alphasignal streams MoE expert weights from NVMe, running 700B+ models on consumer hardware. - Built on llama.cpp, uses standard GGUF files with an OpenAI-compatible API server. - Supports DeepSeek-V4.1-Flash-1017 763B and Muse-Spark-Open 816B today. - Hits 164.5 tok/s prefill on RTX 5090, a 20x speedup over llama.cpp. - DGX Spark shows the biggest gain: 48x faster prefill on Muse-Spark-Open. - macOS Metal and Linux CUDA supported, Windows in progress, Apache-2.0 licensed. InferSSD https://github.com/wici-ai/inferssd?utm source=alphasignal is an open-source inference runtime that uses NVMe storage as another memory tier, allowing a laptop with 48 GB of unified memory to execute a 763-billion-parameter mixture-of-experts model. Built on llama.cpp https://github.com/ggerganov/llama.cpp?utm source=alphasignal , it streams selected expert weights from SSD, reads GGUF model files, and provides terminal chat, browser chat, and an OpenAI-compatible API. The project gives developers local access to models that would otherwise require hundreds of gigabytes of RAM or VRAM. Its reported results favor prompt processing over token generation, making the current release more useful for experiments and long-input workloads than latency-sensitive or concurrent serving. MoE routing shrinks the working set Mixture-of-experts models contain many specialized subnetworks, called experts, at each layer. A router selects only a small number for each token, so inference touches a fraction of the stored weights during any forward pass. InferSSD keeps inactive experts on NVMe and loads selected experts as inference proceeds. Total parameter count describes every weight stored in the checkpoint, while active parameter count approximates the weights used for one token. That distinction explains how a 763B model with 8B to 16B active parameters can run on hardware that cannot hold the full checkpoint in memory. | Model | Total parameters | Active parameters | Distribution | |---|---|---|---| | DeepSeek-V4.1-Flash-1017 | 763B | 8B to 16B | Community GGUF conversion | | Muse-Spark-Open | 816B | 42B | Community GGUF conversion | Current compatibility remains model-specific despite the use of standard GGUF files. Both supported checkpoints are community conversions hosted on Hugging Face and can be downloaded directly through InferSSD, without a separate conversion step. DeepSeek’s Q8 0 This story is for Pro members You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.