{"slug": "inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage", "title": "InferSSD Runs a 763B Open-Source Model on a Laptop Using NVMe Storage", "summary": "Open-source inference runtime InferSSD runs a 763-billion-parameter mixture-of-experts model on a laptop with 48 GB of unified memory by streaming selected expert weights from NVMe storage, according to the project's GitHub repository. Built on llama.cpp and reading standard GGUF files, InferSSD reports 164.5 tokens per second prefill on an RTX 5090 — a 20x speedup over llama.cpp — and a 48x prefill gain on DGX Spark with the 816B Muse-Spark-Open model. The Apache-2.0-licensed project currently supports DeepSeek-V4.1-Flash-1017 (763B, 8B to 16B active parameters) and Muse-Spark-Open (816B, 42B active), with macOS Metal and Linux CUDA supported and Windows in progress.", "body_md": "# InferSSD Runs a 763B Open-Source Model on a Laptop Using NVMe Storage\n\nInferSSD streams mixture-of-experts weights straight off NVMe, letting a single consumer machine run 700B+ GGUF models with double-digit speedups over llama.cpp.\n\n- [InferSSD](https://github.com/wici-ai/inferssd?utm_source=alphasignal) streams MoE expert weights from NVMe, running 700B+ models on consumer hardware.\n- Built on llama.cpp, uses standard GGUF files with an OpenAI-compatible API server.\n- Supports DeepSeek-V4.1-Flash-1017 (763B) and Muse-Spark-Open (816B) today.\n- Hits 164.5 tok/s prefill on RTX 5090, a 20x speedup over llama.cpp.\n- DGX Spark shows the biggest gain: 48x faster prefill on Muse-Spark-Open.\n- macOS Metal and Linux CUDA supported, Windows in progress, Apache-2.0 licensed.\n\n[InferSSD](https://github.com/wici-ai/inferssd?utm_source=alphasignal) is an open-source inference runtime that uses NVMe storage as another memory tier, allowing a laptop with 48 GB of unified memory to execute a 763-billion-parameter mixture-of-experts model. Built on [llama.cpp](https://github.com/ggerganov/llama.cpp?utm_source=alphasignal), it streams selected expert weights from SSD, reads GGUF model files, and provides terminal chat, browser chat, and an OpenAI-compatible API.\n\nThe project gives developers local access to models that would otherwise require hundreds of gigabytes of RAM or VRAM. Its reported results favor prompt processing over token generation, making the current release more useful for experiments and long-input workloads than latency-sensitive or concurrent serving.\n\n## MoE routing shrinks the working set\n\nMixture-of-experts models contain many specialized subnetworks, called experts, at each layer. A router selects only a small number for each token, so inference touches a fraction of the stored weights during any forward pass. InferSSD keeps inactive experts on NVMe and loads selected experts as inference proceeds.\n\nTotal parameter count describes every weight stored in the checkpoint, while active parameter count approximates the weights used for one token. That distinction explains how a 763B model with 8B to 16B active parameters can run on hardware that cannot hold the full checkpoint in memory.\n\n| Model | Total parameters | Active parameters | Distribution | \n|---|---|---|---|\n| DeepSeek-V4.1-Flash-1017 | 763B | 8B to 16B | Community GGUF conversion | \n| Muse-Spark-Open | 816B | 42B | Community GGUF conversion | \n\nCurrent compatibility remains model-specific despite the use of standard GGUF files. Both supported checkpoints are community conversions hosted on Hugging Face and can be downloaded directly through InferSSD, without a separate conversion step.\n\nDeepSeek’s `Q8_0`\n\n## This story is for Pro members\n\nYou've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.", "url": "https://wpnews.pro/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage", "canonical_source": "https://alphasignal.ai/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage", "published_at": "2026-11-13 19:54:37+00:00", "updated_at": "2026-11-14 14:22:41.819320+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["InferSSD", "llama.cpp", "DeepSeek-V4.1-Flash-1017", "Muse-Spark-Open", "NVIDIA RTX 5090", "DGX Spark", "Hugging Face", "wici-ai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage", "markdown": "https://wpnews.pro/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage.md", "text": "https://wpnews.pro/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage.txt", "jsonld": "https://wpnews.pro/news/inferssd-runs-a-763b-open-source-model-on-a-laptop-using-nvme-storage.jsonld"}}