{"slug": "inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus", "title": "InferSSD: Running 800B-Class MoE Models From NVMe on Consumer GPUs", "summary": "WiCi Research released InferSSD, an open-source runtime that keeps routed mixture-of-experts weights on NVMe while attention and KV cache stay in RAM or VRAM, targeting 763B- to 816B-class open models on consumer hardware. On an RTX 5090 with DeepSeek-V4.1-Flash-1017, WiCi's benchmark tables report 164.5 tokens/s prefill and 7.27 tokens/s decode at a 4K prompt tier, which the article cites as roughly 20x prefill and 8x decode versus llama.cpp on the same configuration. InferSSD builds on llama.cpp and GGUF with an OpenAI-compatible API, and WiCi argues its read-mostly static expert reads pose a different SSD endurance profile than dynamic KV-cache offload.", "body_md": "**WiCi Research's [InferSSD project](https://github.com/wici-ai/inferssd?ref=explainx) keeps routed mixture-of-experts weights on NVMe** while attention and KV cache stay in RAM or VRAM, aiming to run **763B- to 816B-class** open models on hardware that cannot hold full checkpoints in memory. The [InferSSD article](https://wici.ai/article/running-800b-models-with-inferssd?ref=explainx) walks through architecture, SSD endurance considerations, and benchmark tables comparing **InferSSD**, **llama.cpp**, and **Colibri** on **RTX 5090**, **RTX 5060 Ti**, **48GB MacBook Pro**, and **DGX Spark** configurations.\n\nThe idea follows **The SSD Is the New VRAM** framing and SGLang's **SSD Expert Pack** line of work, which we connected to the [SGLang consortium summit](https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026). For builders who already downloaded **DeepSeek-V4.1-Flash** or **Muse-Spark-Open** GGUFs, InferSSD is a **runtime choice**, not a new foundation model.\n\n## TL;DR\n\n| Question | Answer (article snapshot) | \n|---|---|\n| Repo | `github.com/wici-ai/inferssd` (branch`main` ) | \n| Base stack | **[llama.cpp](https://github.com/ggml-org/llama.cpp?ref=explainx)** plus GGUF, browser chat, OpenAI-compatible API | \n| DeepSeek on RTX 5090 | **164.5 tok/s prefill** ,**7.27 tok/s decode** (4K tier) | \n| Muse-Spark on RTX 5090 | **106.6 prefill** ,**1.52 decode** | \n| MacBook Pro DeepSeek | **26.27 prefill** ,**2.704 decode** | \n| vs llama.cpp on 5090 | Article cites **~20x prefill** and**~8x decode** ratios for that config | \n| Wear risk | Article contrasts **static expert reads** with**SSD KV offload** endurance issues | \n\n## Why 800B models hit a memory wall\n\nMoE routing: only a subset of experts loads per token while the full expert set stays on disk\n\nDeepSeek-V4.1-Flash-1017 and Muse-Spark-Open sit in the **hundreds of gigabytes** even in compressed GGUF form. A [48GB MacBook Pro](https://explainx.ai/blog/apple-mlx-rdma-thunderbolt-5-mac-studio-m5-ultra-muse-spark-open-local-inference-2026) or **32GB VRAM** desktop holds only a slice.\n\nClassic fixes:\n\n- **Cloud APIs** — simplest, adds cost and data handling.\n- **Multi-GPU tensor parallel** — fast, expensive.\n- **Expert streaming** — keep**all experts off-SSD** , pull**routed subsets** per token.\n\nInferSSD chooses the third path with an explicit pipeline: **router → residency check → grouped reads → execution** under a shared memory budget, documented in the article's **expert inference pipeline** section.\n\nStatic expert weights differ from **dynamic KV caches** written every token. WiCi cites **Gandalf** research on SSD-backed KV caching endurance, then argues expert reads are **read-mostly** workloads — still not zero wear, but a different failure mode than constantly rewriting NAND.\n\n## Reported throughput: DeepSeek-V4.1-Flash-1017\n\nWiCi's tables use a **4K prompt tier** label; actual tokens depend on chat templates. **Decode** numbers cover **tokens 129–256** of a **256-token** output.\n\n**DeepSeek · 4K tier · tokens/s**\n\n| System | Stage | llama.cpp | Colibri | InferSSD | \n|---|---|---|---|---|\n| MacBook Pro 48GB | Prefill | 5.929 | 0.708 | **26.270** | \n| MacBook Pro 48GB | Decode | 0.873 | 0.437 | **2.704** | \n| RTX 5060 Ti | Prefill | 6.950 | 7.020 | **75.100** | \n| RTX 5060 Ti | Decode | 0.242 | 0.900 | **2.550** | \n| RTX 5090 | Prefill | 8.060 | 39.210 | **164.500** | \n| RTX 5090 | Decode | 0.867 | 1.878 | **7.270** | \n| DGX Spark 128GB | Prefill | 2.730 | 0.826 | **127.500** | \n| DGX Spark 128GB | Decode | 1.608 | 0.356 | **7.260** | \n\n**Interpreting the spread:** InferSSD massively improves **prefill** on several rigs — important for long context ingestion — while **decode** remains **single-digit tokens/s** on the largest configs. Interactive chat still feels **cloud-slow** for 800B models even when the technique works.\n\nColibri numbers on Mac DeepSeek used **CPU routed experts** and official Safetensors, per methodology notes — backends differ, so do not treat gaps as pure software magic.\n\n## Muse-Spark-Open: heavier, slower decode\n\n**Muse-Spark-Open · 816B · 4K tier**\n\n| System | Prefill (InferSSD) | Decode (InferSSD) | \n|---|---|---|\n| MacBook Pro | 12.480 | **0.944** | \n| RTX 5060 Ti | 52.100 | **0.219** | \n| RTX 5090 | 106.600 | **1.520** | \n| DGX Spark | 83.900 | **2.140** | \n\nSub-2 tok/s decode on consumer boxes means **batch overnight jobs**, not pair programming. Compare [Edge0-145B's GPU-local agent story](https://explainx.ai/blog/edge0-145b-open-on-device-laptop-agent-model-november-2026) when latency dominates.\n\n## Hardware table WiCi used\n\nBenchmarked setups included:\n\n- **MacBook Pro** with**2TB internal SSD** and**48GB** unified memory.\n- **RTX 5090** box with**PCIe 5.0 x4** to**2TB ZHITAI TiPro9000** .\n- **RTX 5060 Ti** with**PCIe 4.0 x4** to the same drive family.\n- **DGX Spark** with**128GB** unified memory and**4TB Samsung SSD** .\n\nDrive link width, cache budget, and CPU affect whether SSD bandwidth saturates. The article warns Linux numbers are **tester-reported summaries**, not a single automated harness across all rows.\n\n## Checkpoint format caveats\n\nThe DeepSeek **763B label** is the published checkpoint name; the text GGUF WiCi used holds **748.49B parameters**, omitting vision and MTP/DSpark modules. **Q8_0** labels mask a mix of **MXFP4 experts** and **Q8_0/BF16/F32** tensors elsewhere.\n\nMuse uses **Unsloth UD-Q4_K_XL** conversion. Expert packing **does not re-quantize** payload bytes — quality parity with official Safetensors is **not guaranteed** without your own evals, the same caution we apply in [open-weights forensics posts](https://explainx.ai/blog/chinese-open-weights-blackwell-gpu-checkpoint-forensics-november-2026).\n\n## How to run InferSSD locally\n\nWiCi documents:\n\n```\ngit clone --branch main https://github.com/wici-ai/inferssd.git\n```\n\nBuild instructions continue in the repository README; the article also points to **WiCi Infer**, a desktop app with the engine precompiled.\n\nBecause InferSSD keeps **OpenAI-compatible API** and browser chat surfaces atop llama.cpp, you can often drop it into workflows already using [local gateway patterns](https://explainx.ai/blog/self-service-llm-gateway-amazon-bedrock-litellm-fargate-inference-profiles-virtual-keys-guide-2026) — just expect **localhost latency** profiles, not cloud SLAs.\n\nInferSSD streams large MoE expert weights from fast NVMe while keeping attention layers in system RAM\n\n## Pipeline stages WiCi documents\n\nExpert routing sends each token down one of several paths before weights load from NVMe\n\nThe article decomposes inference into explicit stages rather than relying on the OS page cache alone:\n\n1. **Expert indexing** so the runtime knows byte ranges per expert matrix.\n2. **Token grouping** to batch tokens that hit the same experts, amortizing reads.\n3. **Residency checks** against GPU, RAM, and SSD caches before issuing I/O.\n4. **Scheduled execution** that overlaps CPU preparation with GPU matmuls.\n5. **Memory budget enforcement** so KV growth does not silently evict hot experts.\n\nThat is the difference between **mmap accidents** and a **MoE-aware scheduler**. If you profile stalls, measure **I/O wait** separately from **kernel launch** — the article argues many \"slow MoE\" stories blame quantization when **read scheduling** was the bottleneck.\n\n## When InferSSD beats renting GPUs\n\nRough decision math for indie teams:\n\n| Workload | InferSSD local | Cloud frontier API | \n|---|---|---|\n| Overnight batch eval on fixed prompts | **Attractive** if you own 5090 + 2TB NVMe | Bill explodes at 800B context lengths | \n| Interactive coding assistant | **Weak** at sub-3 tok/s decode on Muse | Latency wins | \n| Privacy-sensitive legal/medical RAG | **Attractive** if data cannot leave site | Compliance wins | \n| One-off demos | **Poor** — hours of setup | Faster | \n\nPair this with [AWS Bedrock four inference choices](https://explainx.ai/blog/aws-bedrock-four-inference-choices-cost-cache-flex-guide-2026) when hybrid routing makes sense: **local draft**, **cloud verify**, or **tiered model routing**.\n\n## SSD endurance and what not to offload\n\nWiCi's **read disturb** section is the adult supervision part of the hype:\n\n- **Expert weights** are static during a session — good fit for NVMe reads with caching.\n- **KV cache on SSD** can hammer write endurance — InferSSD focuses expert path separately from Gandalf-style KV offload debates.\n\nIf you operate **24/7 agents**, model **TBW ratings** and cache hit rates before betting production on consumer SSDs.\n\n## Comparison to other disk-streaming projects\n\n| Project | Model scale | Stack | Notes | \n|---|---|---|---|\n| **InferSSD** | 763B–816B | llama.cpp fork | Article benchmarks vs Colibri | \n| **Janas** | Varies | Own engine + MCP | [Janas MoE post](https://explainx.ai/blog/janas-run-moe-llms-larger-than-ram-stream-experts-ssd-october-2026) | \n| **Edge0-145B** | 145B trained for device | edge0-strata | [Edge0-145B post](https://explainx.ai/blog/edge0-145b-open-on-device-laptop-agent-model-november-2026) | \n| **Colibrì** | GLM-5.2 class | Disk streaming | [25GB RAM GLM guide](https://explainx.ai/blog/colibri-glm-5-2-streaming-disk-25gb-ram-july-2026) | \n| **SGLang SSD Expert Pack** | Production serving | Datacenter | [Consortium context](https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026) | \n\nUse **InferSSD** when you already committed to **800B GGUF artifacts** and own **fast NVMe**. Use **Edge0-145B** when you want **agent decode above 80 tok/s** on **one GPU** at **smaller scale**.\n\n## What builders should measure before trusting tables\n\n1. **Your exact GGUF** — module omissions change parameter counts.\n2. **Template and 4K tier definition** — prefill wins may not transfer to your prompt distribution.\n3. **Decode tokens 129–256** — cold start and first-token latency live outside the quoted window.\n4. **Colibri / llama.cpp versions** — article notes**N/S** when Muse unsupported in a Colibri build.\n5. **Thermals during long runs** — SSD and GPU throttle together on laptops.\n\nIndependent Hacker News discussion was **sparse** when this run started (`~5` points). Treat WiCi tables as **vendor-adjacent** until community reproduction threads appear.\n\n## Policy and supply context (why local 800B matters)\n\nWashington's **export fights** and [Nvidia governance crisis](https://explainx.ai/blog/jensen-huang-resigns-colette-kress-interim-ceo-nvidia-november-2026) do not change **open-weight availability** overnight, but they change **who can rent** frontier capacity. Local **800B-class** inference is a **sovereignty and cost hedge**, not a miracle — the same trade-off [IMF financial stability reporting](https://explainx.ai/blog/imf-global-financial-stability-report-ai-share-crash-circular-financing-cyber-risk-2026) highlights when AI capex chains wobble.\n\n## Operator checklist before long runs\n\n1. **Firmware and TRIM** enabled on NVMe — consumer drives throttle when hot.\n2. **Leave headroom** on the drive; expert packs plus OS snapshots fill disks fast.\n3. **Log cache hit rates** if the repo exposes them; otherwise use`iostat` during decode.\n4. **Pin llama.cpp revision** in issue reports — WiCi tracks moving baselines.\n5. **Do not confuse prefill wins with user-perceived speed** ; measure time-to-first-token and tok 200 separately.\n\nIf numbers still disagree with WiCi's tables, publish your **drive model, PCIe lane count, and GGUF filename** when filing upstream issues — without that trio, maintainers cannot reproduce **164 tok/s** claims on DeepSeek tiers.\n\n## Related reading\n\n*Throughput figures and hardware notes come from WiCi Research's InferSSD article and repository documentation as accessed November 13, 2026. Verify on your drives before production loads.*", "url": "https://wpnews.pro/news/inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus", "canonical_source": "https://explainx.ai/blog/inferssd-800b-consumer-hardware-llama-cpp-november-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-13 00:00:00+00:00", "updated_at": "2026-11-13 01:51:36.785147+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["WiCi Research", "InferSSD", "llama.cpp", "Colibri", "DeepSeek-V4.1-Flash-1017", "Muse-Spark-Open", "Nvidia RTX 5090", "DGX Spark"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus", "markdown": "https://wpnews.pro/news/inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus.md", "text": "https://wpnews.pro/news/inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus.txt", "jsonld": "https://wpnews.pro/news/inferssd-running-800b-class-moe-models-from-nvme-on-consumer-gpus.jsonld"}}