Xolotl - running a 552-billion MoE on one gaming PC, and what it asks of a constrained machine

A developer has released Xolotl, an MIT-licensed fork of the Strata inference engine that adds more than 70,000 lines of new code, including dedicated DeepSeek-V4.1/V4-Flash, GLM-5.3-Flash, Qwen3.8-27B and LFM3-Omni engines served behind a single OpenAI- and Anthropic-compatible endpoint. The engine runs a 552-billion-parameter mixture-of-experts model on a single desktop with an RTX 4090, 110 GB of DDR4 and one NVMe drive by tiering experts across VRAM, a pinned RAM arena read by DMA, and NVMe, prefetching the router's choices a layer ahead. The project's README reports decode speeds measured during six agentic coding tasks rather than idle generation, with community runs on 12 GB cards reaching roughly 8-10 tok/s decode and 149 tok/s prefill at 8K.

Xolotl github.com/Indras-Mirror/Xolotl, MIT is a descendant of Strata. The parent project's engine, server and core kernels are still inside it - merged at 0.1.44.1 - but the author is candid that more than 70,000 lines have gone on top: its own DeepSeek-V4.1 / V4-Flash and GLM-5.3-Flash engines, a Qwen3.8-27B dense engine grown from Mirko Perrone's port, an LFM3-Omni multimodal engine, a "fast mode", and the quantisation, expert-tiering and serving work in between. It serves one OpenAI- and Anthropic-compatible endpoint across all of them. The name is the project's own: Xolotl is Quetzalcoatl's dark twin in Aztec myth, the one who carries the dead up from the underworld. The engine carries experts up from RAM and NVMe. That is the whole design in one sentence, and it is also why this piece goes on the list of resource-constrained inference engines rather than the "big GPU" list. Most engines tell you what model they support. Xolotl's README has a separate document for the machine budget, and the numbers are unusual for being honest about which part of the system does the work. The measured setup is one desktop: RTX 4090 24 GB , Ryzen 7 5700X, 110 GB DDR4, one NVMe drive, Linux . The document spells out what each of those budget items means: That is a very different thing to "needs 24 GB of VRAM". The card is the least important number on that list, and the authorship says so: community runs on 12 GB cards an RTX 3060 with 46 GB of RAM reaches 149 tok/s prefill at 8K and 8-10 tok/s decode; an RTX 5070 gets 4.7 tok/s , while the same class of card runs Qwen3.8-Flash-Next at 39-48 tok/s. Same model family, same engine, four times the speed with a different quant - which is a reminder that in this work the quant you choose is the memory plan. A mixture-of-experts model only touches a few experts per token, so Xolotl keeps every expert somewhere: the hottest in VRAM, tens of gigabytes in a pinned RAM arena that the GPU reads by DMA, the rest on NVMe read at the drive's ceiling - and moves whichever experts the current token's router asks for, a layer ahead of time. That "a layer ahead" is the part that separates it from a general runtime that pages weights in lazily and hopes the OS co-operates: an engine that owns its own memory policy is the lesson my Avalon https://blog.localai.hex7.com/avalon-what-it-is-and-the-three-ways-it-differs-from-llama-cpp-and-strata/ and PolyStrata https://blog.localai.hex7.com/polystrata-the-strata-fork-that-stopped-being-one-engine-and-what-that-means-against-llama-cpp-strata-and-avalon/ notes made, and here it is again. Per-model work on top of that: calibrated mixed-precision quantisation the DeepSeek V4.1 build uses the GSQ family, which I covered in GSQ and RCO https://blog.localai.hex7.com/gsq-and-rco-the-non-uniform-quantization-now-sitting-in-gguf-files/ , speculative decoding with exact sampling, a bit-exact 4-bit KV cache for 512K context, an arena that adapts to the conversation, MTP speculative decoding for the Qwen models, GLM's abliteration LoRA on every path plus soft pruning. Two features are worth reading for the engineering rather than the marketing: That last paragraph is the one I'd point at if someone asked me what to learn from this project. A speed claim without an agreement metric beside it is a claim I cannot use; here the trade is quantified in the same table as the speed. Every number in that README comes from one machine, through the real server, while the model works its benchmark: six hard agentic coding tasks graded by hidden tests, ten turns each, thinking enabled with a 32K budget. Decode speed is measured during those agent tasks rather than in an idle generation loop - which is the only honest way to quote a number for anything used by an agent harness. It also scores its own models against the same models' hosted APIs on those tasks, and reports both directions: a local DeepSeek-V4.1 build at 4.99/6 against that model's API score of 4.49; Qwen3.8-27B at 5.50 against an API 4.47. Where it has not measured, it says "to come" - GLM-5.3-Flash is quoted as decode speed without a benchmark score, because there is no run yet. And it retired its own HumanEval table for a reason: every model scored 19-21/21 on it, so the table no longer separated them. It is worth being clear about what this is not: research code from one developer, measured on one PC, with the invitation "reports from other hardware welcome" in its own README. The Windows path exists only for DeepSeek-V4.1 and is native MSVC + CUDA; everything quoted is Linux. Xolotl: https://github.com/Indras-Mirror/Xolotl https://github.com/Indras-Mirror/Xolotl Machine requirements the honest one : https://github.com/Indras-Mirror/Xolotl/blob/main/docs/WILL IT RUN.md https://github.com/Indras-Mirror/Xolotl/blob/main/docs/WILL IT RUN.md Benchmarks and how to measure your own PC: https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/BENCHMARKS.md https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/BENCHMARKS.md Fast mode, what it costs: https://github.com/Indras-Mirror/Xolotl/blob/main/docs/XOLOTL FAST.md https://github.com/Indras-Mirror/Xolotl/blob/main/docs/XOLOTL FAST.md Changelog with PR numbers : https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/CHANGELOG.md https://github.com/Indras-Mirror/Xolotl/blob/main/docs/xolotl/CHANGELOG.md Write-ups by the author: https://indrasmirror.au https://indrasmirror.au llama.cpp: https://github.com/ggml-org/llama.cpp https://github.com/ggml-org/llama.cpp · Strata: https://github.com/Niko1221/Strata https://github.com/Niko1221/Strata