# A Hobbyist Project Runs a 125B AI Model on 8GB of RAM > Source: > Published: 2026-10-26 14:44:50+00:00 *8GB of RAM running a 125-billion-parameter model sounds like a typo. It isn't: developer MaxDevv rebuilt a GLM offloading trick for Alibaba's newest open model to get there.* Most of a huge language model's weight never gets touched on any given token. That's the whole premise of a mixture-of-experts design. It's also the loophole MaxDevv exploited this week with a project called Lean-QwenFlash. According to the project's GitHub page, it runs Qwen3.8-Flash-Next, Alibaba's 125-billion-parameter open-weight model, at its full 262,144-token context window while capped at 16GB of VRAM and just 8GB of system RAM. The numbers posted: 1,600 tokens per second on prefill, 20 tokens per second on generation. Qwen3.8-Flash-Next only activates 6 billion of its 125 billion parameters on any single token. Alibaba released it on August 26: an open preview of the architecture expected to underpin Qwen4. The model carries an unusual feature for exactly this situation, a 51-billion-parameter n-gram lookup table that doesn't need to sit in GPU memory at all. It hashes the last few tokens and reads back a small set of rows, which means it can live on a hard drive instead of competing for VRAM. Lean-QwenFlash takes that idea and pushes it further, treating the model's mixture-of-experts weights the same way. The engine streams whichever experts aren't already cached straight off an NVMe drive, over PCIe, into a bounded slice of GPU memory, using O_DIRECT reads that skip the operating system's page cache. All the actual math, every matrix multiply, still runs on the GPU. The disk is just doing what VRAM usually does: holding weights until they're needed. MaxDevv didn't build this from nothing. The project is adapted from an engine called glm53-flash-offload, built by a developer or team going by sybil-solutions for offloading GLM models. It runs on top of ExLlamaV3, the quantization and inference library from turboderp-org that already supports experimental CPU offloading for MoE layers. The test hardware was an RTX 4090, a 24GB card, but the benchmark artificially capped it to 16GB of VRAM and 8GB of system RAM to simulate a much smaller rig. That's a meaningful distinction. This isn't proof a $1,500 gaming PC with a 4090 in it is secretly a data center. [A wave of narrow AI inference engines is beating vLLM and llama.cpp at their own game](https://startupfortune.com/a-wave-of-narrow-ai-inference-engines-is-beating-vllm-and-llamacpp-at-their-own-game/) Engines like Strata, ninfer, and Splash beat llama.cpp by up to 4x on Qwen3.8-Flash-Next, but each works for exactly one model on one GPU family, a tradeoff buyers often miss. - [narrow AI inference engines outperforming vLLM](https://startupfortune.com/a-wave-of-narrow-ai-inference-engines-is-beating-vllm-and-llamacpp-at-their-own-game/) - [specialized GPU runtime for local AI models](https://startupfortune.com/a-wave-of-narrow-ai-inference-engines-is-beating-vllm-and-llamacpp-at-their-own-game/) It's proof the offloading architecture itself doesn't need much memory to work. That's a different and more useful claim: it means the technique should generalise to genuinely modest hardware, not just a high-end card running in a restricted mode. Compare that to running the model straight. Community reports on Hugging Face's discussion threads for the official GGUF builds put Qwen3.8-Flash-Next's Q4_K_M quantization at around 123GB. Most people trying to run it without heavy offloading report needing 75GB or more of combined RAM and VRAM just to load it. One widely cited test on an M3 Ultra Mac with 256GB of unified memory got roughly 30 tokens per second. Lean-QwenFlash gets the same ballpark generation speed, 20 tokens per second, on a setup with a tiny fraction of that memory. The tradeoff is the one you'd expect: you're borrowing disk bandwidth instead of memory capacity. A slow or heavily fragmented SSD will not deliver 1,600 tokens per second of prefill no matter what the GitHub README promises. This is where the GPU scarcity story gets complicated rather than solved. Qwen3.8-Flash-Next's n-gram table was already a hint that Alibaba's engineers were thinking about this exact problem: how do you ship a 125-billion-parameter model that doesn't require 125 billion parameters' worth of fast memory. Lean-QwenFlash is a hobbyist taking that hint and running with it, not Alibaba shipping an official feature. It's unclear yet how it holds up under sustained, varied workloads rather than a single demonstration run, and NVMe drives degrade in ways VRAM doesn't: thermal throttling, queue depth limits, wear over time. Still, the direction matters more than any one benchmark. If SSD-streamed experts become a standard trick rather than a weekend project, the calculus around renting cloud GPU time for inference on frontier-scale open models gets a lot less obvious. A startup weighing API costs against local deployment now has a third option that isn't "buy more VRAM" - the first time that's been true for a model this large. Whether that option is reliable enough to build a product on is the question nobody has answered yet, including MaxDevv. **Also read:** [Anthropic lost $80.7 billion in six months, and most of it is a warning sign anyway](https://startupfortune.com/anthropic-lost-807-billion-in-six-months-and-most-of-it-is-a-warning-sign-anyway/) • [AI Agents Are Starting to Set the Prices You Pay Online](https://startupfortune.com/ai-agents-are-starting-to-set-the-prices-you-pay-online/) • [Stanford and Berkeley Researchers Build TokenGate to Cut LLM Prefill Costs](https://startupfortune.com/stanford-and-berkeley-researchers-build-tokengate-to-cut-llm-prefill-costs/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](https://startupfortune.com/community/) Almost there. Sign in and your reply posts straight away.