{"slug": "180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers", "title": "180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers", "summary": "Streaming mixture-of-experts (MoE) experts directly from SSD during inference gets within 6% of the Strata engine on token generation, according to a Hugging Face blog post testing Qwen3.8-Flash-Next, a 177B-parameter model, on a 32GB MacBook M5. The post compares how llama.cpp, vLLM and Transformers handle MoE models that exceed available memory, noting llama.cpp's new GPU expert cache (--moe-cache-mib) fragments the graph from 2 splits per token to 50 and loses about 10% even when everything fits, while Transformers' Accelerate offload copies an entire layer's expert bank to the GPU on every token to use 8 or 10 experts. The author cites a separate benchmark of Qwen3.8-Flash-Next on a 32GB Mac M4 running at 13.5 tok/s under Strata, faster than llama.cpp.", "body_md": "[Image-Text-to-Text •  177B • Updated   •  7.3M  •  1.31k](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF)  \n\n# \n\t\t180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers\n\t\n\n [Update on GitHub](https://github.com/huggingface/blog/blob/main/moe-expert-streaming.md)\n\n*Streaming MoE experts directly from SSD gets within 6% of Strata on token generation.*\n\nI've been trying to rely less on online models and run bigger ones locally.\nAt home I have a PC with two RTX 4090s (let's call it **Orthrus** - the two-headed hound\nof Greek mythology), and for work I have a MacBook M5 with 32G of unified memory.\n\nOrthrus runs on Windows 11, because my kids play games all the time. I had a dual boot with Ubuntu, but their games ended up taking most of the two terabytes of disk.\n\nI am also a huge fan of RMD (rooms moving development): one hour in the kitchen sipping my coffee, two hours at my office desk, then the sofa. A fixed desktop tower is not for me.\n\nMy first experiment was to run [Qwen 3.8 27B](https://huggingface.co/Qwen/Qwen3.8-27B) with [`llama.app`](https://llama.app/) on Orthrus and use\nit from my laptop through [`pi`](https://pi.dev/). That model does the coding tasks, an online\nmodel like Opus does the orchestration, and the second RTX runs a FIM model for\nauto-completion in Neovim.\n\nWhen I talked about this on social networks, I got very strong feedback: use\n[Strata](https://github.com/niko1221/strata) with [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).\n\n## \n\t\tWhat is Strata?\n\t\n\nStrata is an inference engine for MoE models like Qwen. Only the hot experts stay in VRAM, the rest goes to RAM, or even to disk. Experts are 90%+ of a MoE model's weights, so a 180B model like this one can run on a GPU with much less memory. Like my RTX 4090!\n\nIt works on a laptop too, and I stumbled on\n[this blog post](https://modelfit.io/blog/strata-qwen38-flash-next-mac/) doing\nexactly that: Qwen3.8-Flash-Next on a 32 GB Mac M4, faster than [llama.cpp](https://github.com/ggml-org/llama.cpp), at\n13.5 tok/s.\n\nThat's too slow for how I use coding agents, but it's an interesting itch to scratch. A whole range of MoE models don't fit in a MacBook's unified memory and could run at a decent speed.\n\n## \n\t\tHow do other engines offload?\n\t\n\nBefore committing to Strata, I looked at what llama.cpp, [vLLM](https://github.com/vllm-project/vllm) and [Transformers](https://github.com/huggingface/transformers)\ndo when a MoE model doesn't fit. Three questions matter: where do the experts\nlive, where does the expert matmul run, and what moves on each token?\n\n**llama.cpp** maps the GGUF file with `mmap` and can keep the experts on the\nCPU (`--cpu-moe`, `--n-cpu-moe N`, `-ot \"exps=CPU\"`, or `--fit on` to pick for\nyou). Those experts are also *computed* on the CPU, so only the activations\ncross PCIe, never the weights. Great when the experts fit in RAM. When they\ndon't, the page cache thrashes. A GPU expert cache (`--moe-cache-mib`) just\nlanded, but it fragments the graph (2 splits per token become 50) and loses\n~10% even when everything fits.\n\n**vLLM**'s `--cpu-offload-gb N` puts N GB of weights in pinned RAM, and kernels\nread them in place over PCIe. For a MoE layer, that's only the routed experts,\nbut nothing is cached: a hot expert crosses PCIe again on every token, and\nthere's no disk tier. By default the offloaded weights include dense layers,\nread on every token; `--cpu-offload-params experts` keeps it to the experts. That fits vLLM's big\nbatches, which touch almost every expert anyway, less so one user on one GPU.\n\n**Transformers** offloads through [Accelerate](https://github.com/huggingface/accelerate) (`device_map=\"auto\"`,\n`offload_folder`), one *module* at a time: hooks copy its weights to the GPU\nbefore `forward`. For a MoE, that module is a layer's whole expert bank, so\nevery token copies every expert of every offloaded layer to use 8 or 10 of them.\nNo cache, no notion of routing. And on a Mac, CPU and GPU share the same memory,\nso \"offload to CPU\" frees nothing.\n\nI ran all of them on [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) on one A10G (24 GB), with plenty of RAM,\nand Harry Mellor reran vLLM with better flags:\n\n| Engine | Decode (tok/s) | \n|---|---|\n| Strata, Q8_0 | 82 - 88 | \n| llama.cpp, Q8_0, `--fit on` | 63 - 64 | \n| llama.cpp, BF16, `--fit on` | 35 - 36 | \n| vLLM, FP8, `--cpu-offload-gb 16 --cpu-offload-params experts` | 20.1 | \n| vLLM, FP8, same flags, `--enforce-eager` | 14.7 | \n| vLLM, FP8, `--cpu-offload-gb 16` | 9.7 | \n| vLLM, BF16, `--cpu-offload-gb 50 --cpu-offload-params experts` | 7.3 | \n| vLLM, BF16, `--cpu-offload-gb 50` | 3.3 | \n| Transformers + Accelerate | 0.16 | \n\nOffloading only the experts doubles vLLM's speed, and FP8 helps on top (half the bytes per expert, more of the model on the GPU). vLLM's other offload backend, which copies groups of layers ahead of time, ran under 1 tok/s.\n\nIf the experts fit in RAM, llama.cpp is hard to beat. If they don't, you have to stream only what the router picks, through a bounded cache, fast enough that the SSD is the bottleneck and not Python, ideally reusing the model's fast expert kernels. That's Qwen3.8-Flash-Next on a 32 GB Mac, and Transformers couldn't do it out of the box.\n\n## \n\t\tMaking Qwen3.8-Flash-Next work on my 32 GB Mac with Transformers\n\t\n\nStrata was already too slow for me on the laptop, and Transformers was not going to beat it anytime soon. But the Accelerate path was so far off that I wanted to try fixing offloading in Transformers itself, with the big model on my laptop as the target, knowing it would probably stay too slow for how I work.\n\nAll of the code was built with AI: the Transformers PR, the model-specific\npackage and the Metal kernels. My part was choosing what to try, reading the\nprofiles, and deciding what to keep. Everything is in a Space:\n[tarekziade/moe-expert-streaming](https://huggingface.co/spaces/tarekziade/moe-expert-streaming).\n\n### \n\t\tStart small\n\t\n\nI started with [OLMoE-1B-7B](https://huggingface.co/allenai/OLMoE-1B-7B-0924), which fits in memory, so I could compare every token\nwith vanilla Transformers.\n\nFirst version: experts on disk, an LRU cache on the device, and a Python loop\nrunning one expert at a time. With every expert already cached, it did 10.1\ntok/s. Vanilla did ~32. The I/O was fine, my loop was not: vanilla uses\n`grouped_mm`, which runs all the experts of a layer in one kernel.\n\nThe fix, and the key idea of this project, was to turn the cache into a **slot\npool**: one preallocated device tensor shared by all layers, plus a per-layer\ntable from expert id to slot. Remap the router's expert ids to slots and the\npool looks like a regular expert bank, so `grouped_mm` runs on it as is. No\ncustom expert matmul needed. That got it to 34.6 tok/s, against 38.1 for\nvanilla.\n\n### \n`experts_offload`\n\nI then turned the standalone loader into an option of `from_pretrained`, in\n[huggingface/transformers#50717](https://github.com/huggingface/transformers/pull/50717).\nTo try it, install Transformers from that PR:\n\n```\npip install \"git+https://github.com/huggingface/transformers.git@refs/pull/50717/head\"\n```\n\nThen, moving from OLMoE to the real target, Qwen3.8-Flash-Next:\n\n```\npath = snapshot_download(\"tarekziade/Qwen3.8-Flash-Next-expert-streaming\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    f\"{path}/dense\",\n    dtype=torch.bfloat16,\n    device_map={\"\": \"mps\", \"experts.*\": \"disk\"},\n    quantization_config=MetalConfig(bits=8),\n    experts_offload={\n        \"cache_bytes\": 12 << 30,\n        \"quantized_experts\": f\"{path}/experts-q3\",\n    },\n)\n```\n\n[That repo](https://huggingface.co/tarekziade/Qwen3.8-Flash-Next-expert-streaming)\nholds the output of the project's\n[conversion script](https://github.com/tarekziade/moe-expert-streaming#readme):\nthe dense weights, the experts quantized to 3 bits, and the n-gram table.\n\nExperts mapped to `\"disk\"` go through the streaming path instead of Accelerate's\noffload: the loader skips their tensors and MoE layers switch to an offloaded\nimplementation. Misses are read by 8 I/O threads, one contiguous read per\nexpert. The next two layers' routers run early on the current hidden states to\nguess which experts to prefetch. Experts can be quantized offline so reads are\nsmaller. It works for MoE models with the standard experts layout.\n\nThe API is still being discussed with the Transformers maintainers. The\n`device_map` form came out of that, and quantized experts will probably go\nthrough the per-layer quantization plan of\n[#50245](https://github.com/huggingface/transformers/pull/50245) rather than\ntheir own option.\n\n### \n\t\tRunning 121 GB of experts on a 32 GB Mac\n\t\n\nQwen3.8-Flash-Next has 48 layers of 512 experts (10 active), 121 GB in FP8. Streaming them isn't enough: there's also a 51 GB n-gram embedding table that needs its own disk path. So:\n\n- experts quantized to 3 bits: 49 GB on disk, 12 GiB cached on the device\n- dense layers in 8 bits (~5.4 GB, +0.2% perplexity vs BF16)\n- the n-gram table stays on disk, rows read when needed\n\nThe conversion takes ~5 hours. After loading, it uses ~18 GiB, and a 6K prompt peaks at 25.4 GiB. That's about the limit: I have to close the browser, or macOS starts swapping.\n\nThe model-specific bits (the n-gram table, Metal kernels) live in a small\npackage, [moe-expert-streaming](https://github.com/tarekziade/moe-expert-streaming),\non top of the PR. The first version ran at about 10 tok/s.\n\nThen I installed Strata on the same M5, with the same prompts:\n\n|  | Transformers | Strata | Gap | \n|---|---|---|---|\n| Decode, 200 tokens | 10.0 tok/s | 14.8 | 1.5x | \n| Prefill, 4K prompt | 68 tok/s | 220 | 3.2x | \n| Decode after a 6K document | 7.3 tok/s | 13.7 | 1.9x | \n\nStrata 0.1.60 used IQ3_XXS experts. Both ran greedy, with thinking, MTP and\nspeculative drafts off, and 1,024-token prefill chunks. The full setup and raw\nresults are on the Space's\n[details page](https://tarekziade-moe-expert-streaming.static.hf.space/details.html).\n\nI also tried [vllm-metal](https://github.com/vllm-project/vllm-metal), vLLM on\nApple GPUs through MLX, which recently got expert offloading. It streams the\nrouted experts from an MLX checkpoint through a bounded cache, like this\nproject. But it can't run Qwen3.8-Flash-Next on 32 GB, for the n-gram reason\nabove: only the experts are streamed, and the n-gram tables stay in memory.\nThat's 32 GB in the MLX checkpoints on the Hub, before the ~5 GB of dense layers and any expert cache.\n\n## \n\t\tClosing the gap\n\t\n\nMost of the time wasn't spent on experts. Here's what helped:\n\n- **Committing GPU work early.**[PyTorch](https://github.com/pytorch/pytorch) on MPS holds small kernels until the\nnext sync, so the GPU idled while Python worked. Forcing a commit after each\nblock gave +10%. This should really be fixed in PyTorch.\n- **Metal kernels** for the parts without a fast path on macOS: Gated DeltaNet\nand the hyper-connections.\n- **Strata's own weights.** I switched to[Strata's GGUF weights](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF) , reducing dense\nweight traffic from 5.4 GB to 3.9 GB per token and individual expert size from\n2.15 MB to 1.75 MB, with a separate cache pool for each expert size.\n- **Prefill** , we went from 68 to ~228 tok/s using bigger chunks, and reading the next layer's\nexperts while the current layer is being computed.\n- **Long documents** : attention over the indexer's selected blocks only, and a\nKV cache that grows in place instead of being copied at every token.\n- **Misses** : Strata doesn't hide them, it waits like I do, and no cache policy\nbeat LRU by much. What helped was reading just one guessed expert per layer,\nwhile the SSD would otherwise be idle.\n\nHere's one MoE layer for one token, as it runs now:\n\nThese are the final results after these optimizations on the same M5 with the same prompts:\n\n|  | Transformers | Strata | Gap | \n|---|---|---|---|\n| Decode, 200 tokens | 13.9 tok/s | 14.8 | 1.06x | \n| Prefill, 4K prompt | 228 tok/s | 220 | 1.0x | \n| Decode after a 6K document | 12.4 tok/s | 13.7 | 1.1x | \n\nThat's about 6% behind Strata on short decode, with slightly faster prefill, all through Transformers.\n\nMost of what's left is host round trips, ~10 ms per token. Waking up after a sync is only ~1 ms of that. The rest is the GPU running dry while Python queues the next layer. Strata queues the whole token's work up front, and the GPU waits on a Metal shared event at each layer while the host reads experts. PyTorch doesn't expose that. I tried to fake it with a kernel spinning on a flag the CPU writes, and the GPU took anywhere from a second to never to see the write. The real fix would be a native op on PyTorch's MPS stream, with a risk of hangs, so I haven't built it.\n\n## \n\t\tConclusion\n\t\n\nIt runs at ~14 tok/s, close to Strata, and it's still too slow for how I use coding agents. Orthrus keeps doing the real work.\n\nTransformers didn't beat Strata, and closing the last ~6% would take deeper\nchanges in the stack, starting with control over that per-layer sync, which\nPyTorch doesn't give you today. The interesting part is how it got that\nclose: streaming only the routed experts, a bounded cache that `grouped_mm` runs\non as is, and a few targeted Metal kernels. There's no C++ extension in the\nproject: just Python, Metal kernels compiled at runtime with\n`torch.mps.compile_shader`, and prebuilt [kernels from the Hub](https://huggingface.co/kernels-community).\nAll the Metal kernels written for this experiment are now\n[published on the Hub](https://tarekziade-moe-expert-streaming.static.hf.space/details.html#published-kernels),\nand the project loads them from there.\n\nBeating Strata was never the only goal, though. Building this was a great way to learn how MoEs and LLMs work from a systems perspective: how sparse the routing really is, how much an expert cache buys you, where memory bandwidth goes per token, and how much time the host and the GPU spend waiting on each other. Transformers makes all of that easy to poke at, one module at a time.\n\nThe generic expert streaming implementation is in\n[huggingface/transformers#50717](https://github.com/huggingface/transformers/pull/50717),\nand the benchmarks are in the [Space](https://huggingface.co/spaces/tarekziade/moe-expert-streaming).\nFeedback on the approach and the API is welcome.\n\nThanks to Pedro, Lysandre and Arthur for their excellent feedback, and to Harry for helping test vLLM.\n\nFinally, thanks to Niko Veit for [Strata](https://github.com/niko1221/strata).\nStrata set the bar for what's possible with expert streaming on consumer\nhardware, and having those numbers to chase is what pushed this work as far as\nit went.", "url": "https://wpnews.pro/news/180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers", "canonical_source": "https://huggingface.co/blog/moe-expert-streaming", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 17:13:37.955044+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["Hugging Face", "Transformers", "Qwen3.8-Flash-Next", "Strata", "llama.cpp", "vLLM", "Accelerate", "Qwen 3.8 27B"], "also_reported_by": [], "takeaway": "Streaming MoE experts from SSD reaches within 6% of Strata on token generation, while llama.cpp's --moe-cache-mib GPU expert cache fragments the graph from 2 to 50 splits per token and loses about 10% even when all experts fit.", "recommendation": {"advice": "For MoE models that exceed VRAM, stream cold experts from SSD and keep only hot experts resident; the source measured SSD streaming within 6% of Strata, versus roughly 10% loss from llama.cpp's GPU expert cache.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers", "markdown": "https://wpnews.pro/news/180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers.md", "text": "https://wpnews.pro/news/180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers.txt", "jsonld": "https://wpnews.pro/news/180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers.jsonld"}}