Every Frontier Open Model Is a MoE Now. Here Is What That Does to Your Hardware Math

Every notable open-weight frontier model released in the past year uses a Mixture-of-Experts (MoE) architecture, according to an analysis by Vetted Consumer. The shift means total parameters range from 26 billion to 2.8 trillion while active parameters per token cluster between 3 billion and 50 billion, effectively freezing compute cost per token and scaling only stored parameters. This changes the hardware calculus for local AI runners, who must now prioritize memory capacity over raw GPU speed.

Line up every notable open-weight release of the past year and one pattern jumps out. Kimi K3: Mixture-of-Experts. DeepSeek V4: MoE. Inkling: MoE. GLM-5.2, MiniMax M3, Nemotron 3, gpt-oss, Qwen's whole current lineup, Gemma 4's larger sizes: MoE, all of them. The dense frontier model, the kind where every parameter works on every token, has quietly stopped being made at the top end. That is not trivia. It flips a rule that local-AI hardware buying was built on, and if you are choosing between a fast GPU and a big pile of memory in 2026, the sparsity takeover should decide the argument. The lineup, in one table A Mixture-of-Experts model splits its feed-forward layers into many small "experts" and routes each token through only a few of them, so two numbers describe it: total parameters what you store and active parameters what computes per token . Our MoE explainer https://vettedconsumer.com/mixture-of-experts-moe-explained-why-active-parameters-decide-what-runs-on-your-machine/ covers the mechanism. Here is what the current generation looks like on those two axes, using vendor-published figures: | Model 2025–26 releases | Total params | Active per token | Active share | |---|---|---|---| | Kimi K2.7 Code https://vettedconsumer.com/kimi-k2-7-code-trillion-parameter-open-coder-local-reality/ / K2 Thinking Inkling https://vettedconsumer.com/inkling-mira-muratis-first-open-model-is-a-975b-moe-you-can-actually-run/ GLM-5.2 https://vettedconsumer.com/glm-5-2-the-most-powerful-open-weight-model-yet-and-the-brutal-reality-of-running-it-locally/ MiniMax M3 https://vettedconsumer.com/minimax-m3-the-first-open-weight-multimodal-frontier-model-and-the-license-catch/ Vendor-stated figures, linked in our per-model coverage and the sources below. NVIDIA's Nemotron-3-Ultra 550B is MoE as well. Kimi K3's active count is not officially published; only its routing ratio is. Read the columns separately and the design brief behind the whole generation appears. Totals span two orders of magnitude, 26B to 2.8 trillion. Active counts barely move: almost everything lands between 3B and 50B, and the frontier class clusters at 3 to 5 percent active. Model makers have effectively frozen the compute cost per token and are scaling the one axis that got cheap: stored parameters. Why every lab converged on this The economics were mapped out in the research years before the products arrived. The original 2017 MoE paper https://arxiv.org/abs/1701.06538?ref=vettedconsumer.com Shazeer et al. showed you could grow capacity over 1,000x with "only minor losses in computational efficiency." DeepMind's Unified Scaling Laws for Routed Language Models https://arxiv.org/abs/2202.01169?ref=vettedconsumer.com Clark et al., 2022 made the key point precise: for routed models, "parameter count and computational requirement form two independent axes" of improvement. Knowledge scales with what you store; cost scales with what you compute. A lab that can add knowledge without adding per-token compute will do it every time, because inference is what they pay for at scale. The recipe the 2026 models share traces to DeepSeekMoE https://arxiv.org/abs/2401.06066?ref=vettedconsumer.com Dai et al., 2024 : slice the experts fine hundreds of small ones, pick several per token and add a shared expert that always fires to hold common knowledge. Look back at the table with that in mind: Kimi's 384 or 896 experts, Inkling's 256 routed plus 2 shared, gpt-oss's 128 experts with top-4 routing. Same blueprint, different sizes. Mixtral https://arxiv.org/abs/2401.04088?ref=vettedconsumer.com proved the approach in the open in 2024, and the DeepSeek-V3 report https://arxiv.org/abs/2412.19437?ref=vettedconsumer.com demonstrated a 671B model trained for a fraction of dense-frontier cost. After that, the argument was over. What it does to your hardware math For a local runner, the two-axis design lands as one sentence: total parameters decide whether it fits, active parameters decide how fast it runs. Generation speed is memory-bandwidth-bound our Bandwidth, Not TFLOPS https://vettedconsumer.com/bandwidth-not-tflops-what-sets-your-local-llm-speed-and-why-the-newest-card-isnt-always-fastest/ piece covers why , and each token only reads the experts that fire, so a 428B model decodes like a ~23B one. But the router picks different experts every token, so all 428B must sit somewhere reachable. Capacity is the entry ticket; bandwidth only sets your speed once you are in. Worked once with real numbers: GLM-5.2 stores 753B parameters, but each token reads roughly 40B of them. At a 4-bit quant that is about 22GB of weight-reads per token, so a 512GB Mac Studio moving 819 GB/s has a theoretical decode ceiling around 35 tokens per second on a model twenty times bigger than anything the box could hold dense. The stored size fell out of the speed equation; only the active slice and your bandwidth remain. Owners see the same thing from the cheap seats. One B60 owner's published llama-bench runs https://www.reddit.com/r/LocalLLaMA/comments/1qsenpy/dont buy b60 for llms/o2xwzmt/?ref=vettedconsumer.com put a dense 14B Q8 at 24.7 tokens per second and a 23B-A3B MoE Q4 at 63 on the same card, the same day. The quant difference explains part of that gap, but most of it is sparsity: the model with more total parameters decoded two and a half times faster because it reads far fewer of them per token. That reweights the whole buying decision: Capacity boxes win. The machines this generation favors hold a lot of memory at moderate bandwidth: a 512GB Mac Studio runs 2-to-3-bit builds of the 753B-to-1T class in one box, as we covered with Inkling https://vettedconsumer.com/what-hardware-runs-inkling-975b-fits-on-one-box-unlike-kimi-k3/ , 128GB unified-memory mini-PCs https://vettedconsumer.com/unified-memory-explained-why-mini-pcs-can-run-70b-models-a-big-gpu-cant-and-where-they-slow-down/ gpt-oss-120b territory , or an ordinary desktop with a mid-range GPU and a big pile of system RAM. The GPU + RAM split got a purpose-built tool. llama.cpp's --n-cpu-moe flag pushes the routed expert weights of N layers into system RAM while attention, KV cache, the router, and shared experts stay on the GPU, exactly the split MoE architecture invites, since the always-active parts stay in fast VRAM and the sparsely-touched experts sit in slow-but-huge RAM. The community offload guide https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide?ref=vettedconsumer.com is the reference. One caveat to keep straight: offload is a fit-enabler, not a speed-up. If the model fit in VRAM, offloading makes it slower. Its value is running a 117B or 284B total on a 12-to-24GB card at usable speed. The floor dropped through the floor. The most-upvoted demonstration of the trend is a post from a builder in Burma https://www.reddit.com/r/LocalLLaMA/comments/1qxcm5g/no nvidia no problem my 2018 potato 8th gen i3/?ref=vettedconsumer.com running a 16B MoE on a 2018 dual-core i3 laptop with 16GB of RAM at ~9.6 tokens per second decode: "MoE is the 'Cheat Code': 16B parameters sounds huge, but it only calculates 2.4B per token." Dual-channel RAM and a sparse model, no GPU involved. Mid-VRAM speed cards lost their pitch. The awkward tier is the 16-to-32GB GPU bought to run models that fit entirely in VRAM. The interesting open models no longer fit: the class that used to be a 30B dense download is now an 80B-to-284B-total MoE. A card in this tier is still excellent for the parts that stay hot attention, KV, shared experts, prompt processing , but it now does its best work paired with 64 to 128GB of system RAM rather than standing alone. The fine print Three things keep this from being a free lunch. First, capacity got expensive at the worst moment : the memory price crisis https://vettedconsumer.com/why-everything-got-more-expensive-the-memory-crisis-explained-via-dave2d/ means the RAM and high-capacity boxes this era rewards cost sharply more than a year ago. Second, active parameters are not a quality rating : a 35B-A3B is much stronger than a dense 3B it stores 10x the knowledge but it is not a dense 35B either; judge each model on evals and owner reports, not the ratio. Third, fast MoE models break some old speed tricks : as we found in our speculative decoding explainer https://vettedconsumer.com/speculative-decoding-explained-the-free-speed-toggle-your-local-llm-is-probably-not-using/ , measured tests show speculation going break-even to slower on sparse models that are already quick, because every drafted token drags fresh expert weights into the verify pass. The decision cheat-sheet | Your memory situation | What the MoE era gives you | |---|---| | 16GB RAM, no GPU dual-channel | 16B-class MoE at near-reading speed on CPU/iGPU | | 8–12GB VRAM + 64GB RAM | Qwen3.6-35B-A3B, gpt-oss-20b fully; larger with --n-cpu-moe | | 24GB VRAM + 128GB RAM | gpt-oss-120b, Qwen3-Coder-Next, DeepSeek-V4-Flash via expert offload | | 128GB unified memory | The 117B–284B totals at Q4-ish quants, no offload games | | 512GB Mac Studio | 2-to-3-bit builds of the 753B–1T class in one box | | Anything smaller than a rack | Kimi K3 and DeepSeek-V4-Pro stay rent-or-API; see the | If one sentence should survive this article, it is the buying rule the trend leaves behind: in the MoE era, buy memory capacity first and bandwidth second, because the models will keep growing on exactly the axis that capacity serves. Every spec in the table above says the labs have already made that bet. Sources and how we researched this - Research spine: Shazeer et al. 2017, "Outrageously Large Neural Networks" arXiv:1701.06538 https://arxiv.org/abs/1701.06538?ref=vettedconsumer.com ; Clark et al. 2022, "Unified Scaling Laws for Routed Language Models" arXiv:2202.01169 https://arxiv.org/abs/2202.01169?ref=vettedconsumer.com ; Dai et al. 2024, DeepSeekMoE arXiv:2401.06066 https://arxiv.org/abs/2401.06066?ref=vettedconsumer.com ; Jiang et al. 2024, Mixtral of Experts arXiv:2401.04088 https://arxiv.org/abs/2401.04088?ref=vettedconsumer.com ; DeepSeek-V3 Technical Report arXiv:2412.19437 https://arxiv.org/abs/2412.19437?ref=vettedconsumer.com . - Model figures: vendor model cards and announcements as covered in our linked per-model spotlights Kimi K3, Inkling, GLM-5.2, MiniMax M3, Kimi K2.7 , plus OpenAI's gpt-oss announcement https://openai.com/index/introducing-gpt-oss/?ref=vettedconsumer.com 116.8B total, 5.1B active, 128 experts and Google's Gemma 4 documentation. - Offload mechanics: llama.cpp https://github.com/ggml-org/llama.cpp?ref=vettedconsumer.com and the MoE CPU-offload guide https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide?ref=vettedconsumer.com . - Community reports: the linked r/LocalLLaMA threads, quoted verbatim and attributed. - We have not benchmarked these models first-hand. This piece synthesizes vendor documentation, the peer-reviewed literature, and attributed owner reports, cited inline. Related: Mixture-of-Experts, explained · Bandwidth, Not TFLOPS · Unified memory, explained · Speculative decoding, explained · The hardware cheat sheet