{"slug": "mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms", "title": "Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs", "summary": "Mixture of Experts (MoE) architectures enable 1-trillion-parameter large language models by activating only a subset of expert networks per token, achieving 8× parameter count with only ~2× compute. Mixtral 8×7B, with 47B total and 13B active parameters, exemplifies this efficiency, as do GPT-4 and Gemini.", "body_md": "Member-only story\n\n# Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs\n\nA 1-trillion-parameter model sounds impossible to run — until you realize that most of those parameters are inactive for any given token. Mixture of Experts (MoE) architectures replace the dense feed-forward layer with N parallel “expert” networks, activating only the top-k for each token via a learned router. The result: 8× the parameter count (expert count N=8 vs a single dense FFN) and only ~2× the compute (top-k=2 routing activates 2/8 = 25% of experts per token). Mixtral 8×7B sits at ~47B total / ~13B active parameters — a 3.6× ratio rather than a clean 4× because shared attention layers don’t multiply with experts. This is how Mixtral, GPT-4 (reportedly), and Gemini achieve large effective capacity at tractable inference cost.\n\nDisclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information.\n\n## The Basic Mechanism\n\nBefore measuring the gains, it helps to see the routing logic in code — the contrast with a dense FFN (feed-forward network) makes the FLOPs (floating-point operations) arithmetic immediate.\n\nA standard transformer FFN processes every token through the same parameters:\n\n```\nFFN(x) = W_down × ReLU(W_up × x)\n```\n\n", "url": "https://wpnews.pro/news/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms", "canonical_source": "https://pub.towardsai.net/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms-655d20b68eb8?source=rss----98111c9905da---4", "published_at": "2026-08-03 14:01:03+00:00", "updated_at": "2026-08-03 14:22:07.069634+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "machine-learning"], "entities": ["Mixtral", "GPT-4", "Gemini", "Google"], "alternates": {"html": "https://wpnews.pro/news/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms", "markdown": "https://wpnews.pro/news/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms.md", "text": "https://wpnews.pro/news/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms.txt", "jsonld": "https://wpnews.pro/news/mixture-of-experts-how-sparse-gating-enables-1t-parameter-llms.jsonld"}}