# Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs > Source: > Published: 2026-08-03 14:01:03+00:00 Member-only story # Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs A 1-trillion-parameter model sounds impossible to run — until you realize that most of those parameters are inactive for any given token. Mixture of Experts (MoE) architectures replace the dense feed-forward layer with N parallel “expert” networks, activating only the top-k for each token via a learned router. The result: 8× the parameter count (expert count N=8 vs a single dense FFN) and only ~2× the compute (top-k=2 routing activates 2/8 = 25% of experts per token). Mixtral 8×7B sits at ~47B total / ~13B active parameters — a 3.6× ratio rather than a clean 4× because shared attention layers don’t multiply with experts. This is how Mixtral, GPT-4 (reportedly), and Gemini achieve large effective capacity at tractable inference cost. Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information. ## The Basic Mechanism Before measuring the gains, it helps to see the routing logic in code — the contrast with a dense FFN (feed-forward network) makes the FLOPs (floating-point operations) arithmetic immediate. A standard transformer FFN processes every token through the same parameters: ``` FFN(x) = W_down × ReLU(W_up × x) ```