Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs

Mixture of Experts (MoE) architectures enable 1-trillion-parameter large language models by activating only a subset of expert networks per token, achieving 8× parameter count with only ~2× compute. Mixtral 8×7B, with 47B total and 13B active parameters, exemplifies this efficiency, as do GPT-4 and Gemini.

Member-only story Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs A 1-trillion-parameter model sounds impossible to run — until you realize that most of those parameters are inactive for any given token. Mixture of Experts MoE architectures replace the dense feed-forward layer with N parallel “expert” networks, activating only the top-k for each token via a learned router. The result: 8× the parameter count expert count N=8 vs a single dense FFN and only ~2× the compute top-k=2 routing activates 2/8 = 25% of experts per token . Mixtral 8×7B sits at ~47B total / ~13B active parameters — a 3.6× ratio rather than a clean 4× because shared attention layers don’t multiply with experts. This is how Mixtral, GPT-4 reportedly , and Gemini achieve large effective capacity at tractable inference cost. Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information. The Basic Mechanism Before measuring the gains, it helps to see the routing logic in code — the contrast with a dense FFN feed-forward network makes the FLOPs floating-point operations arithmetic immediate. A standard transformer FFN processes every token through the same parameters: FFN x = W down × ReLU W up × x