Member-only story
A 1-trillion-parameter model sounds impossible to run — until you realize that most of those parameters are inactive for any given token. Mixture of Experts (MoE) architectures replace the dense feed-forward layer with N parallel “expert” networks, activating only the top-k for each token via a learned router. The result: 8× the parameter count (expert count N=8 vs a single dense FFN) and only ~2× the compute (top-k=2 routing activates 2/8 = 25% of experts per token). Mixtral 8×7B sits at ~47B total / ~13B active parameters — a 3.6× ratio rather than a clean 4× because shared attention layers don’t multiply with experts. This is how Mixtral, GPT-4 (reportedly), and Gemini achieve large effective capacity at tractable inference cost.
Disclaimer: The opinions expressed in this article are my own and do not represent the views of Google. This content is based solely on publicly available information.
The Basic Mechanism #
Before measuring the gains, it helps to see the routing logic in code — the contrast with a dense FFN (feed-forward network) makes the FLOPs (floating-point operations) arithmetic immediate.
A standard transformer FFN processes every token through the same parameters:
FFN(x) = W_down × ReLU(W_up × x)