Member-only story
The number 1.8 trillion is not the size of GPT-4. It is the size of the parameter dictionary.
When GPT-4 processes a token, it activates approximately 36 billion parameters — roughly 2% of the total.
The remaining 98% sit dormant for that specific computation.
This is not a limitation. It is an architectural choice with a precise mathematical justification, and understanding that justification changes how you think about deploying these systems at scale.
The argument is not stylistic. It is mathematical.
The Mixture-of-Experts Architecture #
GPT-4 is a Mixture-of-Experts (MoE) model. In a standard transformer feedforward block, every token passes through the same parameters.
In MoE, the feedforward layer is replaced by a set of E expert networks, and a routing function selects k of them for each token.
Formally, the MoE layer computes:
y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x) Where:
- x is the input token representation
- Eᵢ(x) is the output of expert i applied to x
- G(x)ᵢ is the gating weight for expert i
- Top-k(x) is the set of k experts with highest gating scores