GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.

GPT-4 has 1.8 trillion total parameters but activates only about 36 billion (2%) per token due to its Mixture-of-Experts (MoE) architecture, where a routing function selects k experts for each token. This design is a deliberate mathematical choice for efficiency, not a limitation, according to the analysis.

Member-only story GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature. The number 1.8 trillion is not the size of GPT-4. It is the size of the parameter dictionary. When GPT-4 processes a token, it activates approximately 36 billion parameters — roughly 2% of the total. The remaining 98% sit dormant for that specific computation. This is not a limitation. It is an architectural choice with a precise mathematical justification, and understanding that justification changes how you think about deploying these systems at scale. The argument is not stylistic. It is mathematical. The Mixture-of-Experts Architecture GPT-4 is a Mixture-of-Experts MoE model. In a standard transformer feedforward block, every token passes through the same parameters. In MoE, the feedforward layer is replaced by a set of E expert networks, and a routing function selects k of them for each token. Formally, the MoE layer computes: y = Σᵢ∈Top-k x G x ᵢ · Eᵢ x Where: - x is the input token representation - Eᵢ x is the output of expert i applied to x - G x ᵢ is the gating weight for expert i - Top-k x is the set of k experts with highest gating scores