# GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature. > Source: > Published: 2026-07-20 06:15:30+00:00 Member-only story # GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature. The number 1.8 trillion is not the size of GPT-4. It is the size of the parameter dictionary. When GPT-4 processes a token, it activates approximately 36 billion parameters — roughly 2% of the total. The remaining 98% sit dormant for that specific computation. This is not a limitation. It is an architectural choice with a precise mathematical justification, and understanding that justification changes how you think about deploying these systems at scale. The argument is not stylistic. It is mathematical. ## The Mixture-of-Experts Architecture GPT-4 is a Mixture-of-Experts (MoE) model. In a standard transformer feedforward block, every token passes through the same parameters. In MoE, the feedforward layer is replaced by a set of E expert networks, and a routing function selects k of them for each token. Formally, the MoE layer computes: **y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x)** Where: - x is the input token representation - Eᵢ(x) is the output of expert i applied to x - G(x)ᵢ is the gating weight for expert i - Top-k(x) is the set of k experts with highest gating scores