{"slug": "gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a", "title": "GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.", "summary": "GPT-4 has 1.8 trillion total parameters but activates only about 36 billion (2%) per token due to its Mixture-of-Experts (MoE) architecture, where a routing function selects k experts for each token. This design is a deliberate mathematical choice for efficiency, not a limitation, according to the analysis.", "body_md": "Member-only story\n\n# GPT-4 Has 1.8 Trillion Parameters. It Uses 2% of Them Per Token. Here Is Why That Is a Feature.\n\nThe number 1.8 trillion is not the size of GPT-4. It is the size of the parameter dictionary.\n\nWhen GPT-4 processes a token, it activates approximately 36 billion parameters — roughly 2% of the total.\n\nThe remaining 98% sit dormant for that specific computation.\n\nThis is not a limitation. It is an architectural choice with a precise mathematical justification, and understanding that justification changes how you think about deploying these systems at scale.\n\nThe argument is not stylistic. It is mathematical.\n\n## The Mixture-of-Experts Architecture\n\nGPT-4 is a Mixture-of-Experts (MoE) model. In a standard transformer feedforward block, every token passes through the same parameters.\n\nIn MoE, the feedforward layer is replaced by a set of E expert networks, and a routing function selects k of them for each token.\n\nFormally, the MoE layer computes:\n\n**y = Σᵢ∈Top-k(x) G(x)ᵢ · Eᵢ(x)**\n\nWhere:\n\n- x is the input token representation\n\n- Eᵢ(x) is the output of expert i applied to x\n\n- G(x)ᵢ is the gating weight for expert i\n\n- Top-k(x) is the set of k experts with highest gating scores", "url": "https://wpnews.pro/news/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a", "canonical_source": "https://pub.towardsai.net/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-is-a-feature-5d363d3847bd?source=rss----98111c9905da---4", "published_at": "2026-07-20 06:15:30+00:00", "updated_at": "2026-07-20 13:21:45.098817+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-infrastructure"], "entities": ["GPT-4", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a", "markdown": "https://wpnews.pro/news/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a.md", "text": "https://wpnews.pro/news/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a.txt", "jsonld": "https://wpnews.pro/news/gpt-4-has-1-8-trillion-parameters-it-uses-2-of-them-per-token-here-is-why-that-a.jsonld"}}