{"slug": "building-intuition-about-llm-parameter-counts", "title": "Building intuition about LLM parameter counts", "summary": "A developer building a GPT-2 implementation in JAX discovered that token embeddings and the output head account for nearly half of the model's 163 million parameters, while attention layers use fewer parameters than feed-forward networks. The author created a visualizer to show parameter breakdowns for different GPT-2 sizes, highlighting how embedding components dominate in smaller models, especially with large vocabularies.", "body_md": "When I was [building my GPT-2 implementation in JAX](https://www.gilesthomas.com/2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax),\nI started with just token embeddings for the input, and a separate output head (as I was not using\n[weight tying](https://www.gilesthomas.com/2026/03/llm-from-scratch-32g-interventions-weight-tying)). It wasn't an\nLLM -- no Transformer blocks, no attention, no feed-forward networks.\n\nI was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the \"small\" settings I was using to train -- specifically, an embedding dimension of 768. However, I realised I shouldn't be -- with a vocab size of 50,257, each of those components is essentially a matrix, and that is indeed over 38 million numbers.\n\nBut the finished LLM at the end of the project was only 163 million parameters -- that meant that the input and output components alone were almost half of it. That felt like a surprisingly large percentage.\n\nI had a similar shock when I was first looking into [the feed-forward network](https://www.gilesthomas.com/2025/08/llm-from-scratch-17-the-feed-forward-network),\nand realised that it had roughly twice as many parameters as the attention layers.\n\nWhen we learn about the internals of LLMs, a lot of the focus is on the attention mechanism. This makes sense -- it's the hardest part to get your head around. The rest of the setup, at least for simple GPT-2 type models, is fairly standard stuff.\n\nBut that means that it is easy to overestimate how much of the total parameter count of the model attention uses up -- especially for smaller models, where the token embeddings and the output head are so large in comparison to the Transformer layers that make up the actual body of the LLM.\n\nOpenAI released GPT 5.6 today, so I decided to take its \"Sol\" variant for a ride\nin Codex and asked it to write [a visualiser](https://www.gilesthomas.com/post-assets/llm-parameter-counts/parameter-count-visualizer.html).\nIt shows\nbreakdowns of how the parameters are split between embeddings, attention, the FFNs, and\nthe output head\nfor different sizes of GPT-2 models (or your own custom settings\nwith the same architecture), and you can also add/remove weight tying and QKV bias.\nIt did a really good job -- check it out! Here's a screenshot of what it showed\nfor GPT-2 small without weight tying.\n\nIt's well worth a play. In particular, it's interesting to see what happens as the number of tokens in the vocab gets very large (many modern models have hundreds of thousands). You can very easily create a \"tiny\" model which is almost entirely embeddings and the output head.", "url": "https://wpnews.pro/news/building-intuition-about-llm-parameter-counts", "canonical_source": "https://www.gilesthomas.com/2026/07/llm-parameter-counts", "published_at": "2026-07-10 20:49:32.990447+00:00", "updated_at": "2026-07-10 20:49:34.971740+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "developer-tools"], "entities": ["OpenAI", "GPT-2", "JAX", "Codex", "GPT-5.6"], "alternates": {"html": "https://wpnews.pro/news/building-intuition-about-llm-parameter-counts", "markdown": "https://wpnews.pro/news/building-intuition-about-llm-parameter-counts.md", "text": "https://wpnews.pro/news/building-intuition-about-llm-parameter-counts.txt", "jsonld": "https://wpnews.pro/news/building-intuition-about-llm-parameter-counts.jsonld"}}