Building intuition about LLM parameter counts

A developer building a GPT-2 implementation in JAX discovered that token embeddings and the output head account for nearly half of the model's 163 million parameters, while attention layers use fewer parameters than feed-forward networks. The author created a visualizer to show parameter breakdowns for different GPT-2 sizes, highlighting how embedding components dominate in smaller models, especially with large vocabularies.

When I was building my GPT-2 implementation in JAX https://www.gilesthomas.com/2026/07/llm-from-scratch-34b-building-and-training-gpt-2-small-in-jax , I started with just token embeddings for the input, and a separate output head as I was not using weight tying https://www.gilesthomas.com/2026/03/llm-from-scratch-32g-interventions-weight-tying . It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks. I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train -- specifically, an embedding dimension of 768. However, I realised I shouldn't be -- with a vocab size of 50,257, each of those components is essentially a matrix, and that is indeed over 38 million numbers. But the finished LLM at the end of the project was only 163 million parameters -- that meant that the input and output components alone were almost half of it. That felt like a surprisingly large percentage. I had a similar shock when I was first looking into the feed-forward network https://www.gilesthomas.com/2025/08/llm-from-scratch-17-the-feed-forward-network , and realised that it had roughly twice as many parameters as the attention layers. When we learn about the internals of LLMs, a lot of the focus is on the attention mechanism. This makes sense -- it's the hardest part to get your head around. The rest of the setup, at least for simple GPT-2 type models, is fairly standard stuff. But that means that it is easy to overestimate how much of the total parameter count of the model attention uses up -- especially for smaller models, where the token embeddings and the output head are so large in comparison to the Transformer layers that make up the actual body of the LLM. OpenAI released GPT 5.6 today, so I decided to take its "Sol" variant for a ride in Codex and asked it to write a visualiser https://www.gilesthomas.com/post-assets/llm-parameter-counts/parameter-count-visualizer.html . It shows breakdowns of how the parameters are split between embeddings, attention, the FFNs, and the output head for different sizes of GPT-2 models or your own custom settings with the same architecture , and you can also add/remove weight tying and QKV bias. It did a really good job -- check it out Here's a screenshot of what it showed for GPT-2 small without weight tying. It's well worth a play. In particular, it's interesting to see what happens as the number of tokens in the vocab gets very large many modern models have hundreds of thousands . You can very easily create a "tiny" model which is almost entirely embeddings and the output head.