Member-only story
One sentence, one forward pass, every number shown: how a 768-wide vector moves through 12 decoder layers and turns into a 50,257-way probability distribution.
Feed a GPT-2 base model the four words I like to draw
and it will hand you back a next word. Most explanations stop at that sentence, wave at "attention," and move on. This one does not.
We are going to follow a single vector — the one that represents draw
— from the moment it is a row in a lookup table to the moment it becomes 50,257 numbers that sum to 1. Every matrix multiply gets its shapes written out. Every parameter count gets derived from the shapes, not asserted. Where the source material does not expose real GPT-2 values, the example is labeled a mechanics slice and the arithmetic is done on four dimensions instead of 768 — the operation is identical, only the width shrinks so you can check it by hand.
The claim this walkthrough will earn: a decoder layer never compresses the token and never replaces it with a different token. It rewrites a fixed-width vector in place, over and over, until the last vector is specific enough to name what comes next.
Here are the anchor numbers we will not hand-wave. These are GPT-2 base (“small,” 124M-class) figures.