Member-only story
One token, four numbers, and the arithmetic that turns raw attention scores into a contextual embedding inside GPT-2.
Article 1- https://medium.com/p/5807f169db30 Where we are headed. Four weights, one target. draw leans on I six times harder than on itself. The rest of this piece derives those numbers by hand.
“Attention” sounds like the hard part of a transformer. It is the part everyone name-drops and almost nobody computes on paper. Strip away the branding and what remains is arithmetic a spreadsheet could run: exponentiate a handful of numbers, divide by their sum, and take a weighted average.
The mechanics of that average are the easy part. The question worth sitting with is what gets averaged, and why the two ingredients — the weights and the values — are produced by different machinery inside the model. Get that split clear and the whole block stops being magic.
So we will not describe attention. We will run it. One four-word sentence, one target token, and every number followed from raw score to the embedding that leaves the block.