Multihead Attention: How One Model Reads a Sentence 144 Ways

Multihead attention in transformer models uses 144 separate attention heads across 12 layers to read a sentence from different angles, each producing a 64-length update that is concatenated and recombined into a single 768-length update. This mechanism allows the model to capture diverse linguistic patterns by learning what each head attends to.

Article 1- https://medium.com/p/5807f169db30 https://medium.com/p/5807f169db30 Article 2 — https://medium.com/p/75df74bb5f07 https://medium.com/p/75df74bb5f07 A single self-attention unit gives the model one way to look at a sentence. Multihead attention gives it many. Instead of one unit per layer, each of the 12 layers has 12 separate attention heads. That is 144 heads in total, and each one gets to read the sentence from its own angle. Each head proposes its own update to the word embedding. So we add one more matrix whose job is to take those 12 separate updates, combine them into one, and produce the final adjustment we send on to the rest of the model. To make sense of the math, forget the 12 heads for now. We will walk through one head, running the same self-attention steps as before, with one change: the query, key, and value vectors are shorter. Instead of length 768, they are length 64. That change does not break anything. Self-attention still works exactly the same way. Our running sentence is “I like to draw,” and the word we are updating is draw . We take the embedding for draw , shape 1 x 768 , and multiply it by the query matrix Q, shape 768 x 64 . The inner dimensions cancel and we get a query vector of shape 1 x 64 . Same word, fewer features. The keys work the same way. We project all four word embeddings, shape 4 x 768 , through the key matrix 768 x 64 to get key vectors of shape 4 x 64 . Now we compare draw to every word. We multiply the query 1 x 64 against the four key vectors 64 x 4 , which gives four raw attention values. Run those through softmax and we get the scores, shape 1 x 4 — one weight per word. We project the four embeddings once more, this time through the value matrix 768 x 64 , to get value vectors of shape 4 x 64 . Then we apply the scores. Each value vector is multiplied by its score and the results are summed: 0.62 × value I + 0.12 × value like + 0.19 × value to + 0.08 × value draw That weighted sum is the update for draw . It has length 64. This update should be added back to the original draw embedding to produce the new embedding we pass forward. There is a problem. The update is length 64, but the original embedding is length 768. They do not line up, so we cannot add them yet. The fix is a matrix that projects the update back up to length 768. A good way to think about it: projecting into the value space compressed the embedding from 768 down to 64, and this matrix decompresses the update back to 768. We multiply the update 1 x 64 by a 64 x 768 matrix to get a vector of shape 1 x 768 . Now it matches the original embedding, and the two can be added. Now we put the 12 heads back in. Each of the 12 heads produces its own length-64 update. First we concatenate them into one long vector. Twelve heads times length 64 gives 768 — which is exactly why 64 was chosen. But that is 12 updates stitched together, not a single update. We fix that with one more matrix, shape 768 x 768 . Call it multihead recombination. It turns the 12 concatenated updates into one 768-length update, and it also decides how much weight each head’s update gets in the final result. Add that final update to the input embedding, and we have the finished embedding to send on to the feedforward neural network. So multihead attention is two matrices doing quiet work at the edges: one that decompresses each head’s update, and one that merges 12 narrow views into a single answer. The open question worth sitting with: with 144 heads all reading the same sentence, what does each one actually learn to pay attention to? Multihead Attention: How One Model Reads a Sentence 144 Ways https://pub.towardsai.net/multihead-attention-how-one-model-reads-a-sentence-144-ways-50546707caca was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.