{"slug": "ai-fundamentals-attention-mechanisms-in-transformers-part-1", "title": "AI Fundamentals: Attention Mechanisms in Transformers (Part 1)", "summary": "Attention mechanisms in transformers, a core component of modern AI models, detailing how they allow tokens to weigh the relevance of other tokens. It covers additive attention introduced by Bahdanau et al. in 2014 and scaled dot-product attention from the 2017 paper 'Attention Is All You Need', including the mathematical formula and a Python implementation.", "body_md": "Consider the sentence:\n\n*Lily smiled brightly as* *she* *watched the puppy chase its tail.*\n\nYour brain instantly connects “*she*” back to the name.\n\nA transformer needs mechanisms that allow information from one token to influence the representation of another, using vectors, matrices, and a mechanism called attention.\n\nAttention is what gives a transformer a way to look around its input and ask: **“Which pieces of information actually matter to me right now?”.** The remainder of this article works through what that means in practice.\n\nAt its core, attention is a mathematical way for a model to measure relationships between pieces of information. Think about walking into a crowded room. There might be 100 people talking.\n\nIf someone across the room says your name, your brain immediately goes:\n\n**Wait. That matters.**\n\nYour attention shifts.\n\nTransformers do something surprisingly similar. For every token, the model looks at other tokens and calculates how relevant they are. Important relationships receive larger weights; less relevant ones receive smaller weights.\n\nSo instead of treating a sentence like a pile of unrelated words, the model can build a contextual representation of each token. There are four useful questions to keep in mind when learning attention:\n\nThe underlying question is simple: how related are two pieces of information? Several mechanisms answer it differently.\n\nIntroduced by Bahdanau et al. in 2014, additive attention combines two vectors and passes the result through a small neural network to produce a score:\n\nscore(q, k) = vᵀ · tanh(W₁q + W₂k)\n\nW₁ and W₂ are learned weight matrices applied to the query and key vectors respectively, and v is a learned weight vector that reduces the result to a single scalar score.\n\nThis uses an additional nonlinear computation step for each query-key pair, whereas dot-product attention can express all pairwise scores as a matrix multiplication (QKᵀ), making it particularly efficient on accelerators.\n\nThe idea is to j**ust calculate the dot product.** The dot product gives us a measure of how aligned two vectors are. And because dot products can be expressed as matrix multiplication, GPUs absolutely love this approach.\n\nThis is the version introduced in the famous 2017 paper *Attention Is All You Need*.\n\nAttention(Q, K, V) = softmax( (Q · Kᵀ) / √dₖ ) · V\n\n**Step 1: Q · Kᵀ**\n\nEvery query is compared against every key and the result gives us a collection of raw relevance scores. Basically, every token asks: *“how relevant is everyone else to me?”*\n\nThe resulting matrix contains those answers.\n\n**Step 2: divide by √dₖ**\n\nAs vector dimensions become larger, dot products can also become very large. If those numbers get too extreme, softmax becomes overly confident. Its output can become very close to 0 or 1, which makes gradients tiny and learning harder. Dividing by √dₖ keeps the numbers under control.\n\n**Step 3: softmax**\n\nNow we turn those raw scores into weights. Softmax converts the scores into normalized attention weights where everything adds up to 1.\n\nTaking an example query token: “smiled”\n\n```\nKey/token → Attention WeightsLily → 0.65smiled → 0.20brightly → 0.10 ...​​\n```\n\n**Step 4: multiply by V**\n\nFinally, those attention weights are used to calculate a weighted combination of the value vectors. The token doesn’t just remain the vector it started with. It becomes **context-aware**.\n\n``` python\nimport torchimport torch.nn.functional as Fdef scaled_dot_product_attention(q, k, v, mask=None):    d_k = q.size(-1)    scores = torch.matmul(        q,        k.transpose(-2, -1)    ) / (d_k ** 0.5)    if mask is not None:        scores = scores.masked_fill(            mask == 0,            float('-inf')        )    weights = F.softmax(scores, dim=-1)    return torch.matmul(weights, v), weights\n```\n\nIf a token can look around, **how far can it look?**\n\nThere are a couple of choices:\n\nImagine you’re at a party. With **global attention**, you can talk to anyone in the room and every token can attend to every other token. Given n tokens, you’re potentially calculating relationships between every pair of tokens. Computing the attention scores has roughly O(n²d) time complexity, while storing the attention matrix requires O(n²) memory, so the cost grows quickly.\n\nA sequence of 100 tokens? Not a big deal.\n\nA sequence of 100,000 tokens? Now we’re talking about a *lot* of relationships.\n\n**Local attention** instead gives the tokens a smaller window and dramatically reduces computation. The trade-off is obvious: **cheaper computation, but less global information.**\n\nNote that local attention is an attention mechanism that restricts which tokens directly interact. The context window of a model, by contrast, refers to the maximum sequence length the model can process in a given context.\n\nNow imagine looking through the crowd for a friend. With **soft attention**, you don’t immediately pick one person. It produces continuous weights over candidate tokens, making the operation differentiable. You might look at everyone and assign probabilities:\n\n```\nPerson A → 5%Person B → 10%Person C → 70%Person D → 15%\n```\n\nYou distribute your attention. That’s basically what standard transformer attention does. It’s differentiable, which means gradients can flow through it during training.\n\n**Hard attention** would be more like: “That’s my friend. Ignore everyone else.” That sounds simpler, but now we have a problem. The decision is discrete, so you can’t directly differentiate through the selection operation. Training hard attention therefore typically requires specialized estimators or continuous relaxations.\n\nThere are three common terms seen in attention:\n\nIn self-attention, Q, K, and V all come from the **same sequence**. So, every token can look at other tokens in that sequence.\n\nFor example:\n\nThe river carved through the canyon for centuries. Eventually, the canyon began to shape how the river flowed.\n\nWhen both sentences are present in the same context, self-attention can allow tokens in the second sentence to incorporate information from tokens in the first:\n\n\"the canyon\"\n\n ↓\n\n\"canyon\" from earlier\n\nand:\n\n\"the river\"\n\n ↓\n\n\"river\" from earlier\n\nThe model doesn’t need the words to be right next to each other.\n\nAttention gives it a mechanism for making those connections.\n\nThis is a special type of self-attention, absolutely critical for autoregressive LLMs.\n\nImagine you’re writing:\n\n***The cat sat on the…***\n\nand the model is trying to predict the next token.\n\nIt would be cheating if the model could peek ahead and see:\n\n***mat***\n\nbefore predicting it.\n\nSo causal attention imposes a rule:\n\n**You can look backward, but you cannot look into the future.**\n\n```\nTheThe catThe cat satThe cat sat onThe cat sat on the\n```\n\nWhen predicting the next token, each position can only attend to itself and the tokens before it. Technically, a mask is applied to future positions.\n\nThose future attention scores are set to -∞ before softmax.\n\nAnd because softmax(-∞) ≈ 0, the model effectively gets zero attention from future tokens.\n\nHere, the query comes from one sequence while the keys and values come from another. Imagine a decoder generating a sentence while looking at information produced by an encoder. In some multimodal architectures, text features can similarly serve as queries over image features.\n\nSo instead of:\n\n*“I’m looking around inside my own sequence.”*\n\nit’s more like:\n\n*“I’m looking at information coming from somewhere else.”*\n\nThis is one of the ideas that allows attention to move beyond text.\n\nBefore turning to how multiple heads share Q, K, and V, let’s first understand the vectors themselves, and the weight matrices that produce them.\n\n**The vectors: Q, K, and V**\n\nReturning to the crowded-room analogy from earlier: each conversation in the room broadcasts something identifiable like a topic, a name, a tone, etc., which corresponds to the **key**: a signal a token makes available for others to measure against. A listener holds a sense of what they are currently listening for. This is a **query: a** ttention shifts toward whichever key matches this query most closely. What ultimately gets absorbed once attention lands, however, is not the key itself but the actual content of that conversation, the **value**. In a transformer, every input token produces all three vectors: a query representing what it is looking for in the rest of the sequence, a key representing what it offers to be matched against, and a value representing the content it contributes once another token’s query matches its key.\n\nThese vectors are not stored anywhere between forward passes. Each one is computed fresh, every time, as a linear projection of the token’s current embedding:\n\n```\nQ = X · WqK = X · WkV = X · Wv\n```\n\n**The weights: Wq, Wk, and Wv**\n\nThese matrices are initialized before training according to the model’s initialization scheme, typically with values chosen to keep activations and gradients well behaved.\n\nTraining proceeds by running the model forward on a batch of text, comparing its predicted next tokens against the actual next tokens, and computing a loss from the difference. That loss is what eventually drives updates to every learnable parameter in the network, including Wq, Wk, and Wv. Once training ends, Wq, Wk, and Wv are frozen: training optimizers stop updating them, and they become fixed constants baked into the deployed model. The vectors are still computed fresh on every forward pass at inference time (that computation never stops) but it is now a matrix multiplication against fixed weights rather than a step in a learning process.\n\n**Single-head attention** produces one query, one key, and one value vector for each token. For an embedding of size 512, that's ultimately one attention output per token. The token has one \"opinion\" about what mattered around it, shaped by whatever single notion of relevance those weights happen to encode.\n\n**Multi-Head Attention (MHA)** runs several smaller attention operations per token in parallel, each projecting the full input representation into a lower-dimensional subspace. That same 512-dimensional embedding might get split into 8 heads of 64 each. Each head has its own learned Q/K/V projection parameters, allowing different heads to learn different patterns of interaction. For example, some may become sensitive to syntactic or positional relationships. Each head produces its own small attention output, and those outputs are concatenated back into one full-size vector, then passed through one more learned matrix to mix them together.\n\nEach attention operation (or head) gets its own learned projections and performs attention independently.\n\n```\nInput → Q/K/V ──────┼── Head 1                    ├── Head 2                    └── Head 3\n```\n\nThe outputs from all heads are concatenated and passed through another learned projection.\n\nEvery head maintains its own K and V and during generation, those K/V tensors have to be stored. That memory usage becomes expensive.\n\nEnter MQA.\n\n**Multi-query attention** keeps separate Q projections for every head, but makes all the heads share one K and V.\n\nInstead of:\n\n```\nHead 1 → K₁ V₁Head 2 → K₂ V₂Head 3 → K₃ V₃Head 4 → K₄ V₄\n```\n\nNow:\n\n```\nHead 1 ─┐Head 2 ─┤Head 3 ─┼── Shared K/VHead 4 ─┘\n```\n\nThat’s a huge memory saving and less memory pressure can mean faster generation. But there is a trade-off.\n\nForcing different heads to share the same key/value representation saves memory, but some of the independence that full multi-head attention provides is lost.\n\nAnd this is where **Grouped-query attention (GQA)** comes in. Instead of making **every head** share one K/V pair or giving **every head** its own K/V pair, heads are separated into groups.\n\n```\nHead 1 ─┐Head 2 ─┘ → K/V Group 1\nHead 3 ─┐Head 4 ─┘ → K/V Group 2\n```\n\nNow we’ve got a middle ground. More flexibility than MQA. Less memory than MHA.\n\nThis part covered how attention decides what to look at: how relationships between tokens get scored (additive, dot-product, scaled dot-product), how far a token is allowed to look (global vs. local, soft vs. hard), which direction information flows (self-attention, causal masking, cross-attention), and how Q, K, and V get shared or split across attention heads (single-head, MHA, MQA, GQA).\n\nPart 2 turns to the problems that show up once this mechanism runs at scale: how a model knows where a word sits in a sequence (RoPE), how repeated computation is avoided during generation (the KV cache), how the underlying matrix math is made efficient on real hardware (FlashAttention), and how attention extends beyond text into images and audio (cross-modal attention).\n\n[AI Fundamentals: Attention Mechanisms in Transformers (Part 1)](https://pub.towardsai.net/ai-fundamentals-attention-mechanisms-in-transformers-part-1-a91cce62fbab) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/ai-fundamentals-attention-mechanisms-in-transformers-part-1", "canonical_source": "https://pub.towardsai.net/ai-fundamentals-attention-mechanisms-in-transformers-part-1-a91cce62fbab?source=rss----98111c9905da---4", "published_at": "2026-09-08 23:31:01+00:00", "updated_at": "2026-09-08 23:46:55.865805+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks"], "entities": ["Bahdanau", "Attention Is All You Need"], "alternates": {"html": "https://wpnews.pro/news/ai-fundamentals-attention-mechanisms-in-transformers-part-1", "markdown": "https://wpnews.pro/news/ai-fundamentals-attention-mechanisms-in-transformers-part-1.md", "text": "https://wpnews.pro/news/ai-fundamentals-attention-mechanisms-in-transformers-part-1.txt", "jsonld": "https://wpnews.pro/news/ai-fundamentals-attention-mechanisms-in-transformers-part-1.jsonld"}}