{"slug": "what-happens-inside-an-llm", "title": "What Happens Inside an LLM", "summary": "Manas Pathak's primer 'What Happens Inside an LLM' explains the internal workings of a large language model during a forward pass, detailing how tokens are converted into vectors, processed through transformer blocks, and used to predict the next token. The article highlights that generating a token does not require rereading the entire conversation due to cached key-value pairs, while attention's computational cost grows quadratically with prompt length, a factor mitigated in practice by methods like FlashAttention.", "body_md": "# What actually happens inside an LLM\n\n*A reference companion for “The Inference Wall”. What a large language model actually computes when\nit reads your prompt and writes an answer: how a token becomes a vector, what a transformer block\ndoes to it, and how the next token falls out the end. No benchmarks, no equations, no notation.*\n\n*Manas Pathak · September 8, 2026*\n\n[The Inference Wall](/inference-wall/) · [All posts](/inference-wall/articles/) · **Primer 2**\n\nAsk a language model a question and it answers one token at a time, each token produced by\nrunning the whole model once. That single run is a **forward pass**, and this post is about what\nhappens inside it: what the model is doing between the moment your prompt goes in and the moment\nthe first response token is produced.\n\nA note on scope before we start. This is a picture, not a specification. It deliberately skips the notation, the linear algebra, and a good deal of the engineering, and in a few places it says “this is roughly what happens” where the implementation is more involved. The goal is to give you a working sense of what a model does under the hood, solid enough to reason with; where you want the exact version, the papers and surveys linked along the way have it.\n\nTwo behaviors in particular are hard to explain from the outside, and this post accounts for both by the end:\n\n- **Why generating a token does not require rereading the whole conversation.** The model computes\na key and a value for each token once and reuses them forever after.\n- **Why reading a long prompt gets disproportionately expensive.** One specific step, attention,\ndoes an amount of work that grows with the*square* of the prompt’s length, while the rest of the\nmodel grows only in a straight line with it. That square-law is a property of the plain algorithm\nwe walk through here, and it is the right way to understand*why* attention is the expensive part.\nIn practice, production inference systems do not run attention exactly this way: methods like\nFlashAttention compute the same result without ever building the full square grid in memory, so\ntheir memory cost stays linear even though the arithmetic is still quadratic. We come back to that\nat the end.\n\nBoth behaviors fall out directly from how the model is built, and both are visible by the last section.\n\nThis is a self-contained tour of the model itself. If you also want the serving side, how a GPU\nspends its time and memory turning these forward passes into a live service, the\n[first primer](/inference-wall/articles/primer/) covers that and\ntreats the forward pass as a black box this post opens.\n\n## The shape of one forward pass\n\n*The forward pass as an assembly line: a word goes in as a rough stone, is faceted and polished bench by bench until the last worker can read it, and the next word comes off the end.*\n\nStart from the top, before any detail. The unit the model works in is the **token**: a short chunk\nof text, often a whole word but sometimes a word-piece or a punctuation mark. The model never sees\nletters or words as such; it sees a sequence of tokens, and it produces one token at a time. To\nproduce the next one, the model does three things in order:\n\n1. **Turn each input token into a vector.** From here on the model works only on these vectors,\nnever on the text directly.\n2. **Push the vectors through a stack of identical blocks.** Each block reads the current vectors\nand rewrites them, adding a little more information about what each token means*in the context\nof the others* and what is likely to come next. Nearly all of the model’s weights live in these\nblocks, and nearly all of the computation happens here.\n3. **Turn the last vector into a guess at the next token.** After the final block, the vector\nsitting at the most recent position is converted into a score for every possible next token,\nand the highest-scoring candidates are the model’s prediction.\n\nThat is the entire forward pass. A model runs it in two modes: **prefill** runs all three steps\nover every prompt token at once, and **decode** runs them over a single new token at a time while\nreusing stored work for the rest. Everything below is a\nzoom into step 2, because that is the step with the structure worth understanding. We build up one\nblock, then stack it.\n\n*The whole forward pass: embed each token to a vector, push all vectors through a stack of identical blocks, and turn only the last vector into a probability over the next token.*\n\n## From tokens to vectors: the embedding\n\nThe model keeps a large matrix with one row for every token in its vocabulary, learned during\ntraining; the row’s width is the vector size the model works in. That row is the token’s\n**embedding**, and it stands in for the token. Turning the prompt into vectors is a lookup by row\nindex, one embedding fetched per token, with nothing computed yet.\n\nOne property of that vector matters for everything that follows: its size never changes as it\nmoves through the model. It enters the first block, leaves the same size, enters the next block the\nsame size, and comes out of the last block still the same size. A block never grows or shrinks the\nvector; it only **rewrites the numbers in place**, adjusting them so they carry a bit more meaning.\nSo picture one vector per token, handed from block to block, revised at each step and passed along.\nThat handoff is the thread the rest of this post follows.\n\n## What a transformer block does\n\nRight after the lookup, a token’s vector depends only on the token itself, not on where it sits or what surrounds it. Consider the token “bank”: at this point its vector is the same whether the sentence is “the river bank” or “the savings bank,” because the lookup has no way to see the rest of the sentence. But predicting the next token requires knowing which meaning is in play, and that is fixed entirely by the surrounding tokens. So a block’s job is to update each token’s vector using information from the other tokens, and it does this in two distinct operations run back to back:\n\n1. **Attention** is the only operation in the whole model that moves information*between* token\npositions. It replaces each token’s vector with a mixture that draws in information from the\nearlier tokens relevant to it. Everywhere else, positions are processed in isolation; attention\nis where they interact.\n2. A **feed-forward network** then processes each token’s vector*on its own* , with no reference to\nany other position: the same two-layer network applied independently at every position. This is\nwhere most of the model’s weights sit, and where the model does the bulk of its per-token\ncomputation on the context attention just gathered.\n\nThe order is deliberate. Attention first collects the relevant context into each token’s vector;\nthe feed-forward network then transforms that now-contextual vector. A block is exactly this pair,\n**attention then feed-forward**, and it is the unit that repeats. The next two sections take each\noperation in turn.\n\n*Every transformer block does the same two things in order: attention, where tokens exchange information, then feed-forward, where each token is processed on its own.*\n\n## Attention: a weighted average each token computes for itself\n\nHere is the whole operation in one sentence, then the parts. **Attention rewrites each token’s\nvector as a weighted average of vectors drawn from the earlier tokens, where each token decides\nfor itself how much weight to put on each of the others.** The only real question is where those\nweights come from, and that is what the query, key, and value are for.\n\nAn analogy first, because the three-way split is the part that tripped me up when I first learned\nthis. Think of searching for a book in a library. You walk in with a **query**, a description of\nwhat you are after. Every book on the shelf has a **key** printed on its spine, a short description\nof what it is about, written in the same vocabulary as your query so the two can be compared. You\nmatch your query against every spine, and the books whose keys fit best are the ones you pull down.\nWhat you actually read and take away is not the spine label but the book’s contents, its **value**.\nQuery is what I’m looking for, key is what I advertise, value is what I hand over. The rest of this\nsection is that same idea made mechanical, so take the three names on faith for a moment; the way\nthey fit together will be concrete by the end of the section.\n\n*Attention as a library search: you match your query against every spine, pull the books whose keys fit best, and blend their contents (the values) into one new page.*\n\nNow the mechanism. From each token’s current vector, the block computes three new vectors by multiplying it against three separate learned weight matrices. “Computes a description” here just means it produces another vector, one whose role is set by which matrix made it:\n\n- the **query** : what this token is looking for in the tokens before it,\n- the **key** : what this token offers, written so that it can be compared against a query,\n- the **value** : the information this token will contribute to any token that attends to it.\n\nTo update token number 50, the block takes token 50’s **query** and compares it against the\n**key** of every token from 1 to 50, itself included. The comparison is a dot product, which is large when two\nvectors point in similar directions and small when they do not, so it measures how well token 50’s\nquery lines up with each of those keys. That produces one raw score per token in the range: how\nrelevant is that token to what token 50 is looking for.\n\nThose raw scores are not yet usable as averaging weights: some are negative, and they do not add up\nto anything in particular. **Softmax** is the step that turns them into weights, converting the row\nof raw scores into a set of proportions that are all positive and sum to one. The row now reads as\nsomething like “70% of my attention on this token, 20% on that one, the rest spread thin.” It also\nsharpens the contrast, so a clearly-best match dominates the blend while weak matches contribute\nalmost nothing. Whenever you see softmax in this post, read it as “turn a list of scores into a\nlist of probabilities.”\n\nThe last step is the average itself. The block takes each earlier token’s **value** vector, scales\nit by that token’s softmax weight, and adds them all up. The result is one blended vector, made\nmostly of the values of the tokens token 50 found most relevant, and that blend is written back as\ntoken 50’s updated vector. This is the operation the\n[transformer paper](https://arxiv.org/abs/1706.03762) introduced, and its title is the claim\nitself, that attention is enough to let tokens share information.\n\nTo restate the whole thing in one line: each token uses its **query** to score every earlier\ntoken’s **key**, softmax turns those scores into weights, and the token’s new vector is the\nweighted blend of the earlier tokens’ **values**. That is attention, start to finish.\n\n### Why this is exactly what the KV cache stores\n\nLook at what updating a token needs from the past: the **keys** of the earlier tokens, to score\nthem, and their **values**, to average them. It never needs their queries. A token’s query is used\nonly to update that same token, never to be looked at from elsewhere. And the key and value a token\nproduces at a given block do not change once computed: token 50’s key at block 3 is the same\nwhether the sequence is 51 tokens long or 5,000.\n\nSo the model computes each token’s key and value once, the first time it processes that token, and\nkeeps them. That store is the **KV cache**. When the model later generates token 5,000 it does not\nrerun tokens 1 through 4,999; it processes only the new token, forming that token’s query, key,\nand value, then scores its query against the 4,999 keys already saved plus its own. The new\ntoken’s key and value are added to the cache for the tokens that come after it, and its query is\nused once and discarded. Compute each token’s key and value once and keep them, never recompute\nthe past: the cache is not a bolt-on optimization, it falls straight out of how attention is\ndefined.\n\n### Many comparisons in parallel: attention heads\n\nUp to this point we have used a single query-key-value comparison for illustration, but in practice\na block does not run just one. It runs several in parallel, each with its own query, key, and value\nmatrices, called attention **heads**. One head might learn to track\nthe immediately preceding token, another the last time the subject was mentioned, another the\nmatching bracket or quotation mark. Each head does its own scoring and averaging over the whole sequence,\nthe results are combined, and the vector moves on. The reason it matters for cost: the KV cache\nholds a separate key and value for *every head of every block*, which is why cached tokens add up\nin memory as quickly as they do.\n\nOne constraint has been quietly doing work this whole time: when updating token 50, the block\nscores it against tokens 1 through 50, never against tokens that come after. A token is only ever\nallowed to look backward, because at generation time the later tokens do not exist yet. This is\n**masked** (or **causal**) attention, and the mask is exactly what enforces “earlier tokens only”\nin every score above.\n\n## The feed-forward network: computing on what attention gathered\n\nAfter attention, each token’s vector carries information about its context. The second operation\nin the block is a **feed-forward network**, also called the **MLP** (multi-layer perceptron): two\nlarge weight matrices with a simple nonlinear function between them, applied to each token’s vector\nindependently. It takes the contextual vector attention produced and transforms it, position by\nposition, with no further mixing between positions.\n\nIts mechanics are simpler than attention’s, but do not read that as unimportant: in a typical block\nthe feed-forward network holds about two-thirds of the weights to attention’s one-third, so on a\nreal GPU it is where most of the work of a forward pass ends up, and it is where a lot of what the\nmodel *knows* is stored. A rough way to hold the division of labor: attention is where tokens work\nout how they relate to each other, and the feed-forward network is where the model does its thinking\nabout what the result means. The one distinction to keep is that this operation is strictly\nper-token, where attention was strictly about tokens interacting.\n\n## Stacking blocks\n\nA model is a stack of these blocks, one after another. In the basic architecture there is no variety in the wiring: every block is built identically, attention then feed-forward. State-of-the-art models vary this in places (a few of the common variations come up in the last section), but the uniform stack is the right default to picture. Small models stack a dozen or so; most current LLMs run somewhere between about 32 and 120 of them, with the largest open models at the top of that range. What differs between blocks is the learned weights, and so what each block does to the vector. One way to imagine the division of labor is that early blocks resolve local, grammatical structure while later blocks assemble longer-range meaning, though this is a loose intuition rather than a sharp rule.\n\nEach block reads the vectors the previous block wrote and edits them a little further. One detail\nkeeps the repetition from washing out: a block *adds* its result into the vector rather than\nreplacing it, so what an early block established survives to the end unless a later block\ndeliberately overwrites it. That is what lets dozens of rounds of editing accumulate into a\nsharper and sharper representation instead of blurring into noise.\n\n## Turning the last vector into the next token\n\nAfter the final block, the vector at the most recent position holds a heavily revised\nrepresentation of that token in its full context. Turning it into an actual next token is what the\nphrase **next-token prediction** names, and it is a smaller step than it sounds.\n\nThe model multiplies that final vector by one last large matrix, which produces a single number\nfor *every* token in the vocabulary: a score for how well each one fits as the continuation. A\nsoftmax turns those scores into a probability for each candidate, the same trick as before, used\nhere to turn scores into a probability distribution rather than averaging weights. That\ndistribution is the model’s entire output: not a word, but a probability spread across the whole\nvocabulary. “The model predicts the next token” means exactly this, a ranked list of candidates\nwith probabilities attached.\n\nPicking an actual token from that distribution is a separate step called **sampling**, cheap\ncompared to the forward pass that produced the distribution, and it is where knobs like\n*temperature* live: take the single most probable token, or roll a weighted\ndie over the top few. The token that comes out is fed back in as the newest input, and the whole\npass runs again for the token after it. That feedback loop, one full forward pass per output\ntoken, is the decode loop, now with its insides visible.\n\n## The two phases, from the inside\n\nWith the block open, the difference between reading a prompt and generating an answer comes down to one thing, and it is all about attention:\n\n- **Prefill** reads the whole prompt at once, so every one of the N prompt tokens forms a query and\nscores it against the keys of all the others in the same pass. That is N queries against up to N\nkeys: an N-by-N grid of scores. Double the prompt and that grid quadruples. This is the one part\nof the model whose cost grows with the*square* of the sequence length.\n- **Decode** generates one token at a time, so each step forms exactly*one* query and scores it\nagainst all the keys stored so far. One row, not a grid. Decode’s attention cost grows only in a\nstraight line with how deep the sequence is, but it pays that cost again on every single token it\nemits.\n\nEverything else scales gently. Forming each token’s query, key, and value, running the feed-forward network, and the final scoring are all per-token work: their cost tracks the number of tokens, not its square. It is only attention’s scoring-and-averaging step, where each token looks at every earlier one, whose cost depends on how far back it has to look: quadratic while reading a prompt, linear while writing an answer. That is why reading a long prompt gets disproportionately expensive, the second thing this post set out to explain.\n\n## This is the basic version; real models add to it\n\n*The plain transformer is the frame; real models keep it and bolt on parts. This post is about the frame.*\n\nEverything above is the plain transformer, and it is the right skeleton to carry in your head. Most production LLMs, though, differ from it in the details. Real models keep the skeleton, attention then feed-forward, repeated, and add refinements at nearly every step. A few of the common ones, so the names are not a surprise when you meet them:\n\n- **Position information.** The attention described above has no notion of token order; scoring a\nquery against a key does not care which came first. Real models inject order separately, most\noften with rotary position encodings, so the model knows token 3 from token 300.\n- **Cheaper keys and values.** Grouped-query attention lets several heads share one set of keys and\nvalues, which shrinks the KV cache substantially with little quality loss, and is why modern\nlong-context models are practical to serve at all.\n- **More feed-forward, used selectively.** Mixture-of-experts replaces the single feed-forward\nnetwork with many, and routes each token to just a few of them, so the model can hold far more\nweights than any one token pays to use.\n- **Cheaper mixing in some blocks.** Some architectures replace full attention in a fraction of\ntheir blocks with a mixing step whose cost stays fixed as the sequence grows, trading a little of\nattention’s reach for an escape from its quadratic cost. These hybrid and linear-attention\ndesigns are a bet that long context is worth restructuring for.\n- **Normalization.** Small normalization steps sit around each operation to keep the numbers\nwell-behaved so the model trains stably.\n\nOne more addition is not a change to the model at all, but to how attention is computed, and it is\nthe one promised back in the intro. The N-by-N grid of scores from the prefill section is the\nplain algorithm’s way of doing attention; it is also a lot of memory to hold at once. **FlashAttention**\ncomputes the exact same weighted average without ever building that full grid in memory, streaming\nthrough the keys and values in tiles instead. The arithmetic is still quadratic, but the memory it\ntouches grows only in a straight line, which is what makes long prompts practical to serve. It is a\nfaithful shortcut, not an approximation: the numbers that come out are the same.\n\nThese refinements do not overturn the picture this post drew. The skeleton, attention then\nfeed-forward, repeated, embed at the front and score at the back, is still exactly what a real model\nruns. For the full catalog, a survey like [Zhao et al., 2023](https://arxiv.org/abs/2303.18223)\nwalks through them.\n\n## What you can now see\n\n- A forward pass is: embed each token into a vector, push the vectors through a stack of identical blocks that rewrite them, then turn the last vector into a probability over the next token.\n- A block runs two operations in order: **attention** , the only place tokens interact, and a**feed-forward network** , which processes each token on its own and holds most of the weights.\n- Attention rewrites a token’s vector as a **weighted average of value vectors** , with weights set\nby scoring the token’s**query** against every earlier token’s**key** and passing the scores\nthrough a softmax. The**KV cache** exists because a token’s key and value never change once\ncomputed, so they are stored and reused while the query is discarded.\n- The model’s output is a **probability over the whole vocabulary** , and picking a token from it is\na separate, cheap sampling step whose result is fed back to start the next pass.\n\nThat is the whole model: embed, a stack of blocks that mix across tokens and then compute per token, and a final scoring into the next token. Keep this picture for any time you need to reason about what an LLM is actually computing, rather than treating it as an oracle that turns prompts into text.\n\n## Glossary\n\nThe terms this post introduced, in one place:\n\n| Term | Meaning | \n|---|---|\n| **Token** | the unit the model reads and writes: a short chunk of text, often a whole word but sometimes a word-piece or punctuation. The model works in tokens, not letters. | \n| **Forward pass** | one full run of the model, start to finish, that produces one output token. | \n| **Embedding** | the vector a token is looked up as, before any block has touched it. | \n| **Vector** | the fixed-size list of numbers that stands in for a token and gets rewritten in place by each block; its size never changes as it moves through the model. | \n| **Transformer block** | the repeated unit, attention then feed-forward, that a model stacks; each block reads the current vectors and edits them a little further. | \n| **Attention** | the only operation that moves information between token positions; it rewrites each token’s vector as a weighted average of the earlier tokens’ values. | \n| **Query, key, value** | the three vectors each token produces: the query is what it is looking for, the key is what it advertises to be matched against, the value is what it contributes to a token that attends to it. | \n| **Softmax** | the step that turns a list of raw scores into a list of positive weights that sum to one; read it as “scores to probabilities.” | \n| **KV cache** | the store of every token’s key and value, kept because they never change once computed, so the model reuses them instead of recomputing the past. | \n| **Attention head** | one of several parallel query-key-value comparisons a block runs at once, each learning to track a different relationship. | \n| **Masked (causal) attention** | the rule that a token may only attend to earlier tokens, never to ones that come after it. | \n| **Feed-forward network (MLP)** | the second operation in a block: a per-token network, holding most of the model’s weights, that computes on the context attention gathered. | \n| **Prefill / decode** | the two modes of the forward pass: prefill reads the whole prompt at once (quadratic attention), decode generates one token at a time (one row of attention per step). | \n| **Next-token prediction / sampling** | turning the last vector into a probability over the whole vocabulary, then picking one token from that distribution to feed back in. | \n| **FlashAttention** | a way to compute attention’s exact result without ever building the full score grid in memory, keeping memory use linear though the arithmetic stays quadratic. | \n\n## About the author\n\nI am **Manas Pathak**. Questions, corrections, or a sharper way to picture any of this are all\nwelcome: email me at [mapathak@gmail.com](mailto:mapathak@gmail.com) or find me on\n[LinkedIn](https://www.linkedin.com/in/manas-pathak-806b002a/).\n\n**Read next:** for the serving side, how these forward passes become a live service on a GPU, see\n[the first primer: how an LLM actually serves a request](/inference-wall/articles/primer/) · [All posts](/inference-wall/articles/)\n\n*Disclaimer: This blog is written and published in my personal capacity. The opinions, findings,\nand conclusions expressed herein are solely my own and do not necessarily represent the views,\npolicies, or endorsements of my current or past employers.*", "url": "https://wpnews.pro/news/what-happens-inside-an-llm", "canonical_source": "https://mapathak-commits.github.io/inference-wall/articles/primer-2/", "published_at": "2026-09-08 18:26:46+00:00", "updated_at": "2026-09-08 18:49:09.937213+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["Manas Pathak", "FlashAttention"], "alternates": {"html": "https://wpnews.pro/news/what-happens-inside-an-llm", "markdown": "https://wpnews.pro/news/what-happens-inside-an-llm.md", "text": "https://wpnews.pro/news/what-happens-inside-an-llm.txt", "jsonld": "https://wpnews.pro/news/what-happens-inside-an-llm.jsonld"}}