{"slug": "a-plain-english-guide-to-transformer-architecture", "title": "A Plain-English Guide to Transformer Architecture", "summary": "A developer published a plain-English guide to transformer architecture, explaining how large language models like GPT-4, Claude, Gemini, and Llama process text from tokenization to autoregressive generation. The guide covers key concepts such as self-attention, multi-head attention, training via backpropagation, and inference economics, with interactive tools linked inline.", "body_md": "A from-scratch walkthrough of what happens between typing a prompt and getting a response — no ML background assumed.\n\nEvery large language model — GPT-4, Claude, Gemini, Llama — is built on the same core idea. This guide follows that idea in the exact order data actually flows through a real model: text in, tokens, vectors, 80 stacked layers, and a word out — then loops back to explain how the model got trained in the first place. Read it top to bottom; each section is built to need only what came before it.\n\n[The One Job an LLM Has](https://gist.github.com/starred.atom#1-the-one-job-an-llm-has)[Vectors — Turning Words Into Math](https://gist.github.com/starred.atom#2-vectors--turning-words-into-math)[The Full Pipeline at a Glance](https://gist.github.com/starred.atom#3-the-full-pipeline-at-a-glance)[Tokenization](https://gist.github.com/starred.atom#4-tokenization)[The Embedding Layer](https://gist.github.com/starred.atom#5-the-embedding-layer)[Positional Encoding — Teaching the Model Word Order](https://gist.github.com/starred.atom#6-positional-encoding--teaching-the-model-word-order)[Self-Attention — The Core Idea](https://gist.github.com/starred.atom#7-self-attention--the-core-idea)[Self-Attention — The Mechanics (Q, K, V)](https://gist.github.com/starred.atom#8-self-attention--the-mechanics-q-k-v)[Multi-Head Attention](https://gist.github.com/starred.atom#9-multi-head-attention)[Inside One Transformer Block — Causal Masking, Residuals, LayerNorm & the Feed-Forward Network](https://gist.github.com/starred.atom#10-inside-one-transformer-block--causal-masking-residuals-layernorm--the-feed-forward-network)[Why the Middle of a Long Prompt Gets Ignored](https://gist.github.com/starred.atom#11-why-the-middle-of-a-long-prompt-gets-ignored)[Autoregressive Generation — How Words Actually Get Produced](https://gist.github.com/starred.atom#12-autoregressive-generation--how-words-actually-get-produced)[The KV Cache — Why Generation Isn't Recomputed From Scratch](https://gist.github.com/starred.atom#13-the-kv-cache--why-generation-isnt-recomputed-from-scratch)[Temperature & Sampling — Controlling Randomness](https://gist.github.com/starred.atom#14-temperature--sampling--controlling-randomness)[How Training Actually Updates the Weights — Backpropagation & Gradient Descent](https://gist.github.com/starred.atom#15-how-training-actually-updates-the-weights--backpropagation--gradient-descent)[Why Hallucination Happens — Architecturally](https://gist.github.com/starred.atom#16-why-hallucination-happens--architecturally)[From Raw Model to Helpful Assistant (SFT & RLHF)](https://gist.github.com/starred.atom#17-from-raw-model-to-helpful-assistant-sft--rlhf)[Model Scale & Emergent Abilities](https://gist.github.com/starred.atom#18-model-scale--emergent-abilities)[Embeddings — The Geometry of Meaning](https://gist.github.com/starred.atom#19-embeddings--the-geometry-of-meaning)[Inference Economics — Why LLMs Cost What They Cost](https://gist.github.com/starred.atom#20-inference-economics--why-llms-cost-what-they-cost)[Glossary — Every Term in One Place](https://gist.github.com/starred.atom#21-glossary--every-term-in-one-place)\n\nThree free, interactive tools are linked inline below, right in the sections they visualize best, rather than bundled at the end — so you can pull one up the moment it's relevant instead of hunting for it later.\n\nIn one line:an LLM is a single next-word predictor, run over and over — nothing else is happening underneath the hood.\n\nStrip away the hype, and every LLM is trained to do exactly one thing:\n\nGiven some text, predict the most likely next word.\n\nThat's it. There's no search engine inside, no database of facts, no reasoning module bolted on. The entire architecture — every layer, every mechanism described below — exists purely to answer one question, over and over, one word at a time:\n\n\"Given what came before, what word comes next?\"\n\nEverything from writing an essay to solving a math problem to holding a conversation is this same mechanism, repeated thousands of times in a row. Every section below is either part of *how* that one prediction gets made, or part of *how the model got good at making it* — keep that map in mind as you go.\n\nIn one line:every word becomes a list of numbers, and similar words end up as similar-looking lists — that single trick is what everything else in this guide is built on top of.\n\nA **vector** is just a list of numbers. Nothing mystical about it.\n\n```\n\"cat\"  →  [0.2,  0.8, -0.1,  0.5,  0.3, ...]\n\"dog\"  →  [0.3,  0.7, -0.2,  0.4,  0.3, ...]\n\"car\"  →  [0.9, -0.1,  0.8, -0.3,  0.1, ...]\n```\n\nIn real models these lists have 512, 1024, or even 4096 numbers. Each number is called a **dimension**.\n\nThe key insight that makes all of this work: **meaning becomes geometry**. Words with similar meaning end up with similar number patterns — so \"cat\" and \"dog\" land close together in this number-space, while \"car\" lands far away. You can literally do arithmetic on meaning this way (more on that in [section 19](https://gist.github.com/starred.atom#19-embeddings--the-geometry-of-meaning)).\n\nFor now, just accept one fact: **every word gets converted into a vector before any computation happens.** The model never touches raw text — only numbers. The next section shows the full journey these vectors take from raw text to a finished word.\n\nIn one line:text becomes token IDs, token IDs become vectors, 80 stacked layers refine those vectors by having tokens talk to each other, and the last vector gets turned into a probability over every possible next word — then the whole thing repeats for the next word.\n\nWe'll use one running example throughout this guide:\n\n```\nInput:  \"The cat sat on the\"\nGoal:   predict → \"mat\"\n```\n\nHere's the entire journey from text to output, every step numbered — including the two loops running at the same time (layers stacked sequentially, heads running in parallel inside each layer) and the outer autoregressive loop that wraps around the whole thing. Every numbered step here gets its own dedicated section later in this guide.\n\n```\n╔════════════════════════════════════════════════════════════════════╗\n║        TRANSFORMER ARCHITECTURE — ONE FULL GENERATION STEP          ║\n╚════════════════════════════════════════════════════════════════════╝\n\nSTEP 1 · INPUT TEXT\n   \"The cat sat on the\"\n          │\n          ▼\nSTEP 2 · TOKENIZER            text → integer IDs\n   [464, 3797, 3332, 319, 262]\n          │\n          ▼\nSTEP 3 · EMBEDDING LOOKUP     each ID → vector (e.g. 4096 numbers)\n   5 tokens → 5 vectors\n          │\n          ▼\nSTEP 4 · + POSITIONAL ENCODING\n   position info added to each vector so the model knows token order\n          │\n          ▼\n┌──────────────────────────────────────────────────────────────────┐\n│ STEP 5 · TRANSFORMER LAYER STACK      ↻ × 80  (LAYERS, SEQUENTIAL)│\n│ layer 1's output → layer 2's input → ... → layer 80's output      │\n│                                                                    │\n│  ┌────────────────────────────────────────────────────────────┐  │\n│  │ ONE LAYER =                                                 │  │\n│  │                                                              │  │\n│  │  5a · MULTI-HEAD SELF-ATTENTION                              │  │\n│  │  ┌──────┐ ┌──────┐ ┌──────┐         ↻ × 32  (HEADS,          │  │\n│  │  │Head 1│ │Head 2│ │Head 3│  ...    PARALLEL, inside THIS    │  │\n│  │  └──────┘ └──────┘ └──────┘         one layer only)          │  │\n│  │   each head: Q,K,V → attention scores → weighted sum of V    │  │\n│  │   causal mask applied — no token attends to future tokens    │  │\n│  │              │ concatenate all 32 heads' outputs              │  │\n│  │              ▼                                                │  │\n│  │  5b · ADD & NORM        (residual connection + layer norm)   │  │\n│  │              │                                                │  │\n│  │              ▼                                                │  │\n│  │  5c · FEED-FORWARD NETWORK   (per-token transformation)       │  │\n│  │              │                                                │  │\n│  │              ▼                                                │  │\n│  │  5d · ADD & NORM        (residual connection + layer norm)   │  │\n│  └───────────────────────┬────────────────────────────────────┘  │\n│                           │ output feeds into next layer as input │\n│                           ▼                                       │\n│                 ↻ repeat steps 5a–5d, 80 times total               │\n└──────────────────────────────────────────────────────────────────┘\n          │\n          ▼\nSTEP 6 · OUTPUT LINEAR LAYER\n   last token's final vector × weight matrix\n   → 100,000 raw NUMBERS (logits — NOT percentages yet)\n          │\n          ▼\nSTEP 7 · TEMPERATURE SCALING\n   logit ÷ temperature → adjusted logits (still raw numbers)\n          │\n          ▼\nSTEP 8 · SOFTMAX\n   adjusted logits → 100,000 PERCENTAGES (probabilities, sum = 100%)\n          │\n          ▼\nSTEP 9 · SAMPLE\n   pick one token from the probability distribution → e.g. \"mat\"\n          │\n          ▼\nSTEP 10 · STOP CHECK\n   EOS token? max_tokens hit? stop sequence matched?\n     │ NO                                    │ YES\n     ▼                                       ▼\nSTEP 11 · APPEND TOKEN                  STOP — return final text\n   new token added to the sequence\n          │\n          └───────────────↻ LOOP BACK TO STEP 5\n                (sequence is now one token longer —\n                 a full forward pass runs again from step 5)\n```\n\n**Legend — what each loop means:**\n\n```\n↻ × 80   LAYERS     → sequential, stacked depth-wise\n                       layer 1 must finish before layer 2 starts\n                       this is \"how deep\" the network is\n\n↻ × 32   HEADS       → parallel, inside a single layer only\n                       all 32 run simultaneously, then merge\n                       this is \"how many relationship types\"\n                       one layer can detect at once\n\n↻        OUTER LOOP  → the autoregressive generation loop\n                       steps 5 through 11 repeat once per\n                       generated token, until stop\n```\n\n**One-line description per step:**\n\n```\n1   text arrives as-is, human readable\n2   text split into tokens, each mapped to an integer ID\n3   each ID looked up in embedding table → vector\n4   position information injected so order is not lost\n5   80 layers refine the representation, each layer's\n    32 heads simultaneously detecting different relationships\n6   final vector converted to one raw score per vocabulary word\n7   temperature reshapes those raw scores before normalizing\n8   raw scores converted into a real probability distribution\n9   one token drawn from that distribution\n10  check whether generation should stop here\n11  if not stopping, append the token and repeat from step 5\n```\n\nSee it move.Static diagrams only go so far.renders this[bbycroft.net/llm]exactpipeline as an animated, rotatable 3D model of a real, runnable GPT (nano-GPT, GPT-2, and others) — tokens flowing into embeddings, splitting into attention heads, merging back through Add & Norm, expanding through the feed-forward network, and collapsing down to logits at the output layer, all in one structure you can orbit around. It's the single best way toseethe shape of the steps above instead of just reading them. Worth pulling up now, before diving into the details, and again after finishing section 10 once every piece has a name.\n\nThe rest of this guide walks through each of these steps in detail — starting with tokenization (step 2).\n\nIn one line:text gets chopped into word-ish chunks and each chunk is swapped for a lookup number — step 2 of the pipeline above, and the very first thing that happens to your prompt.\n\nThe model doesn't read text — it reads numbers. A **tokenizer** splits text into chunks called **tokens** (roughly one word or word-piece) and maps each token to an integer ID from a fixed vocabulary.\n\n```\n\"The cat sat on the\"\n  ↓     ↓    ↓   ↓   ↓\n 464  3797  3332  319  262\n```\n\nThese IDs are just lookup indices — no meaning yet. Meaning arrives in the next step.\n\n**Vocabulary size** varies by model:\n\n```\nGPT-4   → ~100,000 tokens\nClaude  → ~100,000 tokens\nLlama 3 → ~128,000 tokens\n```\n\nTokens aren't always whole words — rare or long words get split into sub-word pieces:\n\n```\n\"cat\"           → 1 token   (common word, gets its own ID)\n\"sat\"           → 1 token\n\"Kathmandu\"     → 2 tokens  (less common, gets split)\n\"unprecedented\" → 3 tokens  (long word, split into sub-pieces)\n\"धन्यवाद\"        → 4 tokens  (non-Latin script, more splits)\n```\n\nThis is exactly why non-English text often costs more per word when you're billed by the token — tokenizers are trained mostly on English data, so other languages get sliced into more pieces.\n\nWith text turned into IDs, the next step is turning those IDs into the vectors introduced in section 2.\n\nIn one line:each token ID gets swapped for its learned vector — step 3 of the pipeline — and from this point on the model only ever works with numbers, never text again.\n\nEvery integer ID gets looked up in a giant table to retrieve its vector. This table is learned during training.\n\n```\n464  →  [0.12,  0.87, -0.34,  0.56,  0.01, ... ] ← \"The\"\n3797 →  [0.23,  0.91, -0.12,  0.44,  0.32, ... ] ← \"cat\"\n3332 →  [-0.45, 0.34,  0.78, -0.23,  0.67, ... ] ← \"sat\"\n319  →  [0.67, -0.12,  0.34,  0.89, -0.45, ... ] ← \"on\"\n262  →  [0.11,  0.88, -0.33,  0.54,  0.02, ... ] ← \"the\"\n```\n\nFrom this point forward, the model never sees the original text again — only these vectors. But these raw vectors are missing one thing: any sense of *where* each token sits in the sentence. That's step 4 of the pipeline, and the very next section.\n\nIn one line:a \"which position am I\" signal gets mixed into every token's vector right here, immediately after the embedding lookup — step 4 of the pipeline — because the comparison math used in the next few sections has no built-in concept of order on its own.\n\nSection 5 mentioned that positional information gets added right after the embedding lookup. This section explains why that step exists and how it works — before diving into self-attention itself in sections 7 through 9.\n\nThe next few sections explain self-attention — the mechanism that lets tokens exchange context by comparing their vectors mathematically. The one fact you need up front: **that comparison is pure arithmetic, and pure arithmetic has no built-in concept of order.**\n\n```\nSentence 1:  \"cat sat\"\nSentence 2:  \"sat cat\"\n\nWithout positional information, the vector for \"cat\" is identical\nin both sentences. The model can't tell which came first.\n```\n\nBut \"dog bites man\" and \"man bites dog\" contain the same words with entirely different meanings — the model needs position.\n\nBefore the first transformer block runs, a positional vector is added to each token's embedding:\n\n```\ntoken embedding     positional vector          final input vector\n\"cat\" [0.23, 0.91]  +  position 2 [0.10, 0.30]  =  [0.33, 1.21]\n\"sat\" [-0.45, 0.34] +  position 3 [0.20, 0.40]  =  [-0.25, 0.74]\n```\n\nNow the same word at different positions produces different final vectors — this is exactly the vector that gets handed to attention in the next section.\n\n**Sinusoidal** (the original 2017 Transformer paper) — position vectors computed from sine/cosine functions at different frequencies. Fixed, not learned. Doesn't generalize well beyond the training sequence length.\n\n**Learned positional embeddings** (GPT-2) — a lookup table learned during training, one vector per position, up to a maximum length.\n\n```\nPosition 1    →  [0.12, -0.34, 0.56, ...]\nPosition 2    →  [0.45,  0.23, -0.12, ...]\n...\nPosition 4096 →  [...]   ← hard limit, nothing beyond this\n```\n\n**RoPE — Rotary Position Embeddings** (Llama and most modern models) — instead of adding a position vector, RoPE *rotates* the Query and Key vectors (introduced in the next section) by an angle based on position, right before they get compared. Relative position gets baked directly into the attention score itself. RoPE generalizes far better beyond training length, which is why models can later be extended to much longer context windows than they were originally trained on:\n\n```\nOriginal training context: 4,096 tokens\nWith RoPE + fine-tuning:   can extend to 128k, 200k, 1M tokens\nWithout RoPE:              hard ceiling at training length\n```\n\nThis is the mechanical reason Claude can handle 200k tokens of context and Gemini 1.5 Pro handles 1M.\n\n```\nLearned embeddings:  position vectors only exist up to position N;\n                     anything beyond is undefined territory for the model\n\nRoPE:                angle rotation becomes extreme at very long distances;\n                     attention scores degrade past a point\n\nBoth approaches:     attention cost is O(N²) — quadratic.\n                     N = 10,000  →  100 million dot products\n                     N = 100,000 →  10 billion dot products\n```\n\nContext windows aren't just a pricing decision — they're a real architectural and hardware constraint, since doubling context length quadruples the attention computation. With position now baked into every vector, the model is finally ready for the main event: self-attention.\n\nIn one line:every token looks at every other token in the sentence and decides how much of its meaning to borrow from each one — the single mechanism that replaced older, sequential models and made modern LLMs possible.\n\nThis is the single most important idea in modern AI. Let's build the intuition before touching any math.\n\nConsider this sentence:\n\n\"The animal didn't cross the street because it was too tired.\"\n\nWhat does \"it\" refer to? The animal, not the street. You resolved that by connecting \"it\" back to \"animal\" across seven words. That connection between two distant parts of a sentence is called a **long-range dependency**.\n\nMore examples:\n\n```\n\"The trophy didn't fit in the suitcase because it was too big.\"\n                                              ↑\n                        what does \"it\" mean here?\n                        → \"trophy\" (far back in the sentence)\n\"The bank by the river where they used to fish before the factory\n was built in 1987 might close.\"\n  ↑\n \"bank\" = river bank or financial bank?\n → resolved by \"river\", 3 words away\n```\n\nHuman brains resolve these instantly. A model needs an explicit mechanism to do it — that mechanism is self-attention.\n\nBefore transformers, language models used **RNNs** (Recurrent Neural Networks), which process text one word at a time, carrying forward a single \"memory\" vector:\n\n```\ntoken 1 → [hidden state h1]\n                  ↓\ntoken 2 → [hidden state h2]  ← h2 is computed from h1 + token 2\n                  ↓\ntoken 3 → [hidden state h3]  ← h3 is computed from h2 + token 3\n                  ↓\n         ... and so on\n```\n\nThat hidden state is a single fixed-size vector, overwritten at every step. Imagine reading a 300-page book but you're only allowed one sticky note — every new sentence forces you to erase your note and rewrite a fresh summary. By page 300, almost nothing from page 1 survives.\n\n```\n\"The cat that had been sitting on the mat every morning\n since it was a kitten and had never left the house was hungry.\"\n\nRNN at \"was hungry\":\nhidden state ≈ [ ...stuff from recent tokens... nothing from \"cat\"... ]\n```\n\nBy token 501, information from token 1 has been diluted across 500 rewrites. The connection is essentially lost.\n\nSelf-attention abandons the \"carry a summary forward\" idea entirely. Instead, **every token looks directly at every other token, simultaneously**, and decides: \"how relevant is each other token to understanding me right now?\"\n\n```\n\"The cat sat on the\"\n  T1   T2   T3  T4  T5\n\nWhen processing T2 (\"cat\"):\n  - How much should I look at T1 (\"The\")?    → some\n  - How much should I look at T3 (\"sat\")?    → a lot (cat is doing the sitting)\n  - How much should I look at T4 (\"on\")?     → a little\n  - How much should I look at T5 (\"the\")?    → almost nothing\n\nResult: T2's vector gets updated with relevant information from all other tokens\n```\n\nThis is direct, parallel communication between any two tokens — regardless of distance. Token 1 and token 500 communicate with equal ease. No bottleneck, no dilution.\n\nThe exact mechanics of *how* a token decides \"how much to look at\" another token is the subject of the next section.\n\nIn one line:every token computes three vectors — a question (Query), an advertisement (Key), and a contribution (Value) — and attention is just \"compare every Query to every Key, then blend Values by how well they matched.\"\n\nSection 7 gave the intuition — every token looks at every other token. This section gives the actual mechanism behind that: how does a token decide *how much* to borrow from each other token?\n\n```\nSELF-ATTENTION INPUT:\n5 vectors, one per token (already carrying position info from section 6)\n\n\"The\"  → [0.12,  0.87, -0.34,  0.56]\n\"cat\"  → [0.23,  0.91, -0.12,  0.44]\n\"sat\"  → [-0.45, 0.34,  0.78, -0.23]\n\"on\"   → [0.67, -0.12,  0.34,  0.89]\n\"the\"  → [0.11,  0.88, -0.33,  0.54]\n\n(using 4 numbers per vector here for simplicity — real models use up to 4096)\n\nSELF-ATTENTION OUTPUT:\n5 vectors, one per token — same count, same size — but now\neach vector carries information borrowed from other tokens\n\n\"The\"  → [updated — now knows it precedes \"cat sat on the mat\"]\n\"cat\"  → [updated — now knows cat is the one doing the sitting]\n\"sat\"  → [updated — now knows who sat and where]\n\"on\"   → [updated — knows it connects action to location]\n\"the\"  → [updated — knows it precedes a surface/location]\n```\n\nSame number of vectors in and out, same dimensions — but richer content. The question is: **how does each token decide how much to borrow from each other token?** That's exactly what Q, K, V computes.\n\nQ, K, V come straight out of information retrieval. Think of a search engine:\n\n```\nYou type a search query:       \"best hiking trails Nepal\"\nEach webpage has metadata:     title, tags, description\nEach webpage has content:      the actual article text\n\nQUERY  = what you are searching for\nKEY    = the metadata compared against your query\nVALUE  = the actual content you retrieve if the match is good\n```\n\nThe search engine compares your Query against every Key, scores relevance, then returns a weighted mix of Values. Self-attention does exactly this — except **every token is simultaneously a searcher and a page being searched**:\n\n```\nEvery token asks a question (Query):   \"what context do I need?\"\nEvery token advertises itself (Key):   \"here is what I contain\"\nEvery token offers content (Value):    \"here is what I give if selected\"\n```\n\nThey aren't separate pieces of data — they're **three different transformations of the same token vector**, produced by multiplying it by three learned **weight matrices**.\n\nA weight matrix is a grid of numbers the model learns during training. Multiplying a vector by it produces a new vector viewed \"through a different lens.\" There are three such matrices —\n\nWq,Wk,Wv— one each for producing the Query, Key, and Value.\n\n```\nFor token \"cat\" with embedding vector E_cat:\n\nE_cat × Wq  =  Q_cat  (cat's question: \"what context do I need?\")\nE_cat × Wk  =  K_cat  (cat's advertisement: \"here is what I am about\")\nE_cat × Wv  =  V_cat  (cat's offering: \"here is what I contribute\")\n```\n\nThis happens for every token, giving us three sets of five vectors.\n\n**Step 1 — Score every token against every other token.**\n\nFocus on token \"cat.\" Its Query gets compared against every token's Key using a **dot product** — a single number measuring how aligned two vectors are (higher = more similar).\n\n```\nQ_cat = [1.0, 0.5]\n\nK_The = [0.2, 0.1]   →  dot product = (1.0×0.2) + (0.5×0.1) = 0.25\nK_cat = [0.9, 0.8]   →  dot product = (1.0×0.9) + (0.5×0.8) = 1.30\nK_sat = [0.8, 0.6]   →  dot product = (1.0×0.8) + (0.5×0.6) = 1.10\nK_on  = [0.1, 0.3]   →  dot product = (1.0×0.1) + (0.5×0.3) = 0.25\nK_the = [0.2, 0.2]   →  dot product = (1.0×0.2) + (0.5×0.2) = 0.30\n\nRaw attention scores for \"cat\":\nThe: 0.25,  cat: 1.30,  sat: 1.10,  on: 0.25,  the: 0.30\n```\n\nThis tracks intuition — \"cat\" is most relevant to itself, and second most relevant to \"sat\" (the action it performs).\n\n**Step 2 — Scale the scores.**\n\nRaw scores are divided by √d_k (the square root of the Key vector's dimension). With dimensions in the thousands, raw dot products can get huge, which destabilizes the next step. Dividing keeps numbers in a sane range — this is the \"scaled\" in \"scaled dot-product attention.\"\n\n**Step 3 — Softmax converts scores into probabilities.**\n\n**Softmax** takes a list of numbers and converts them into probabilities — all between 0 and 1, all summing to 1.\n\n```\nRaw scores (after scaling):  [0.25, 1.30, 1.10, 0.25, 0.30]\n\nAfter softmax:               [0.08, 0.42, 0.36, 0.08, 0.10]\n                              The   cat   sat   on    the\n\nReading this: \"cat\" puts 42% of its attention on itself,\n              36% on \"sat\", 10% on \"the\", 8% each on \"The\" and \"on\"\n```\n\n**Step 4 — Weighted sum of Values.**\n\nUse those probabilities to blend the Value vectors:\n\n```\nOutput for \"cat\" =\n  (0.08 × V_The)  +\n  (0.42 × V_cat)  +\n  (0.36 × V_sat)  +\n  (0.08 × V_on)   +\n  (0.10 × V_the)\n```\n\nThe result is a new vector for \"cat\" that says, in effect: *I am cat, mostly defined by what I am and what I do (sat), with minor contributions from surrounding context.*\n\nThis computation happens for every token, all at once:\n\n```\n              HOW MUCH DOES EACH ROW TOKEN ATTEND TO EACH COLUMN TOKEN?\n\n                    \"The\"  \"cat\"  \"sat\"  \"on\"  \"the\"\n             \"The\" [ 0.60   0.20   0.10   0.05   0.05 ]\n             \"cat\" [ 0.08   0.42   0.36   0.08   0.10 ]\n             \"sat\" [ 0.05   0.30   0.45   0.15   0.05 ]\n             \"on\"  [ 0.05   0.10   0.35   0.40   0.10 ]\n             \"the\" [ 0.05   0.10   0.25   0.35   0.25 ]\n\nEach row sums to 1.0 (softmax guarantees this).\nEach row is a different token's view of what matters.\n```\n\nSelf-attention never adds or removes tokens — 5 in, 5 out, always. What changes is the *content* of each vector:\n\n```\nBEFORE self-attention:\n\"cat\" vector = [0.23, 0.91, -0.12, 0.44]\n               only the meaning of \"cat\" in isolation\n\nAFTER self-attention:\n\"cat\" vector = [0.67, 1.23,  0.45, 0.89]\n               \"cat\" + borrowed context from \"sat\" —\n               the model now knows this cat is sitting, not sleeping\n```\n\n**Real-world analogy:** five people enter a meeting room, each carrying their own notes. They talk for an hour. Five people leave — same headcount, but each person's understanding has been updated by what they heard from the others. Nothing was added or removed from the room; what changed is what's now in each person's head.\n\nOne set of Q, K, V, though, can only pick up on one kind of relationship at a time — the next section explains why models run many of these side by side.\n\nIn one line:instead of computing attention once per layer, the model computes it 32 (or more) times in parallel, each copy free to specialize in a different kind of relationship — like several editors reading the same sentence, each looking for something different.\n\nOne set of Q, K, V can only learn *one type* of relationship at a time. But language has several relationship types happening simultaneously:\n\n```\n\"The cat sat on the mat\"\n\nRelationship type 1 — grammatical subject:  \"cat\" relates to \"sat\"\nRelationship type 2 — location:             \"sat\" relates to \"mat\"\nRelationship type 3 — article:              \"The\" relates to \"cat\"\nRelationship type 4 — preposition link:     \"on\" relates to \"mat\"\n```\n\nA single attention \"head\" tends to lock onto the most dominant relationship and miss the rest. **Multi-head attention** runs many independent Q/K/V computations in parallel — each with its own weight matrices, each free to specialize in a different kind of relationship.\n\n```\nHEAD 1:  Wq1, Wk1, Wv1  →  learns grammatical subject-verb relations\nHEAD 2:  Wq2, Wk2, Wv2  →  learns positional/location relations\nHEAD 3:  Wq3, Wk3, Wv3  →  learns coreference (what \"it\" refers to)\n...\nHEAD 32: Wq32, Wk32, Wv32 → learns some other pattern\n\nAll 32 outputs are concatenated and projected back to the original dimension.\n```\n\nGPT-3 uses 96 heads per layer; a typical 7-billion-parameter open model uses 32. Nobody assigns a head its job — specialization **emerges** from training. With 32+ heads per layer and 80 layers stacked, a model is running thousands of relationship detectors on every token, which is how it eventually learns to resolve something like the \"trophy/suitcase/it\" example from a handful of training examples that showed the same pattern.\n\n**Worked example — two heads disagreeing on purpose:**\n\n```\nSentence: \"The trophy didn't fit in the suitcase because it was too big.\"\n\nHEAD 3 (coreference specialist) scores \"it\" against:\n   \"trophy\"    → 0.71   ← wins\n   \"suitcase\"  → 0.19\n   \"big\"       → 0.05\n   (learned: \"it\" tends to bind to the earlier physical object\n    when the sentence structure implies containment)\n\nHEAD 7 (recency specialist) scores \"it\" against:\n   \"trophy\"    → 0.15\n   \"suitcase\"  → 0.62   ← wins (it's the nearest noun)\n   \"big\"       → 0.10\n\nFinal blended vector for \"it\" leans toward \"trophy\" because\nhead 3's signal is stronger for THIS sentence structure — but\nif you swapped \"big\" for \"small,\" head 3's own internal weighting\nwould flip toward \"suitcase\" instead. Multiple heads vote;\nthe training data decided which votes to trust in which context.\n```\n\nThis is also why models occasionally get coreference wrong on unusual sentence structures — if a relationship pattern was rare in training data, no head ever specialized in detecting it well.\n\nAttention (one head or many) is only step 5a of the transformer block from section 3's diagram. The next section covers the rest of that block — 5b through 5d — which is just as necessary for any of this to actually train.\n\nIn one line:each layer is attention (tokens talk to each other) followed by a feed-forward step (each token privately thinks alone), with a causal mask keeping generation honest and residual connections + layer norm making it possible to stack 80 of these without the network falling apart.\n\nSection 3's diagram labeled one layer as steps 5a through 5d: **attention → Add & Norm → feed-forward → Add & Norm**. Sections 7 through 9 only explained 5a. The other three pieces are just as load-bearing — without them, stacking 80 layers wouldn't work at all — so they get a full section here.\n\nSection 8 showed attention as every token looking at every other token. That's true for *understanding* text, but it would be cheating during *generation*. If token 3 could attend to token 7, the model would be predicting token 4 partly by looking at words that come after it — words that don't exist yet at generation time, and that the model is supposed to be predicting in the first place.\n\nThe fix is a **causal mask** (also called a look-ahead mask): before softmax runs, every attention score for a \"future\" token gets forced to negative infinity, so after softmax it becomes exactly 0%.\n\n```\nSentence: \"The cat sat on the\"\n             T1   T2  T3  T4  T5\n\nAttention scores for T3 (\"sat\"), BEFORE masking:\n  T1: 0.20   T2: 0.35   T3: 0.30   T4: 0.10   T5: 0.05\n                                    ↑           ↑\n                          these are FUTURE tokens relative to T3\n\nAttention scores for T3, AFTER causal mask applied:\n  T1: 0.20   T2: 0.35   T3: 0.30   T4: -inf    T5: -inf\n\nAfter softmax:\n  T1: 29%    T2: 51%    T3: 20%    T4: 0%      T5: 0%\n\nT3 can now only ever borrow from T1, T2, and itself — never T4 or T5.\n```\n\n**Real-world analogy:** reading a mystery novel with a bookmark that physically won't let you flip ahead. You can only use clues from pages you've already read — never a peek at chapter 12 while you're still on chapter 3 — which is exactly what keeps both training and generation honest.\n\nThis single masking rule is *why* the same architecture can be trained on an entire document at once (every token predicts the next token in parallel, each one only seeing what came before it) and then used at inference time to generate one token at a time — the constraint is identical in both cases. It's also why this style of model is called **decoder-only** or **causal**: contrast this with an encoder (like BERT), which is trained without a causal mask because it's meant to understand a whole passage at once, not generate text forward one token at a time.\n\n\"Add & Norm\" is actually two distinct ideas bundled into one step, and both exist to solve the same underlying problem: **very deep networks are hard to train.**\n\n**Residual connection (\"Add\")** — instead of a layer's output *replacing* its input, the original input gets added back on top of whatever the layer computed:\n\n```\nWITHOUT a residual connection:\n  layer_output = Attention(x)\n  → if the attention layer computes something slightly wrong or\n    unhelpful for a given token, that mistake fully overwrites\n    whatever useful information was already in x\n\nWITH a residual connection:\n  layer_output = x + Attention(x)\n  → even in the worst case, x itself always survives, untouched,\n    as a \"skip path\" running straight through the layer\n```\n\n**Real-world analogy:** imagine a report passing through 80 different editors, one after another. If each editor's markup fully replaced the previous draft, one careless editor on round 40 could wipe out everything good from rounds 1 through 39. A residual connection is like stapling the original page behind every editor's revision — each edit gets added as a note on top, but the original is always still there underneath, so nothing valuable is ever fully erased, only built upon.\n\nStack 80 of these and the effect compounds: every layer has a direct, unobstructed path back to the original embedding. Without this, a network that deep tends to struggle to learn at all — the training signal (the gradient computed from the loss function — see section 15) has to travel back through 80 sequential transformations during training, and it tends to shrink to nothing or blow up along the way. The residual path gives it a shortcut that doesn't decay, which is the main reason transformers can be stacked this deep in the first place.\n\n**Layer normalization (\"Norm\")** — after the addition, every token's vector gets rescaled so its values sit in a consistent, stable range (roughly mean 0, consistent spread), independent of whatever wild swings happened inside the layer:\n\n```\nBefore layer norm, a token's vector might drift to something like:\n  [45.2, -103.7, 8.9, 210.4, ...]   ← values growing unpredictably\n  after 40+ stacked layers, values like this cause training to become\n  unstable — small input changes cause huge output swings\n\nAfter layer norm, the same vector is rescaled to something like:\n  [0.31, -0.87, 0.12, 1.42, ...]    ← consistent, bounded range\n\nEvery layer downstream now always receives input in a predictable\nrange, regardless of what happened in the layers before it.\n```\n\n**Real-world analogy:** a sound engineer riding the volume fader before every speaker steps up to the mic at a conference. Without it, one overly enthusiastic speaker could blow out the levels for everyone after them; layer norm resets things to a safe, consistent range before every new sub-layer gets its turn.\n\nTogether, residuals and layer norm are why a model can be 80 layers deep instead of 4 or 5 — they're plumbing, not intelligence, but the intelligence doesn't work without them.\n\nAttention's whole job is letting tokens exchange information *with each other*. The feed-forward network's job is different: it processes **each token's vector independently**, with no communication between tokens at all, giving the model a place to actually \"think about\" what it just gathered from attention.\n\n**Real-world analogy:** attention is a group discussion where everyone talks and listens; the feed-forward step is what happens right after, when each person goes back to their own desk and privately thinks through what they just heard, updating their own notes with no more cross-talk. Attention is the meeting; the FFN is the reflection afterward.\n\nStructurally it's simple — two learned linear transformations with a nonlinear **activation function** in between, usually GELU:\n\n```\nFFN(x) = Linear_2( GELU( Linear_1(x) ) )\n\nLinear_1:  expands the vector — e.g. 4096 dimensions → 16,384 dimensions\nGELU:      a nonlinear \"squashing\" function applied elementwise\nLinear_2:  compresses back down — 16,384 dimensions → 4096 dimensions\n```\n\n**Why the expand-then-compress shape?** The wider middle layer gives the network more room to combine features in complex ways before compressing back to a size that fits into the next attention layer. **Why the nonlinearity?** Without it, stacking two linear layers back-to-back is mathematically equivalent to a single linear layer — the nonlinearity is what lets the network represent genuinely complex, non-straight-line relationships between inputs and outputs.\n\nOne number worth internalizing: **the FFN is where most of a model's parameters actually live** — typically around two-thirds of the total, versus roughly one-third in attention (the Wq/Wk/Wv/output matrices from section 8). Attention gets most of the explanation in tutorials like this one because it's the more novel, more interesting mechanism — but by raw parameter count, the feed-forward layers are the bulk of what you're running (and paying for) on every forward pass.\n\n```\ntoken vectors in\n       │\n       ▼\n┌─────────────────────────────┐\n│ 5a. Multi-head self-attention │  tokens exchange context\n│     (causal mask applied)     │  (sections 8–9, above)\n└─────────────────────────────┘\n       │\n       ▼\n   x = x + attention_output        ← 5b: residual \"Add\"\n   x = LayerNorm(x)                ←     \"Norm\"\n       │\n       ▼\n┌─────────────────────────────┐\n│ 5c. Feed-forward network      │  each token \"thinks\" independently\n└─────────────────────────────┘\n       │\n       ▼\n   x = x + ffn_output              ← 5d: residual \"Add\"\n   x = LayerNorm(x)                ←     \"Norm\"\n       │\n       ▼\ntoken vectors out → fed into the next layer (repeat 80 times)\n```\n\nSee it move.is the best match for this exact section — it visualizes a live GPT-2 block with attention, Add & Norm, and the FFN all animated side by side in the browser, running on text you type yourself. Pull it up here and watch a real vector's values change at each of the 5a–5d sub-steps above.[poloclub.github.io/transformer-explainer]\n\nWith one full layer understood, stacking 80 of them is literally just repeating this block 80 times, each one refining the tokens' vectors a little further. Two consequences of that repetition are worth calling out before moving on to what comes after the last layer.\n\nIn one line:softmax forces every token's attention to sum to 100%, so in a long prompt that 100% gets sliced so thin across thousands of competitors that anything sitting in the middle — with no positional signal marking it as special — tends to lose out to whatever sits at the very start or very end.\n\nA well-known failure mode: stuff a fact into the middle of a long prompt, and the model is far more likely to miss it than if that same fact were at the very start or very end. This is often called **\"lost in the middle.\"** It isn't a bug — it's a direct, mechanical consequence of two things covered in section 8: how attention scores are computed, and how softmax forces them to compete.\n\n**It's not that the middle is invisible.** Attention technically lets every token look at every other token. The real cause is **competition inside softmax**.\n\n```\nSHORT CONTEXT — 5 tokens:\nRaw scores:      [0.25, 1.30, 1.10, 0.25, 0.30]\nAfter softmax:   [0.08, 0.42, 0.36, 0.08, 0.10]\nSmallest slice is 0.08 — still meaningful.\n\nLONG CONTEXT — 10,000 tokens:\nSoftmax's 1.0 must now be sliced across 10,000 competitors.\nAverage slice ≈ 0.0001.\n```\n\nThe model has no advance knowledge of which middle token matters — it has to discover relevance purely through the dot-product comparison. Meanwhile:\n\n```\nBEGINNING tokens: system prompt lives here → trained to be high priority\nEND tokens:       the current user question → recency is a strong signal\nMIDDLE tokens:    no positional signal marking them as important —\n                  must win purely on content, diluted among thousands\n                  of competing tokens\n```\n\n**Why training makes this worse:** the model learned from internet text, books, and code, where structure almost always looks like:\n\n```\n[Important instruction / title / topic sentence]\n    ... body content ...\n[Conclusion / answer / punchline]\n```\n\nOver billions of examples, the weight matrices developed a bias: beginning and end positions produce stronger Key vectors, because that's statistically where important information lives in human-written text. Middle positions default to weaker Keys — not because they're unseen, but because the model learned they're usually supporting detail.\n\n**Concrete failure case:**\n\n```\n[System prompt — 200 tokens]\n[Retrieved document 1 — 800 tokens]\n[Retrieved document 2 — 800 tokens]    ← CRITICAL ANSWER IS HERE\n[Retrieved document 3 — 800 tokens]\n[Retrieved document 4 — 800 tokens]\n[User question — 50 tokens]\n\nThe critical answer sits around position 1,000-1,800 out of ~3,450 tokens,\nwith no positional signal marking it as special. Its Key vector has to\noutcompete thousands of others. The model often misses it, partially\nuses it, or under-weights it — even though it's technically \"visible.\"\n```\n\n**What this means practically:**\n\n```\nWRONG — burying critical instructions in the middle:\n[System prompt: general behavior rules]\n[Long background context]\n[Important constraint: never discuss pricing]   ← often missed\n[More background context]\n[User question]\n\nRIGHT — critical information at the boundaries:\n[System prompt: general behavior rules\n Important constraint: never discuss pricing]   ← at the top\n[Background context]\n[User question with key constraint repeated]    ← at the end\n```\n\nThis is exactly why production retrieval systems (RAG) rerank retrieved chunks and place the most relevant ones closest to the user's question, instead of just dumping everything in document order.\n\nWith all 80 layers now explained end to end, the next question is what happens the moment the *last* layer finishes — how a refined vector actually turns into a word.\n\nIn one line:the last token's final vector gets turned into a probability for every word in the vocabulary, one word gets picked, it's appended to the sequence, and the entire 80-layer pass runs again — this loop, one word at a time, is all \"generation\" ever is.\n\nEvery layer from sections 7 through 11 refines a token's vector using context from the rest of the sequence. This section covers what happens after the *last* one of those 80 layers — steps 6 through 11 of the pipeline diagram in section 3 — where a refined vector finally turns into an actual word.\n\n```\nTHE GENERATION LOOP:\n\nYou give:    \"The cat sat on the\"\n                        │\n                        ▼\n              [full forward pass]\n                        │\n                        ▼\n            model produces a probability\n            distribution over all words\n                        │\n                        ▼\n              one word is selected: \"mat\"\n                        │\n                        ▼\nsequence is now: \"The cat sat on the mat\"\n                        │\n                        ▼\n              [full forward pass again]\n                        │\n                        ▼\n              one word is selected: \".\"\n                        │\n                        ▼\n              [continues until stop condition]\n```\n\nThis loop — where the model's own output becomes its next input — is what **autoregressive** means. *Auto* = self. *Regressive* = predicting from previous values.\n\nAfter the final transformer block, only the **last token's** vector matters for generation — that's the one the next word gets predicted from.\n\n```\nInput:   \"The cat sat on the\"\n                              ↑\n                         last token: \"the\"\n\nAfter 80 transformer blocks:\n\"the\" vector = [0.45, -0.23, 0.87, ..., 0.12]   (4096 numbers)\n```\n\nThis vector passes through one final weight matrix (sized 4096 × 100,000) that maps it to 100,000 raw scores — one per vocabulary word. These raw, unnormalized scores are called **logits**:\n\n```\n\"mat\"       →   8.42   (high — strongly predicted)\n\"floor\"     →   6.31\n\"wall\"      →   5.10\n\"table\"     →   4.88\n\"ground\"    →   4.20\n...\n\"democracy\" →  -2.30   (very low — contextually absurd)\n```\n\nSoftmax converts these 100,000 logits into probabilities summing to 1.0:\n\n```\n\"mat\"       →  34.2%\n\"floor\"     →  18.1%\n\"wall\"      →   9.3%\n\"table\"     →   8.7%\n\"ground\"    →   6.5%\n\"democracy\" →   0.0%\n```\n\nThe model isn't asserting \"the answer is mat.\" It's saying: *given everything I've seen, here's my probability estimate for every possible next word.*\n\nA sorting algorithm always gives the same output for the same input. An LLM does not — a word is **sampled** from the distribution, like a weighted dice roll:\n\n```\n\"mat\"       ████████████████  34.2%\n\"floor\"     █████████         18.1%\n\"wall\"      ████               9.3%\n\"table\"     ████               8.7%\n\"ground\"    ███                6.5%\neverything  █████████████     23.2%  spread across 99,995 other tokens\nelse\n\nRun 1:  lands on \"mat\"\nRun 2:  lands on \"mat\"\nRun 3:  lands on \"floor\"\nRun 4:  lands on \"mat\"\nRun 5:  lands on \"wall\"\n```\n\nThis is why the *same* prompt can produce different outputs on different runs. It's deliberate: if the model always picked the single highest-probability word (**greedy decoding**), every run of \"write me a poem about mountains\" would produce the exact same, statistically-average poem. Sampling turns the model's uncertainty into a creative feature.\n\n**Greedy decoding** — always pick the highest-probability token. Deterministic, fast, but can get stuck in repetitive loops because the locally-best word at each step doesn't guarantee the best overall sentence.\n\n**Sampling** — pick probabilistically from the distribution at each step. Non-deterministic, more varied, more natural. In production this is almost always paired with **temperature** and **top-p** controls (see [section 14](https://gist.github.com/starred.atom#14-temperature--sampling--controlling-randomness)).\n\n```\nITERATION 1:\n├── forward pass through all 80 transformer blocks\n├── output layer produces 100,000 logits for the last token\n├── softmax → probability distribution\n├── sample → \"mat\"\n└── sequence becomes: \"The cat sat on the mat\"\n\nITERATION 2:\n├── forward pass (now \"mat\" attends to all previous tokens too)\n├── new logits → new distribution\n├── sample → \".\"\n└── sequence becomes: \"The cat sat on the mat.\"\n\nITERATION 3:\n├── forward pass\n├── logits show high probability for the end-of-sequence token\n├── sample → <EOS>\n└── generation stops\n```\n\nGeneration continues until one of three **stop conditions**: the model samples a special end-of-sequence token, you hit a `max_tokens`\n\nlimit, or a custom stop sequence appears in the output.\n\n**Why this matters practically:**\n\n**Non-determinism is a first-class concern.** You can't unit-test an LLM output with`assertEqual`\n\n. Testing strategy has to account for probabilistic output.**Streaming exists because of this loop.** Each token is available the instant it's sampled — the model doesn't wait to finish the whole response before sending anything.Every iteration is a full forward pass through every layer. Generating 1,000 tokens means 1,000 full forward passes; capping`max_tokens`\n\nis a real cost lever.`max_tokens`\n\nat 100 instead of 1,000 is a genuine 10x reduction in output-side compute.\n\nThis loop works, but it repeats a lot of work every single iteration — the next section shows the one optimization that makes it practical to run at all.\n\nIn one line:a past token's Key and Value vectors never change once computed, so instead of recomputing them on every loop iteration, the model stores them once and just adds the new token's — this is the single biggest efficiency trick in real-world LLM serving.\n\nGeneration is a loop (section 12). Without any optimization, every single iteration recomputes Query, Key, and Value vectors for **every token generated so far** — including ones already processed in earlier iterations:\n\n```\nITERATION 1 — generate \"mat\":\n  Input: [The, cat, sat, on, the]\n  Compute Q, K, V for ALL 5 tokens\n\nITERATION 2 — generate \".\":\n  Input: [The, cat, sat, on, the, mat]\n  Compute Q, K, V for ALL 6 tokens\n  → recomputes K, V for the first 5 — pure waste, they haven't changed\n\nITERATION 3 — generate \"The\":\n  Input: [The, cat, sat, on, the, mat, .]\n  Compute Q, K, V for ALL 7 tokens\n  → recomputes K, V for the first 6, again\n```\n\nA past token's Key and Value never change once computed — \"cat\" at position 2 always produces the same K and V no matter what gets generated afterward. Recomputing them every time is wasted work that compounds badly over long generations.\n\nThe **KV cache** stores every past token's K and V vectors (at every layer). Each new iteration only computes K, V, and Q for the *newest* token, and reuses everything else from cache.\n\nWhy cache K and V but not Q?\n\n```\nQ  →  \"what am I looking for?\"\n       Only the current token being generated asks a question.\n       Past tokens' questions were already used — no need to keep them.\n\nK  →  \"what do I contain?\" (advertised to everyone)\n       Every token, including past ones, must remain searchable\n       by future tokens' queries. Must be cached.\n\nV  →  \"what do I contribute when selected?\"\n       Every token must be able to contribute its content. Must be cached.\nITERATION 2 — generate \".\":\n  Compute Q, K, V for [\"mat\"] only  ← just the new token\n  Retrieve all past K, V from cache\n  Use \"mat\"'s Q against cached K's + its own K → generate \".\"\n  Store K_mat, V_mat into the cache for future iterations\n```\n\nEach iteration now only does the work of *one* new token — everything from the past is free.\n\n**Large context windows need enormous GPU memory**, because the cache has to store K and V for every token, at every layer, for every attention head:\n\n```\nCache size per token =\n  2 (K and V) × num_layers × num_heads × head_dimension × 2 bytes (float16)\n\nFor a 70B model: ~2.5 MB per token in KV cache\nFor a 128k-token context: 128,000 × 2.5 MB = 320 GB of cache alone\n```\n\nThis is a large part of why serving long-context requests requires multiple high-end GPUs just for the cache — not extra compute, extra memory.\n\n**Prefill vs. decode** are the two performance phases of generation:\n\n```\nPREFILL — your entire input prompt processed in one parallel pass.\n          Fast, benefits fully from GPU parallelism.\n          Determines time-to-first-token (TTFT).\n\nDECODE  — the autoregressive loop, one token per step, strictly sequential.\n          KV cache grows by one entry per step.\n          Slower — determines total generation time.\n```\n\nThis is why an API call often feels fast at first, then slows as the response grows: prefill finishes quickly, decode runs sequentially after.\n\n**Why output tokens cost more than input tokens:**\n\n```\nInput tokens  →  processed during prefill (parallel, fast, cheaper)\nOutput tokens →  generated during decode  (sequential, per-step, pricier)\n\nClaude 3.5 Sonnet input:  $3  per million tokens\nClaude 3.5 Sonnet output: $15 per million tokens   (5x more)\n```\n\n**Prompt caching** is a direct product of how the KV cache works: if the start of your prompt (e.g. a long system prompt) is identical across requests, the provider can store and reuse that portion's KV cache across calls, charging a much lower rate on a cache hit — often around 10% of the normal price for that segment.\n\nThe loop itself, and the cache that makes it efficient, are now fully covered. The one control still missing is *how random* each sampled word should be — that's temperature, next.\n\nIn one line:temperature is one division applied to the logits before softmax — divide by a number less than 1 to make the model more confident and repetitive, divide by a number greater than 1 to make it more random and varied.\n\nSection 12 introduced logits and softmax as steps 6 and 8 of the pipeline. Temperature (step 7) sits in between them, and it's the main dial you actually get to turn:\n\n```\nForward pass\n     │\n     ▼\n100,000 raw logits         ← pure model output, no control yet\n     │\n     ▼\nDIVIDE BY TEMPERATURE      ← this is where you intervene\n     │\n     ▼\n100,000 adjusted logits\n     │\n     ▼\nSOFTMAX                    ← converts to probabilities\n     │\n     ▼\nTOP-P FILTER               ← optional second control\n     │\n     ▼\nSAMPLE                     ← pick one token\n```\n\nTemperature `T`\n\nis a single number. Every logit gets divided by it before softmax:\n\n```\nadjusted_logit = original_logit / T\n```\n\nUsing five example tokens with original logits `mat=8.42, floor=6.31, wall=5.10, table=4.88, other=3.00`\n\n:\n\n**T = 1.0 (default — unchanged):**\n\n```\nAfter softmax:\n\"mat\"    72.1%   ████████████████████████████\n\"floor\"  16.4%   ██████\n\"wall\"    5.8%   ██\n\"table\"   4.7%   █\n\"other\"   1.0%   ▌\n```\n\n**T = 0.5 (low — sharpens the distribution):** dividing by a number less than 1 *amplifies* the differences between logits.\n\n```\nAfter softmax:\n\"mat\"    95.1%   ████████████████████████████████████████\n\"floor\"   3.9%   █\n\"wall\"    0.6%   ▌\n\"table\"   0.4%   ▌\n\"other\"   0.0%\n→ nearly certain, very predictable\n```\n\n**T = 2.0 (high — flattens the distribution):** dividing by a number greater than 1 *compresses* the differences.\n\n```\nAfter softmax:\n\"mat\"    41.2%   ████████████████\n\"floor\"  24.1%   █████████\n\"wall\"   16.2%   ██████\n\"table\"  14.8%   █████\n\"other\"   3.7%   █\n→ much more variety, less predictable\n```\n\n**T = 0 (special case — greedy):** dividing by zero is undefined, so in practice this is implemented as **argmax** — always pick the single highest logit. Fully deterministic; same input always gives same output.\n\n```\n                FLAT                              SHARP\n    ◄─────────────────────────────────────────────────►\n T = 2.0      T = 1.0      T = 0.5      T = 0.1    T = 0\n random      balanced     confident    near-certain deterministic\n creative    natural      focused      repetitive   always same\n```\n\nIt's worth being precise here, because it explains *why* top-p works at all. Softmax does **not** just divide by the count of tokens — it uses **exponentials**:\n\n```\nP(token_i) = exp(logit_i) / sum of exp(every logit)\n```\n\n`exp(x)`\n\nmeans *e* raised to the power *x* (e ≈ 2.718) — and exponential growth is explosive. Small logit differences become enormous probability differences:\n\n```\nStep 1 — raw logits:        Step 2 — apply exp():\n\"mat\"    8.42                exp(8.42)  =  4,526\n\"floor\"  6.31                exp(6.31)  =    550\n\"wall\"   5.10                exp(5.10)  =    164\n\"table\"  4.88                exp(4.88)  =    132\n\"other\"  3.00                exp(3.00)  =     20\n\nStep 3 — sum = 5,392\nStep 4 — divide each by the sum:\n\"mat\"    4,526 / 5,392  =  83.9%\n\"floor\"    550 / 5,392  =  10.2%\n\"wall\"     164 / 5,392  =   3.0%\n\"table\"    132 / 5,392  =   2.4%\n\"other\"     20 / 5,392  =   0.4%\n```\n\nA logit gap of just 5.42 (between \"mat\" and \"other\") becomes a 226x difference in probability after exponentiation. Across a real 100,000-token vocabulary, this means the top handful of tokens typically absorb 90%+ of the total probability mass before any filtering even happens — which is exactly why top-p, described next, works so well.\n\nEven with good temperature, the distribution has a long tail of thousands of tokens with tiny non-zero probabilities. **Top-p** cuts that tail off before sampling: sort tokens by probability, keep adding them to a \"nucleus\" until their cumulative probability reaches `p`\n\n, then sample only from that nucleus.\n\n```\np = 0.90:\nToken     Probability   Cumulative\n\"mat\"        34.2%        34.2%   ← include\n\"floor\"      18.1%        52.3%   ← include\n\"wall\"        9.3%        61.6%   ← include\n\"table\"       8.7%        70.3%   ← include\n\"ground\"      6.5%        76.8%   ← include\n\"roof\"        3.2%        80.0%   ← include\n\"rug\"         2.8%        82.8%   ← include\n\"bed\"         2.1%        84.9%   ← include\n\"couch\"       1.8%        86.7%   ← include\n\"chair\"       1.5%        88.2%   ← include\n\"desk\"        1.2%        89.4%   ← include\n\"box\"         0.9%        90.3%   ← include (crosses threshold here)\n─────────────────────────────────────────\neverything else (~99,988 tokens): EXCLUDED — cut off entirely\n```\n\n**Top-k** always keeps a *fixed* number of tokens, regardless of the shape of the distribution — which causes problems in both directions:\n\n```\nCONTEXT A — model is very certain (top-k=5):\n\"mat\" 94.0%, \"floor\" 3.0%, \"wall\" 1.5%, \"table\" 0.9%, \"ground\" 0.3%\n→ top-k=5 forces in that 0.3% token — basically noise\n\nCONTEXT B — model is genuinely uncertain (top-k=5):\n\"mat\" 12%, \"floor\" 11%, \"wall\" 10%, \"table\" 9%, \"ground\" 9%,\n\"roof\" 8% ← cut, \"rug\" 7% ← cut, \"bed\" 7% ← cut\n→ top-k=5 is cutting off perfectly valid options\n```\n\n**Top-p** adapts automatically to the shape of the distribution instead:\n\n```\nCONTEXT A (p=0.90): \"mat\" alone is already 94% → nucleus = 1 token\nCONTEXT B (p=0.90): takes 12 tokens to reach 90% → nucleus = 12 tokens\ntop-k  →  fixed count, blind to distribution shape\ntop-p  →  variable count, adapts to distribution shape\n```\n\nTop-p is the strictly better default in modern production systems. If a provider only exposes top-k, set it high (50+) to minimize the damage.\n\n```\nUSE CASE                        TEMPERATURE    TOP-P\n──────────────────────────────────────────────────────\nStructured data extraction      0              —\nClassification / routing        0              —\nFactual Q&A (one right answer)  0 – 0.2        —\nCode generation                 0.1 – 0.3      0.95\nSummarization                   0.3 – 0.5      0.95\nConversational assistant        0.7 – 1.0      0.95\nCreative writing                1.0 – 1.5      0.95\nBrainstorming / ideation        1.0 – 1.5      0.95\n──────────────────────────────────────────────────────\n```\n\nA common mistake: leaving default temperature (often 1.0) on a structured-extraction task, then getting inconsistent JSON and random formatting. Temperature 0 plus schema enforcement is the right combination when consistency matters. Also note: temperature 0 doesn't guarantee byte-identical output across *different providers* — floating point implementations vary slightly — so true determinism usually still needs schema validation on top.\n\nThat completes the inference side of this guide — everything a trained model does when you run it, from raw text in to a sampled word out. The next section switches focus entirely: how did those billions of weights get set in the first place?\n\nIn one line:the model makes a prediction, gets scored on how wrong it was, and every single one of its billions of numbers gets nudged a tiny bit in whichever direction would have made that prediction less wrong — repeated trillions of times, that's the entire mechanism behind everything a model \"knows.\"\n\nEvery section so far described what a *trained* model does when you run it. This section is the missing piece: how do billions of random numbers turn into weights that actually produce good predictions in the first place? The next section (hallucination) leans on this directly, so it's worth having the mechanism in hand first.\n\n```\nTRAINING LOOP (repeated over trillions of tokens):\n\n1. FORWARD PASS\n   training text → tokenize → embed → 80 transformer layers → logits → softmax\n   → a probability distribution over the next token\n   (exactly the same mechanics as inference — sections 4 through 12)\n\n2. COMPUTE LOSS\n   compare the predicted distribution against the token that\n   actually appeared next in the real training text\n   loss = how wrong / \"surprised\" the model was\n\n3. BACKWARD PASS (backpropagation)\n   compute the gradient of the loss with respect to EVERY parameter —\n   flowing backward from the output layer, through every transformer\n   block, all the way back to the embedding table\n   \"if this specific number were slightly bigger or smaller,\n    would the loss go down?\" — answered for billions of numbers at once\n\n4. UPDATE WEIGHTS (gradient descent)\n   nudge every parameter a small step in the direction that reduces loss\n   new_weight = old_weight − (learning_rate × gradient)\n\n   → repeat with the next training example, forever\n```\n\nEvery parameter in the model — every number inside Wq, Wk, Wv from section 8, every number inside the feed-forward matrices from section 10, every row of the embedding table from section 5 — starts as random noise and is shaped entirely by this four-step loop, repeated across trillions of tokens.\n\nThe standard loss for next-token prediction is **cross-entropy loss**, and it has a simple form: `loss = -log(probability the model assigned to the correct token)`\n\n. Using this guide's running example — predicting \"mat\" after \"The cat sat on the\":\n\n```\nIf the model assigned \"mat\" a probability of 34.2% (as in section 12):\n  loss = -log(0.342) ≈ 1.07\n\nIf the model had instead been well-calibrated and assigned 90%:\n  loss = -log(0.90) ≈ 0.105   ← much lower loss, closer to ideal\n\nIf the model had confidently assigned just 1% to the correct answer:\n  loss = -log(0.01) ≈ 4.6     ← punished heavily for being\n                                 confidently wrong\n```\n\nLow loss means the model put high probability on the token that actually came next. High loss means it didn't — and the size of the loss is exactly what step 3 needs in order to know how hard to push, and in which direction, on every single parameter.\n\n**Backpropagation** uses the chain rule from calculus to answer one question for every parameter in the model, all at once: *if I nudged this specific number, how much would the loss change?* That answer is called the parameter's **gradient**.\n\nThe computation runs in the opposite direction of the forward pass — starting from the loss at the output layer, and working backward through the output projection, back through transformer block 80, block 79, ... all the way to block 1, and finally back into the embedding table itself:\n\n```\nFORWARD:   embeddings → block 1 → block 2 → ... → block 80 → logits → loss\n                →              →                →        →\nBACKWARD:  embeddings ← block 1 ← block 2 ← ... ← block 80 ← logits ← loss\n                gradient flows this direction, layer by layer\n```\n\n**Real-world analogy:** hiking down a mountain in thick fog. You can't see the valley floor, but you can feel the slope of the ground under your feet in every direction — that's the gradient. You take a small step downhill, re-check the slope, take another step, and repeat. Millions of small, locally-informed steps eventually walk you down to a low point, even though you never had a map of the whole mountain.\n\nOnce every parameter's gradient is known, each one gets nudged a small step in the direction that reduces loss:\n\n```\nnew_weight = old_weight − (learning_rate × gradient)\n```\n\nThe **learning rate** controls how big that step is. Too large, and updates overshoot and destabilize training. Too small, and training crawls and never converges in a reasonable amount of time — this is one of the most important numbers a training run has to get right.\n\nSection 10 explained that residual connections give the training signal a shortcut back to the original input, and this is exactly the training signal it meant: the gradient computed here has to travel backward through all 80 stacked layers before it can update the earliest ones. Without a residual shortcut, that gradient tends to shrink toward zero (or blow up) somewhere along the way — a problem serious enough that, before residual connections became standard, networks this deep simply couldn't be trained effectively at all.\n\nThe word **batch** shows up in two different training/inference contexts and it's easy to conflate them:\n\n```\nTRAINING BATCH:   a group of training examples whose losses get\n                  averaged together before ONE gradient update is\n                  applied — done for training stability, not speed\n\nINFERENCE BATCH (section 20):  a group of concurrent user requests\n                  processed together in one forward pass, purely to\n                  use GPU parallelism efficiently — no gradients,\n                  no loss, no weight updates involved at all\n```\n\nSame word, two unrelated jobs — one is a training-stability concept, the other is a serving-efficiency concept.\n\n**This is why pre-training is so expensive and fine-tuning is comparatively cheap.** Fine-tuning (section 17) is the exact same four-step loop — forward pass, loss, backward pass, update — just run on a much smaller, curated dataset starting from already-good weights instead of random noise. Same mechanism, far fewer iterations needed.**This is why training needs dramatically more compute and memory than inference.** Training runs a backward pass (roughly 2x the compute of the forward pass alone) for every single example, and has to hold onto intermediate values from the forward pass in memory so the backward pass can use them. Inference (section 20) only ever runs the forward pass — no gradients, no backward pass, no loss — which is a large part of why serving a model is so much cheaper than training one.**\"The model learned X\" always cashes out to this loop.** Whenever this guide (or anyone) says a model \"learned\" a pattern, a bias, or a capability, this four-step loop — applied enough times, on enough examples showing that pattern — is the entire mechanism behind it. There is no other channel through which a transformer acquires anything.\n\nWith training itself explained, the next three sections cover what training actually produces: a model that's fluent but not always truthful (16), a process for turning that raw model into an assistant (17), and what happens as all of this gets scaled up (18).\n\nIn one line:the model was trained to predict statistically plausible text, never to be truthful — on accurate training data those two mostly line up, but \"mostly\" is exactly where confident, fluent, completely fabricated answers come from.\n\nSection 15 explained that a model is trained purely to minimize a loss based on plausibility — how surprised it was by the next real token. This section follows that fact to its most consequential conclusion.\n\n```\nWHAT MOST PEOPLE THINK THE MODEL DOES:\n\n\"Who founded Apple?\"  →  [searches internal knowledge]  →\n  finds fact: \"Steve Jobs, Steve Wozniak, Ronald Wayne — 1976\"  →  returns it\n\nWHAT THE MODEL ACTUALLY DOES:\n\n\"Who founded Apple?\"  →  \"given these tokens, what tokens are\n  most likely to come next?\"  →  generates a statistically\n  plausible continuation  →  returns that continuation\n```\n\nThere's no search, no lookup, no fact-retrieval step. The model generates what's statistically likely to follow — and it's usually correct simply because it trained on mostly-accurate text. **Correctness is a side effect of the training data, not the objective the model was optimized for.**\n\nThe model was trained on one task at massive scale: given \"The cat sat on the ___\", predict the next token. The **loss function** from section 15 — what the model was actually trying to minimize — measured only this:\n\n```\nLOSS = how surprised was the model by the actual next token?\n\nLow surprise  = high probability assigned to the correct token = good\nHigh surprise = low probability assigned to the correct token  = bad\n```\n\n**Truth never appears in that objective.** On a dataset of mostly-accurate human writing, predicting accurately and predicting plausibly mostly line up — but they are not the same thing, and the gap between them is exactly where hallucination lives.\n\nThe model has no database, no index, no filesystem. All of it is compressed into **parametric memory** — billions of weight values adjusted during training (section 15).\n\n```\nWhen the model \"knows\" Paris is the capital of France, it does NOT have\na record like { country: \"France\", capital: \"Paris\" }.\n\nIt has weight values that, when processing \"What is the capital of\nFrance?\", produce a high logit for \"Paris\" and low logits for\neverything else. The knowledge IS the pattern of weights — it has\nno other form.\n```\n\nCritically, this means the model has **no way to distinguish \"things I know well\" from \"things I barely saw in training.\"** Both are produced by the exact same generation mechanism.\n\nA database query either returns a row or returns null — a clean signal for \"I don't have this.\" The generation mechanism has no equivalent. It always produces *something*:\n\n```\n\"What is 2 + 2?\"\n→ high confidence → \"4\" → correct\n\n\"What did [an obscure private individual] eat for breakfast on\n March 3rd 2019?\"\n→ never seen this in training\n→ generation mechanism still runs anyway\n→ produces a plausible-sounding, completely fabricated answer\n→ delivered with the exact same fluency as the correct answer above\n```\n\n**Why doesn't it just say \"I don't know\"?** It can — the phrase exists in its vocabulary and training data — but the model also learned that, statistically, a confident answer is far more likely to follow a question than an admission of uncertainty. Wikipedia doesn't say \"I don't know.\" Textbooks don't hedge. Instruction tuning and RLHF (next section) substantially improve calibrated uncertainty, but the underlying generation mechanism still has no *native* uncertainty detector — expressing doubt is itself a learned pattern, not a genuine internal check.\n\n**1. Factual confabulation** — a fact was rare or absent in training data, so the model fills the gap with a plausible-sounding invention.\n\n```\n\"What papers did researcher X publish in 2019?\" (X is real but obscure)\n→ a list of realistic-looking paper titles, journals, co-authors —\n  none of which exist. The \"published papers\" format is a strong\n  pattern; the model fills it with statistically plausible content.\n```\n\n**2. Knowledge cutoff confusion** — confidently stating something that was true when training data was collected, but has since changed.\n\n```\n\"Who is the CEO of Twitter?\" → depending on training cutoff, might\nanswer Jack Dorsey, Parag Agrawal, or Elon Musk — with no awareness\nthat time has passed since training ended.\n```\n\n**3. Reasoning errors stated confidently** — a wrong intermediate step in a chain of reasoning produces a plausible-looking continuation that compounds.\n\n```\n\"3 boxes with 4 apples each, give away half, then buy 7 more —\n how many apples?\"\nCorrect: 3×4=12, 12/2=6, 6+7=13\nThe model might land on 13 (correct) or on a wrong number like 15,\nstated with identical confidence either way — each token was locally\nplausible even if the overall chain went wrong.\n```\n\nChain-of-thought prompting (having the model show its work step by step) substantially reduces this category by giving it room to self-correct before committing to a final answer.\n\n**4. Source fabrication** — asked for citations, the model produces realistic-looking but nonexistent references.\n\n```\n\"Cite three academic papers about transformer attention\"\n→ \"Smith et al. (2021). Attention Mechanisms in Modern NLP.\n   Journal of Machine Learning Research, 22(4), 112-134.\"\nCompletely fabricated — but every field (author, year, journal,\nvolume, pages) matches the *format* of a real citation perfectly,\nbecause that format is what the model actually learned.\n```\n\nThe architectural fix for hallucination is **RAG (Retrieval-Augmented Generation)** — instead of asking the model to recall facts from its parametric memory, you retrieve the actual relevant facts and place them directly in the prompt:\n\n```\nWITHOUT RAG:\n\"What is our refund policy?\"\n→ model generates a plausible-sounding policy from general training\n  patterns — may be entirely wrong for your specific business\n\nWITH RAG:\nretrieve: [actual refund policy text from your own database]\n\"Based on this policy: [text], what is our refund policy?\"\n→ model summarizes the text you handed it\n→ grounded in your real document, hallucination risk drops sharply\n```\n\nAlso worth internalizing: the model writes \"the answer is X\" with identical grammatical confidence whether X is correct or fabricated. Linguistic confidence is not a reliability signal — any system where factual accuracy actually matters (legal, medical, financial) needs grounding or independent verification, not just careful prompting.\n\nA raw, hallucination-prone next-token predictor is also nearly unusable as an assistant for a second reason: it doesn't even know it's supposed to act like one. The next section covers the training stages that fix that.\n\nIn one line:a base model only knows how to continue text plausibly; two more rounds of training — supervised examples of good answers, then reinforcement learning from ranked preferences — are what turn it into something that behaves like ChatGPT or Claude.\n\nThe model described so far — trained purely to predict the next token (section 15), and prone to hallucination as a direct result (section 16) — is almost unusable as an assistant on its own. Here's the three-stage pipeline that turns it into something like ChatGPT or Claude.\n\n```\nSTAGE 1: PRE-TRAINING\n─────────────────────────────────────────────────\nObjective:  predict next token\nData:       entire internet + books + code (trillions of tokens)\nDuration:   months, thousands of GPUs\nCost:       tens to hundreds of millions of dollars\nResult:     BASE MODEL — extraordinarily capable but completely\n            unaligned. Ask it a question and it might just continue\n            your question rather than answer it.\n\nSTAGE 2: SUPERVISED FINE-TUNING (SFT)\n─────────────────────────────────────────────────\nObjective:  teach the model the *format* of being an assistant\nData:       tens of thousands of human-written good question →\n            good answer pairs\nDuration:   days\nCost:       thousands of dollars\nResult:     INSTRUCTION-TUNED MODEL — knows it should answer\n            questions directly, but isn't yet well-calibrated on\n            what \"good\" means across edge cases.\n\nSTAGE 3: RLHF\n─────────────────────────────────────────────────\nObjective:  align behavior with human preferences\nData:       human rankings of multiple candidate model outputs\nDuration:   weeks\nCost:       millions of dollars\nResult:     CHAT MODEL — refuses harmful requests, expresses\n            uncertainty appropriately, behaves the way the lab intended.\nYou give the base model:  \"What is the capital of France?\"\n\nIt might output:\n\"What is the capital of France? This is a question commonly asked\n in geography quizzes. The answer varies depending on...\"\n```\n\nIt just continues the text like a document — it has no concept of \"being an assistant\" yet.\n\n**Fine-tuning** means continuing to train an already-trained model on new data — not training from scratch. It's the exact same forward-pass/loss/backward-pass/update loop from section 15, just run on a much smaller, curated dataset. The model keeps everything from pre-training and additionally learns new patterns from that dataset:\n\n```\nExample 1:\n  HUMAN:     \"What is the capital of France?\"\n  ASSISTANT: \"Paris is the capital of France.\"\n\nExample 2:\n  HUMAN:     \"Write a Python function to reverse a string.\"\n  ASSISTANT: \"Here is a Python function that reverses a string:\n              def reverse_string(s): return s[::-1]\"\n\nExample 3:\n  HUMAN:     \"How do I make a bomb?\"\n  ASSISTANT: \"I can't help with that.\"\n```\n\nTens of thousands of examples like these, written by human contractors, trained with the same next-token objective as pre-training — except now the \"correct\" next tokens are the human-written assistant responses. SFT teaches **format and style**: questions get answered directly, harmful requests get declined, instructions get followed. It doesn't yet teach the model to reliably tell a good answer from a mediocre one across the huge variety of real user requests — that's RLHF's job.\n\n**Reinforcement learning** is a training paradigm where a model (the \"policy\") takes actions, gets a reward signal for how good those actions were, and learns to maximize that reward over time — as opposed to supervised learning, where you just show correct examples directly.\n\nRLHF runs in two sub-stages:\n\n**A — Train a reward model.** You can't have humans rate every output during training (too slow, too expensive), so you first build an automated stand-in for human judgment:\n\n```\nStep 1: Generate multiple responses to the same prompt:\n  \"Explain what a transformer is.\"\n  A: clear, accurate, appropriately detailed\n  B: vague, technically true but not actually explanatory\n  C: about electrical transformers — wrong context entirely\n\nStep 2: Human raters rank them: A > B > C\n\nStep 3: Train a separate reward model on thousands of these\n        (prompt, response A, response B, ranking) triplets.\n\nStep 4: The trained reward model can now score any response\n        without a human in the loop:\n        Response A → 0.91,  Response B → 0.54,  Response C → 0.12\n```\n\n**B — RL training against the reward model:**\n\n```\nLOOP:\n1. Give the SFT model a prompt\n2. It generates a response\n3. The reward model scores the response\n4. Update the model's weights to produce higher-scoring responses\n5. Repeat millions of times\n```\n\nA **KL divergence penalty** keeps the model from drifting too far from the original SFT model during this process — without it, the model can find ways to get high reward scores that have nothing to do with what humans actually want (**reward hacking**):\n\n```\nExample: if the reward model was trained on data where longer,\nmore detailed responses scored higher, an unconstrained model might\nlearn to pad every answer with filler text to farm reward — great\nscore, terrible for users. The KL penalty caps how far it can drift.\n```\n\nStandard RLHF depends on human raters, who are expensive, inconsistent, and don't scale easily. Anthropic's alternative, **Constitutional AI (CAI)** — a form of **RLAIF**, Reinforcement Learning from AI Feedback — replaces much of the human ranking step with AI self-critique against an explicit set of written principles:\n\n```\n1. Define a \"constitution\": principles like \"be helpful, harmless,\n   and honest,\" \"prefer responses that don't assist illegal\n   activity,\" \"choose the response least likely to contain\n   misinformation.\"\n\n2. Have the model critique its own candidate responses against\n   these principles: \"Which of these two responses better follows\n   the principle of being helpful without causing harm?\"\n\n3. Train the reward model on these AI-generated rankings, instead\n   of (or alongside) human rankings — which scales far more\n   cheaply and consistently.\n```\n\nThis is why Claude has a distinct, consistent character — it was deliberately shaped by the specific principles written into its constitutional training.\n\n**Direct Preference Optimization (DPO)** achieves a similar result to RLHF without training a separate reward model at all — human preference data (response A preferred over response B) is used to update the model's weights directly, in one step:\n\n```\nRLHF:  SFT model → reward model → RL training loop → chat model\n       (three stages, more complex, can be unstable)\n\nDPO:   SFT model → direct preference training → chat model\n       (two stages, simpler, more reproducible)\n```\n\nMany recent open models (Llama 3, Mistral) use DPO instead of full RLHF.\n\n**Base models and chat models are different products.** Never use a raw base model in production directly — always use the instruction-tuned / RLHF (or DPO) variant.**Model behavior can shift silently between versions.** RLHF training isn't perfectly reproducible, so provider model updates can quietly change refusal thresholds, verbosity, or formatting. Pinning a specific model version (e.g.`claude-3-5-sonnet-20241022`\n\nrather than`-latest`\n\n) protects a prompt that already works from breaking under you.**Refusals are learned, not hardcoded.** They come from the reward model assigning low scores to harmful responses across training — which means they're probabilistic. The same underlying request phrased differently can get a different outcome. Don't treat model refusal as your only safety layer in a real system.**System prompts work because SFT taught the model to treat them specially**— the fine-tuning data explicitly included examples of following system-prompt instructions, so the behavior is learned, not a special hardcoded mode.\n\nEverything so far has been about a single model of fixed size. The next section covers what changes — often surprisingly abruptly — as that size scales up.\n\nIn one line:bigger models don't just get incrementally better at the same things — past certain parameter-count thresholds, entirely new capabilities (multi-step reasoning, few-shot learning) switch on that smaller models literally cannot do at all.\n\n```\nGPT-2   (2019):  1.5 billion parameters  → decent text completion\nGPT-3   (2020):  175 billion parameters  → surprisingly good reasoning\nGPT-4   (2023):  ~1 trillion parameters  → passes bar exam, complex math\n\nThe jump from GPT-2 to GPT-3 wasn't just \"better at the same things.\"\nGPT-3 could do things GPT-2 literally could not do at all — few-shot\nlearning, chain-of-thought reasoning, complex coding. These capabilities\ndidn't improve gradually. They appeared suddenly past a scale threshold.\nThat's emergence.\n```\n\nA **parameter** is a single number inside one of the model's weight matrices (like the Wq, Wk, Wv from section 8), adjusted during training (section 15).\n\n```\nA small weight matrix (4×4, for illustration):\n\nWq = [ 0.23  -0.45   0.12   0.87 ]\n     [ 0.56   0.34  -0.23   0.45 ]\n     [-0.12   0.78   0.56  -0.34 ]\n     [ 0.89  -0.12   0.34   0.23 ]\n\n16 parameters in this one small matrix. A real model has thousands\nof matrices, each with millions of parameters — billions in total.\n```\n\nEvery parameter is a number nudged countless times during training to reduce prediction error. The entire collection of parameter values *is* what the model \"knows\" — its parametric memory (section 16).\n\n```\nModel             Parameters    Approx GPU memory to run\n──────────────────────────────────────────────────────────────\nLlama 3.2 1B      1 billion     ~2 GB    ← runs on a phone\nLlama 3.2 3B      3 billion     ~6 GB    ← runs on a laptop\nLlama 3.1 8B      8 billion     ~16 GB   ← runs on 1 consumer GPU\nLlama 3.1 70B     70 billion    ~140 GB  ← needs 2-4 datacenter GPUs\nLlama 3.1 405B    405 billion   ~810 GB  ← needs 8-16 datacenter GPUs\n```\n\nMemory scales roughly linearly with parameter count, since each parameter is typically stored as 2 bytes (float16): `7B × 2 bytes = 14 GB`\n\n, `70B × 2 bytes = 140 GB`\n\n. This is exactly why you can't run GPT-4 on a laptop, and why API access exists at all.\n\n```\nKNOB 1 — Depth (number of layers):\n  GPT-2: 48 layers   GPT-3: 96 layers   GPT-4: 120+ (estimated)\n\nKNOB 2 — Width (model dimension, size of each token's vector):\n  GPT-2: 1,600 numbers per vector   GPT-3: 12,288 numbers per vector\n\nKNOB 3 — Number of attention heads:\n  GPT-2: 25 heads per layer   GPT-3: 96 heads per layer\n```\n\n**Scaling laws** are empirically discovered relationships between model size, training data size, compute budget, and resulting performance. The key one, the **Chinchilla scaling law** (DeepMind, 2022):\n\n```\nFor a given compute budget: model parameters ≈ training tokens × 20\n\n10 billion parameter model  →  needs ~200 billion training tokens\n70 billion parameter model  →  needs ~1.4 trillion training tokens\n\nThe original GPT-3 (175B parameters) was trained on only 300B tokens\n→ significantly undertrained by this standard\n→ a 70B model trained on 1.4T tokens can outperform it\n→ this is why Llama 2 70B is competitive with early GPT-3 despite\n  having fewer than half the parameters\n```\n\nThe practical lesson: **a smaller model trained on more data can beat a bigger model trained on less.** Parameter count alone doesn't predict capability — the ratio of parameters to training tokens does.\n\n**Emergence**: a capability that appears suddenly at a scale threshold, rather than improving gradually. Below the threshold, the model can't do it at all; above it, it can — closer to a phase transition (water turning to ice at 0°C) than a smooth curve.\n\n```\nCAPABILITY                            APPEARS AROUND\n──────────────────────────────────────────────────────────\nFew-shot learning (learn from          10B+ parameters\n  examples in the prompt, no retraining)\nChain-of-thought reasoning             60B+ parameters\nInstruction following without examples 60B+ parameters\nCode generation                        60B+ parameters\nCalibrated uncertainty (\"I'm not sure\" 100B+ parameters\n  when genuinely uncertain)\nTheory of mind basics (reasoning about 175B+ parameters\n  what others know or believe)\n```\n\n**Why does this happen?** Researchers don't have a fully settled answer. Two leading hypotheses, likely both partially true:\n\n```\nHYPOTHESIS 1 — Capability composition:\n  A complex capability like chain-of-thought reasoning actually\n  requires several sub-skills (following multi-step instructions,\n  holding intermediate results, recognizing a completed step,\n  moving on correctly). Each sub-skill improves gradually with scale.\n  If all of them individually cross their own threshold around the\n  same parameter count, the composite capability appears to \"switch\n  on\" suddenly — but it's really the last sub-skill crossing its line.\n\nHYPOTHESIS 2 — Metric non-linearity:\n  The benchmark itself may be scored as strictly right/wrong, while\n  the underlying capability actually improves smoothly underneath.\n  A 55B model might be \"40% of the way\" to solving a math problem\n  correctly, but a binary correct/incorrect metric records that as\n  a flat 0. At 65B it crosses into fully correct — the metric jumps\n  from 0 to 1, which looks like emergence but may just be a\n  measurement artifact on top of gradual improvement.\n```\n\n**Mixture of Experts**: instead of every token passing through every parameter (a \"dense\" model), the model has many specialized sub-networks (\"experts\"), and a router activates only a small subset of them per token.\n\n```\nDENSE MODEL:  every token → passes through ALL parameters → expensive\nMOE MODEL:    every token → router picks 2-4 relevant experts →\n              token only passes through those → rest sit idle for it\n\nGPT-4 is believed to use MoE:\n  Estimated total parameters:  ~1.8 trillion\n  Estimated active parameters: ~220 billion per token\n\nMeaning: GPT-4 has the *knowledge* of a 1.8T-parameter model but\nroughly the *inference cost* of a ~220B-parameter model, because\nmost parameters are inactive for any given token.\n```\n\nThis is why bigger total parameter count doesn't automatically mean a proportionally bigger API bill — cost tracks *active* parameters, not total ones.\n\n**Model selection is a capability question before it's a cost question.** If a task genuinely needs chain-of-thought reasoning, a 7B model will fail no matter how cleverly you prompt it — the capability doesn't exist yet at that scale. If the task is simple classification, using a frontier model is just waste.**Smaller, better-trained models keep closing the gap**— reassess model tier choices periodically as training efficiency improves (e.g. Llama 3.1 8B, trained on 15 trillion tokens, beats GPT-3 175B on many benchmarks).**Public benchmark scores aren't your task's score.** A model scoring 90% on a general benchmark can still fail badly on a narrow domain that was underrepresented in its training data — always evaluate against your actual use case.\n\nEverything up to this point has been about the LLM itself. The last two sections zoom out to two things built *around* the model in production: the embedding vectors that power search and retrieval, and the economics of actually serving all of this.\n\nIn one line:the same \"meaning becomes geometry\" trick from section 2 also works at the sentence and document level, which is the entire foundation of semantic search, RAG, and every vector database.\n\nThere are two distinct uses of the word \"embedding\" in this space, and conflating them is a common source of confusion:\n\n```\nUSE 1 — TOKEN EMBEDDINGS (inside the LLM itself)\n  Each token → one vector, used internally during generation,\n  refined through the attention and feed-forward mechanics covered\n  in sections 5 through 10 above.\n\nUSE 2 — SENTENCE / DOCUMENT EMBEDDINGS (for search & retrieval)\n  An entire sentence or paragraph → a single vector.\n  This is what powers semantic search and RAG, and what vector\n  databases like Pinecone, pgvector, or Weaviate actually store.\n```\n\nBoth share the same foundational idea from section 2: **meaning as geometry.**\n\n```\nIF meaning can be represented as a point in space\nTHEN similar meanings should sit close together\nAND different meanings should sit far apart\n\n\"cat\"  and \"dog\"     → close together  (both animals, pets)\n\"cat\"  and \"car\"      → far apart       (animal vs machine)\n\"happy\" and \"joyful\"  → very close      (synonyms)\n\"happy\" and \"sad\"     → far apart       (antonyms)\n```\n\nAn embedding model is a function that converts text into a point in high-dimensional space such that semantic relationships become geometric relationships you can measure with math.\n\nA vector's **dimension** is just how many numbers are in its list:\n\n```\n2-dimensional:    [0.5, 0.8]              — a point on a 2D plane\n3-dimensional:    [0.5, 0.8, 0.3]         — a point in 3D space\n1536-dimensional: [0.5, 0.8, 0.3, ..., 0.12]  — a point in a space\n                  humans can't visualize, but the math is identical\n```\n\nModern embedding models use 768 to 4096 dimensions. The model learns what each dimension represents during training — nobody hand-labels them. A rough 3D sketch of what the resulting space looks like:\n\n```\n                        \"joyful\"  \"happy\"\n                              ●  ●\n                         \"sad\" ●\n                                        ● \"cat\"\n                                        ● \"kitten\"\n                                   ● \"dog\"\n\n                    ● \"car\"\n               ● \"truck\"\n          ● \"bus\"\n\nSimilar concepts cluster together. Different concepts sit far apart.\n```\n\nSee it move.This ASCII sketch is a cartoon of something you can explore for real, on actual trained embeddings, at. It loads real word-embedding datasets, projects them down to 3D so you can rotate and zoom the actual cluster structure, and lets you search for a word to see its true nearest neighbors in the embedding space — the best way to build real intuition for \"meaning as geometry\" beyond a hand-drawn diagram.[projector.tensorflow.org]\n\nThe embedding matrix starts as random numbers. Prediction pressure from the next-token objective (section 15) is what sculpts it into something meaningful:\n\n```\nTraining sentence: \"The cat sat on the mat\"\n\nTo predict \"sat\" after \"The cat,\" the model must extract useful\nsignal from \"cat\"'s vector. If that vector encodes something like\n\"living creature that can perform actions,\" that helps predict\naction words → lower loss → the vector gets nudged to encode this\neven better.\n\nElsewhere in training data: \"dog sat,\" \"bird sat,\" \"child sat\" —\nall push their subject vectors toward a similar region, because\nthey all share similar predictive context.\n```\n\nTokens that appear in similar contexts end up with similar vectors — similar context is, functionally, similar meaning.\n\n```\nking vector - man vector + woman vector ≈ queen vector\n\nMeaning: start at \"king,\" remove the \"man\" concept, add the\n\"woman\" concept, land near \"queen.\" Not hardcoded — it emerges\nbecause kings/queens and man/woman appear in parallel contexts\nthroughout the training data.\n\nMore examples:\nParis  - France  + Italy   ≈ Rome    (capital relationship transfers)\nwalked - walk    + run     ≈ ran     (tense relationship transfers)\ndoctor - man     + woman   ≈ nurse   (an unfortunate gender bias\n                                       learned directly from training text)\n```\n\nThat last one matters: embedding geometry reflects whatever biases exist in the training data, not ground truth — a real, practical problem in production systems that use embeddings for classification or retrieval. (The TensorFlow Projector tool above lets you test this kind of vector arithmetic yourself on real embeddings.)\n\n```\nTOKEN EMBEDDINGS (what the LLM uses internally):\n\"The cat sat\"  →  3 tokens  →  3 separate vectors, one per token\nEach vector's meaning shifts with context (via attention).\nYou cannot directly compare two whole sentences with these.\n\nSENTENCE EMBEDDINGS (what RAG/search uses):\n\"The cat sat on the mat\"  →  ONE single vector for the whole sentence\nTwo sentences can now be compared directly by comparing their\nsingle vectors.\n```\n\nSentence embeddings come from specialized embedding models, distinct from the generative LLM:\n\n```\nOpenAI text-embedding-3-large              → 3072 dimensions\nOpenAI text-embedding-3-small              → 1536 dimensions\nsentence-transformers/all-MiniLM-L6-v2     → 384 dimensions (open-source)\nBGE-M3 (BAAI)                              → 1024 dimensions (multilingual)\nCohere Embed v3                            → 1024 dimensions\n```\n\nA **bi-encoder** encodes two pieces of text *independently* into vectors, then compares the vectors afterward. (\"Bi\" = two separate passes of the same encoder.)\n\n```\nDOCUMENT: \"Nepal is a landlocked country in South Asia\"\n             ↓ [encoder model]\n          [0.34, -0.12, 0.67, ...]  ← stored in a vector database\n\nQUERY:    \"Where is Nepal located?\"\n             ↓ [same encoder model]\n          [0.31, -0.09, 0.71, ...]\n\nCOMPARISON: are the document vector and query vector close in space?\n            close  → relevant document\n            far    → not relevant\n```\n\nYou might reach for Euclidean (straight-line) distance first, but it's sensitive to vector *length*, not just direction — which produces wrong answers:\n\n```\n\"cat\"    → [0.5,  0.8]    (short vector)\n\"feline\" → [2.0,  3.2]    (long vector, same DIRECTION as \"cat\")\n\"car\"    → [0.8, -0.5]    (different direction)\n\nEuclidean distance:\n\"cat\" to \"feline\" = large  ← wrong! same meaning, different magnitude\n\"cat\" to \"car\"    = small  ← wrong! different meaning, similar magnitude\n```\n\n**Cosine similarity** measures the *angle* between two vectors, ignoring length entirely — same direction gives 1.0, opposite gives -1.0, perpendicular gives 0:\n\n```\ncosine(\"cat\", \"feline\") = 1.0   ← correct: same direction = same meaning\ncosine(\"cat\", \"car\")    = 0.3   ← correct: different direction = different meaning\n\nThe formula: cosine_similarity(A, B) = (A · B) / (|A| × |B|)\nwhere A · B is the dot product (section 8) and |A|, |B| are vector\nlengths. Dividing by the lengths cancels out magnitude, leaving\nonly direction.\n```\n\nIn practice, relevant document/query pairs in a RAG system typically score 0.7–1.0, while irrelevant pairs land around 0.0–0.5.\n\nThese are often confused, but it's one underlying idea used two different ways:\n\n```\nCROSS-ENCODER (single encoder, one pass):\n[\"query text\" + \"document text\"]  →  [encoder]  →  one similarity score\nBoth texts are seen TOGETHER in one pass. The model can directly\nmodel interaction between them. Output is a score, not a vector.\n\nBI-ENCODER (same encoder, two separate passes):\n\"query text\"     →  [encoder]  →  vector_A  ┐\n                                             ├→ compare → similarity score\n\"document text\"  →  [encoder]  →  vector_B  ┘\nEach text is encoded alone — the model never sees them together.\nOutput is two vectors, compared afterward.\n\nANALOGY:\nCross-encoder = interviewing two candidates together in one room —\n                they interact, context is shared.\nBi-encoder    = interviewing each candidate separately, then\n                comparing your notes afterward.\n```\n\nA cross-encoder is **more accurate** (it can model direct interaction between query and document) but **cannot scale to large corpora** — it produces exactly one score per (query, document) pair, so scoring against a million documents means running the model a million separate times:\n\n```\n5 documents     →   5 runs    →  fast, fine\n50 documents    →   50 runs   →  acceptable for reranking\n1,000,000 docs  →   1,000,000 runs →  hours — completely unusable\n```\n\nThis is why production retrieval systems use a **two-stage pipeline**:\n\n```\nYOUR CORPUS: 1,000,000 documents\n\nOFFLINE (done once): every document → [bi-encoder] → vector → stored\n\nQUERY ARRIVES: \"what is the retry limit?\"\n\nSTAGE 1 — Bi-encoder retrieval (fast, milliseconds):\n  query → [bi-encoder] → query vector\n  compare against all 1,000,000 stored vectors → top 50 candidates\n\nSTAGE 2 — Cross-encoder reranking (accurate, 1-2 seconds):\n  cross-encoder(query + doc_4)   → 0.94\n  cross-encoder(query + doc_892) → 0.87\n  cross-encoder(query + doc_234) → 0.71\n  ...50 runs total...\n  sort scores → keep the top 5\n\nSTAGE 3 — Feed those top 5 documents to the LLM as context → answer\n```\n\nBi-encoder for broad, cheap recall across everything; cross-encoder for precise, expensive reranking of a short list. (One edge case: concatenating multiple chunks into a single cross-encoder call *technically* works but reliably hurts accuracy, since attention gets diluted across multiple unrelated topics at once — always score chunks separately.)\n\nGeneric embedding models are trained on general internet text, so their vector space reflects everyday language:\n\n```\nGeneral embedding model:\n\"consideration\" → near \"thought,\" \"reflection,\" \"deliberation\"\n                  (everyday English meaning)\n\nLegal meaning:\n\"consideration\" = something of value exchanged in a contract\n                  (a completely different, domain-specific meaning)\n\nQuery: \"What is the consideration in this contract?\"\n→ using a general embedding model retrieves documents about\n  \"reflecting on\" things → misses the actual contract clauses\n  → retrieval — and therefore the whole RAG answer — fails\n```\n\nThis is why domain-specific embedding models exist, and why embedding model choice is a real engineering decision, not an afterthought.\n\n**The embedding model is the foundation of a RAG system.** A bad embedding model produces bad vectors, which produces bad retrieval — and no amount of LLM quality can compensate for retrieval that surfaces the wrong documents in the first place.**The embedding model and vector database must agree on dimensions**, and switching embedding models later means re-embedding your entire document set and rebuilding the index — not a cheap operation at scale, so choose carefully upfront.**Embedding model updates can silently break retrieval quality** if a provider updates a model internally and new query vectors end up in a slightly different space than your already-stored document vectors — no error is thrown, answers just quietly get worse. Pin versions and monitor retrieval quality directly.**Cosine similarity thresholds are a tuning parameter**, not a universal constant — typical starting points are 0.75+ for high-confidence-only retrieval, ~0.65 as a balanced default, and 0.50+ for broad recall paired with reranking afterward.**Multilingual products need multilingual embedding models**(e.g. BGE-M3, multilingual-e5-large) — an English-only embedding model simply can't match an English query to a relevant document written in another language.\n\nThe last remaining question this guide hasn't touched: given everything above, what actually determines how fast and how expensive it is to run one of these models in production?\n\nIn one line:four things determine LLM cost and speed — how many parameters engage per token, what precision they're stored at, how many requests get batched together, and how fast the GPU can move those parameters from memory to compute.\n\nFour factors determine how fast and how expensive it is to run an LLM, and they interact with each other:\n\n```\n1. Parameter count       → how much computation per forward pass\n2. Quantization          → how much memory each parameter consumes\n3. Batch size            → how many requests get processed simultaneously\n4. GPU memory bandwidth  → how fast parameters move from memory to compute\n```\n\nEvery parameter (section 18) participates in every forward pass, for every token generated:\n\n```\n7B model:  7,000,000,000 parameters engaged per token\n70B model: 70,000,000,000 parameters engaged per token → 10x the computation\n\nResult: a 70B model generates tokens roughly 10x slower than a 7B\nmodel on identical hardware, and costs roughly 10x more to serve.\n```\n\nThis is a large part of why provider pricing scales with model size:\n\n```\nGPT-4o mini   →  $0.60 / million output tokens  (small, fast)\nGPT-4o        →  $15.00 / million output tokens (large, capable)\nClaude Haiku  →  $1.25 / million output tokens\nClaude Sonnet →  $15.00 / million output tokens\n```\n\n**Quantization**: reducing the numerical precision each parameter is stored at, trading some quality for less memory and faster compute.\n\n```\nfloat32  →  32 bits/parameter  → full precision, used during training\nfloat16  →  16 bits/parameter  → half the memory of float32,\n                                  negligible quality loss, standard for inference\nint8     →   8 bits/parameter  → quarter the memory, small quality loss\nint4     →   4 bits/parameter  → eighth the memory, noticeable quality loss\n```\n\nConcretely, for a 70B model:\n\n```\nfloat32:  70B × 4 bytes = 280 GB  ← needs 4× H100 GPUs\nfloat16:  70B × 2 bytes = 140 GB  ← needs 2× H100 GPUs\nint8:     70B × 1 byte  =  70 GB  ← fits on 1× H100 GPU\nint4:     70B × 0.5 byte = 35 GB  ← fits on 1× A100 40GB GPU\n```\n\nThis is exactly why you can run a 70B open-weight model locally on two consumer GPUs using int4 — something completely impossible at full float32 precision. Lower precision also speeds things up: less data to move means faster loading into compute units (see Factor 4).\n\n**Quick decision guide:** float16 for anything quality-sensitive and running on real GPU infrastructure; int8 as a solid middle ground for local deployment; int4 only when memory is the hard constraint and you've verified the quality drop is acceptable for your specific task.\n\n**Batch size** (the inference sense — see section 15 for how this differs from a *training* batch): the number of independent requests processed together in one forward pass. GPUs are built for massive parallelism, and a single request barely taxes that capacity:\n\n```\nWITHOUT BATCHING:\nRequest 1 → forward pass → response\nRequest 2 → forward pass → response\nGPU utilization: ~5% per request. 3 requests take ~3x as long.\n\nWITH BATCHING:\nRequests 1, 2, 3 → ONE forward pass for all three → 3 responses\nGPU utilization: much higher. 3 requests processed in roughly\nthe time of 1.\n```\n\nThis is how providers serve thousands of concurrent users on a fixed pool of GPUs. The tradeoff: bigger batches mean higher throughput but higher latency per individual request (since a request has to wait for the batch to fill). This is also why API latency can vary with traffic load — busier periods fill batches faster and can actually be more efficient per-request, while quiet periods mean smaller batches and lower utilization.\n\n**Memory bandwidth**: the rate data moves from GPU memory (VRAM) into the compute cores, measured in GB/s. For every token generated, the GPU has to load all relevant parameters from VRAM, compute, store results, and repeat across every layer.\n\n```\n70B model at float16 = 140 GB of parameters to load, per token generated\n\nH100 GPU memory bandwidth: 3.35 TB/s\n\nTime to load 140 GB: 140 / 3,350 = 0.042 seconds = 42 ms per token\nTheoretical max speed on one H100: ~24 tokens/second for a 70B model\n```\n\nThis is why inference is described as **memory bandwidth bound**, not compute bound — for single-user generation, the GPU's compute cores mostly sit idle, waiting for parameters to arrive from VRAM. More raw compute doesn't help; more bandwidth does. It's also why newer GPU generations (H100 vs. A100) improve inference speed mainly through higher memory bandwidth, not just more cores.\n\n```\nPREFILL (processing the input prompt): all tokens processed in one\n  parallel pass → compute bound → more cores = faster → determines\n  time-to-first-token (TTFT)\n\nDECODE (generating output): one token per step, sequential →\n  memory-bandwidth bound → more bandwidth = faster → determines\n  total generation time\n\nShort prompt + long output  → fast prefill, slow decode\nLong prompt + short output  → slower prefill, fast decode\n```\n\nEvery concurrent user needs their own KV cache (section 13) sitting in VRAM alongside the model's own parameters:\n\n```\n70B model parameters:      140 GB\n100 concurrent users'\nKV caches combined:        potentially 50-100 GB more\n\nTotal VRAM needed: 190-240 GB → requires 3 H100 GPUs minimum\n\nServing more concurrent users often requires more GPUs not because\nof extra compute, but because of KV cache memory pressure.\nGIVEN:\n  Model: 70B parameter (Claude Sonnet-tier pricing)\n  Traffic: 1,000 requests/day\n  Average input: 1,000 tokens/request\n  Average output: 500 tokens/request\n\nSTEP 1 — Token volumes:\n  Input tokens per day:  1,000 × 1,000 = 1,000,000\n  Output tokens per day: 1,000 × 500   =   500,000\n\nSTEP 2 — Cost:\n  Input:  1,000,000 × ($3/1M)  = $3.00/day\n  Output:   500,000 × ($15/1M) = $7.50/day\n  Total: $10.50/day ≈ $315/month\n\nSTEP 3 — Optimization levers:\n  Output tokens cost 5x input tokens, so cutting average output\n  from 500 → 200 tokens saves 300,000 × ($15/1M) = $4.50/day\n  ≈ $135/month.\n  Semantic caching with a 30% hit rate saves ~30% of total cost\n  ≈ another $94.50/month.\n```\n\n**Output tokens are the biggest cost lever you directly control.** Setting`max_tokens`\n\ntightly and asking for concise responses (\"answer in one sentence\" vs. \"explain in detail\") can mean a 10x cost difference with no real quality loss for many tasks.**Model tiering is a real economics decision, not just an optimization nice-to-have.** Routing simple classification or yes/no decisions to a small, cheap model and reserving the expensive model for genuinely complex requests can cut costs 60-70% with no quality loss where it matters — if 70% of your traffic is simple, running 100% of it through the expensive model is pure waste.**API vs. self-hosting is a real break-even calculation, not a vibe.** API access wins for unpredictable or low traffic and no appetite for GPU ops. Self-hosting wins at high, predictable volume — e.g. at 1 billion output tokens/month, API cost (~~$15,000/month at Sonnet-tier pricing) can dwarf an amortized self-hosted H100 setup (~~$2,000/month).**Diagnose latency by bottleneck, not by guessing.** Slow time-to-first-token points at prefill (shorten the prompt, use faster compute). Slow total generation time points at decode (stream the response, use a smaller model, use higher-bandwidth hardware). Slowdowns specifically under many concurrent users point at KV cache memory pressure (more GPUs, shorter context limits, cache eviction policies).\n\nThat covers the full arc: text in, vectors, attention, generation, training, scale, embeddings, and cost. The glossary below is a flat reference for looking any single term back up without re-reading a whole section.\n\nOrdered to match the order terms first appear in this guide, so it also works as a quick recap if read top to bottom.\n\n| Term | Plain-English Definition |\n|---|---|\nVector |\nA list of numbers representing a word, sentence, or concept. |\nToken |\nA chunk of text (roughly a word or word-piece) mapped to an integer ID. |\nEmbedding |\nThe vector a token or piece of text gets converted into; similar meanings → similar vectors. |\nWeight matrix |\nA learned grid of numbers used to transform one vector into another. |\nPositional encoding |\nInformation added to embeddings so the model knows token order. |\nRoPE |\nA modern positional encoding method that rotates Q/K vectors by an angle based on position; enables longer context windows. |\nSelf-attention |\nThe mechanism letting every token directly consider every other token when updating its meaning. |\nQuery (Q) |\nA token's \"what context do I need?\" vector. |\nKey (K) |\nA token's \"here's what I contain\" vector, compared against queries. |\nValue (V) |\nA token's \"here's what I contribute if selected\" vector. |\nDot product |\nA single number measuring how aligned two vectors are. |\nSoftmax |\nConverts a list of raw numbers into probabilities that sum to 1. |\nMulti-head attention |\nRunning several independent Q/K/V attention computations in parallel, each specializing in a different relationship type. |\nCausal mask |\nBlocks a token from attending to any token that comes after it, so generation never \"cheats\" by looking at the future. |\nResidual connection |\nAdding a layer's input back onto its output so information (and training signal) can skip past the layer untouched. |\nLayer normalization |\nRescaling a vector's values to a consistent, stable range after each sub-layer, so deep stacks of layers train reliably. |\nFeed-forward network (FFN) / MLP |\nThe per-token sub-layer inside each transformer block that processes each token's vector independently, with no cross-token communication. |\nActivation function (e.g. GELU) |\nA nonlinear function applied inside the FFN; without it, stacked linear layers would collapse into a single linear transformation. |\nDecoder-only / causal model |\nAn architecture (GPT, Claude, Llama) trained with a causal mask so it only ever predicts forward, one token at a time — as opposed to encoder-only models (BERT) built to understand a whole passage at once. |\nLogits |\nRaw, unnormalized output scores before softmax turns them into probabilities. |\nAutoregressive generation |\nProducing text one token at a time, feeding each output back in as input for the next step. |\nGreedy decoding |\nAlways picking the single highest-probability token — deterministic. |\nSampling |\nPicking a token probabilistically from the distribution — non-deterministic. |\nTemperature |\nA number that sharpens (low) or flattens (high) the probability distribution before sampling. |\nTop-p (nucleus sampling) |\nKeeps only the smallest set of top tokens whose probabilities sum to at least p, then samples from those. |\nTop-k |\nKeeps a fixed number of top tokens regardless of the shape of the distribution. |\nKV cache |\nStored Key/Value vectors for past tokens, reused instead of recomputed on every generation step. |\nPrefill |\nThe parallel processing phase for your input prompt; determines time-to-first-token. |\nDecode |\nThe sequential, one-token-at-a-time generation phase; determines total generation time. |\nLoss function / cross-entropy loss |\nThe number that measures how \"surprised\" the model was by the real next token; training tries to minimize this. |\nGradient |\nFor a single parameter: how much the loss would change if that parameter were nudged slightly. |\nBackpropagation |\nThe algorithm that computes every parameter's gradient by working backward from the loss through every layer. |\nGradient descent |\nNudging every parameter a small step in the direction that reduces loss, repeated until the model converges. |\nLearning rate |\nHow big each gradient-descent step is; too high overshoots, too low crawls. |\nHallucination |\nConfident, fluent, but factually wrong or fabricated output — a side effect of optimizing for plausibility, not truth. |\nParametric memory |\nKnowledge implicitly encoded in a model's weights, with no explicit database-like structure. |\nRAG (Retrieval-Augmented Generation) |\nRetrieving real, relevant documents and injecting them into the prompt to ground the model's answer in actual facts. |\nPre-training |\nThe initial, massive-scale next-token-prediction training run that produces the base model. |\nSFT (Supervised Fine-Tuning) |\nFurther training a base model on curated human-written examples to teach it the assistant format. |\nRLHF |\nReinforcement Learning from Human Feedback — aligning a model's behavior using a reward model trained on human preference rankings. |\nReward model |\nA separate model trained to score how good a response is, standing in for human judgment during RLHF. |\nConstitutional AI / RLAIF |\nAnthropic's approach: using AI self-critique against written principles instead of relying solely on human rankings. |\nDPO (Direct Preference Optimization) |\nA simpler alternative to RLHF that updates model weights directly from preference data, without a separate reward model. |\nParameter |\nA single learned number inside a model's weight matrices. |\nScaling laws |\nEmpirical relationships between model size, training data size, and resulting performance. |\nEmergent capability |\nAn ability that appears suddenly at a scale threshold rather than improving gradually. |\nMixture of Experts (MoE) |\nAn architecture that activates only a small subset of specialized sub-networks per token, instead of the whole model. |\nSentence/document embedding |\nA single vector representing the meaning of an entire sentence or document, used for semantic search. |\nCosine similarity |\nA measure of how similar two vectors' directions are, ignoring their length. |\nBi-encoder |\nEncodes two texts independently, then compares the resulting vectors — fast, used for broad retrieval. |\nCross-encoder |\nEncodes two texts together in one pass to produce a single relevance score — accurate but slow, used for reranking. |\nQuantization |\nReducing the numeric precision parameters are stored at, to save memory at some cost to quality. |\nBatch size |\nThe number of requests processed together in a single forward pass (inference) — distinct from a training batch, see section 15. |\nMemory bandwidth |\nHow fast data moves from GPU memory to compute cores — the main bottleneck during generation. |\n\n*This guide distills the core mental model behind every modern LLM, in the order the data itself flows: text becomes vectors, position gets baked in, attention lets tokens exchange context (with a causal mask keeping generation honest, and residuals/LayerNorm keeping 80 stacked layers trainable), generation happens one probabilistic token at a time, and a training pipeline (backpropagation → pre-training → SFT → RLHF) turns a raw next-token predictor into a helpful assistant. Everything else — RAG, prompting techniques, agents, fine-tuning — is built on top of this foundation.*\n\n*Three interactive tools are linked inline above, right where they're most useful, rather than bundled at the end: the whole pipeline in animated 3D ( section 3), a live transformer block running in your browser (section 10), and a real, explorable embedding space (section 19).*", "url": "https://wpnews.pro/news/a-plain-english-guide-to-transformer-architecture", "canonical_source": "https://gist.github.com/rabin999/91aed4a645e92af4f195a351f497f627", "published_at": "2026-07-04 05:47:40+00:00", "updated_at": "2026-07-08 13:58:07.424899+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "neural-networks", "developer-tools"], "entities": ["GPT-4", "Claude", "Gemini", "Llama"], "alternates": {"html": "https://wpnews.pro/news/a-plain-english-guide-to-transformer-architecture", "markdown": "https://wpnews.pro/news/a-plain-english-guide-to-transformer-architecture.md", "text": "https://wpnews.pro/news/a-plain-english-guide-to-transformer-architecture.txt", "jsonld": "https://wpnews.pro/news/a-plain-english-guide-to-transformer-architecture.jsonld"}}