A Plain-English Guide to Transformer Architecture

A developer published a plain-English guide to transformer architecture, explaining how large language models like GPT-4, Claude, Gemini, and Llama process text from tokenization to autoregressive generation. The guide covers key concepts such as self-attention, multi-head attention, training via backpropagation, and inference economics, with interactive tools linked inline.

A from-scratch walkthrough of what happens between typing a prompt and getting a response — no ML background assumed. Every large language model — GPT-4, Claude, Gemini, Llama — is built on the same core idea. This guide follows that idea in the exact order data actually flows through a real model: text in, tokens, vectors, 80 stacked layers, and a word out — then loops back to explain how the model got trained in the first place. Read it top to bottom; each section is built to need only what came before it. The One Job an LLM Has https://gist.github.com/starred.atom 1-the-one-job-an-llm-has Vectors — Turning Words Into Math https://gist.github.com/starred.atom 2-vectors--turning-words-into-math The Full Pipeline at a Glance https://gist.github.com/starred.atom 3-the-full-pipeline-at-a-glance Tokenization https://gist.github.com/starred.atom 4-tokenization The Embedding Layer https://gist.github.com/starred.atom 5-the-embedding-layer Positional Encoding — Teaching the Model Word Order https://gist.github.com/starred.atom 6-positional-encoding--teaching-the-model-word-order Self-Attention — The Core Idea https://gist.github.com/starred.atom 7-self-attention--the-core-idea Self-Attention — The Mechanics Q, K, V https://gist.github.com/starred.atom 8-self-attention--the-mechanics-q-k-v Multi-Head Attention https://gist.github.com/starred.atom 9-multi-head-attention Inside One Transformer Block — Causal Masking, Residuals, LayerNorm & the Feed-Forward Network https://gist.github.com/starred.atom 10-inside-one-transformer-block--causal-masking-residuals-layernorm--the-feed-forward-network Why the Middle of a Long Prompt Gets Ignored https://gist.github.com/starred.atom 11-why-the-middle-of-a-long-prompt-gets-ignored Autoregressive Generation — How Words Actually Get Produced https://gist.github.com/starred.atom 12-autoregressive-generation--how-words-actually-get-produced The KV Cache — Why Generation Isn't Recomputed From Scratch https://gist.github.com/starred.atom 13-the-kv-cache--why-generation-isnt-recomputed-from-scratch Temperature & Sampling — Controlling Randomness https://gist.github.com/starred.atom 14-temperature--sampling--controlling-randomness How Training Actually Updates the Weights — Backpropagation & Gradient Descent https://gist.github.com/starred.atom 15-how-training-actually-updates-the-weights--backpropagation--gradient-descent Why Hallucination Happens — Architecturally https://gist.github.com/starred.atom 16-why-hallucination-happens--architecturally From Raw Model to Helpful Assistant SFT & RLHF https://gist.github.com/starred.atom 17-from-raw-model-to-helpful-assistant-sft--rlhf Model Scale & Emergent Abilities https://gist.github.com/starred.atom 18-model-scale--emergent-abilities Embeddings — The Geometry of Meaning https://gist.github.com/starred.atom 19-embeddings--the-geometry-of-meaning Inference Economics — Why LLMs Cost What They Cost https://gist.github.com/starred.atom 20-inference-economics--why-llms-cost-what-they-cost Glossary — Every Term in One Place https://gist.github.com/starred.atom 21-glossary--every-term-in-one-place Three free, interactive tools are linked inline below, right in the sections they visualize best, rather than bundled at the end — so you can pull one up the moment it's relevant instead of hunting for it later. In one line:an LLM is a single next-word predictor, run over and over — nothing else is happening underneath the hood. Strip away the hype, and every LLM is trained to do exactly one thing: Given some text, predict the most likely next word. That's it. There's no search engine inside, no database of facts, no reasoning module bolted on. The entire architecture — every layer, every mechanism described below — exists purely to answer one question, over and over, one word at a time: "Given what came before, what word comes next?" Everything from writing an essay to solving a math problem to holding a conversation is this same mechanism, repeated thousands of times in a row. Every section below is either part of how that one prediction gets made, or part of how the model got good at making it — keep that map in mind as you go. In one line:every word becomes a list of numbers, and similar words end up as similar-looking lists — that single trick is what everything else in this guide is built on top of. A vector is just a list of numbers. Nothing mystical about it. "cat" → 0.2, 0.8, -0.1, 0.5, 0.3, ... "dog" → 0.3, 0.7, -0.2, 0.4, 0.3, ... "car" → 0.9, -0.1, 0.8, -0.3, 0.1, ... In real models these lists have 512, 1024, or even 4096 numbers. Each number is called a dimension . The key insight that makes all of this work: meaning becomes geometry . Words with similar meaning end up with similar number patterns — so "cat" and "dog" land close together in this number-space, while "car" lands far away. You can literally do arithmetic on meaning this way more on that in section 19 https://gist.github.com/starred.atom 19-embeddings--the-geometry-of-meaning . For now, just accept one fact: every word gets converted into a vector before any computation happens. The model never touches raw text — only numbers. The next section shows the full journey these vectors take from raw text to a finished word. In one line:text becomes token IDs, token IDs become vectors, 80 stacked layers refine those vectors by having tokens talk to each other, and the last vector gets turned into a probability over every possible next word — then the whole thing repeats for the next word. We'll use one running example throughout this guide: Input: "The cat sat on the" Goal: predict → "mat" Here's the entire journey from text to output, every step numbered — including the two loops running at the same time layers stacked sequentially, heads running in parallel inside each layer and the outer autoregressive loop that wraps around the whole thing. Every numbered step here gets its own dedicated section later in this guide. ╔════════════════════════════════════════════════════════════════════╗ ║ TRANSFORMER ARCHITECTURE — ONE FULL GENERATION STEP ║ ╚════════════════════════════════════════════════════════════════════╝ STEP 1 · INPUT TEXT "The cat sat on the" │ ▼ STEP 2 · TOKENIZER text → integer IDs 464, 3797, 3332, 319, 262 │ ▼ STEP 3 · EMBEDDING LOOKUP each ID → vector e.g. 4096 numbers 5 tokens → 5 vectors │ ▼ STEP 4 · + POSITIONAL ENCODING position info added to each vector so the model knows token order │ ▼ ┌──────────────────────────────────────────────────────────────────┐ │ STEP 5 · TRANSFORMER LAYER STACK ↻ × 80 LAYERS, SEQUENTIAL │ │ layer 1's output → layer 2's input → ... → layer 80's output │ │ │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ ONE LAYER = │ │ │ │ │ │ │ │ 5a · MULTI-HEAD SELF-ATTENTION │ │ │ │ ┌──────┐ ┌──────┐ ┌──────┐ ↻ × 32 HEADS, │ │ │ │ │Head 1│ │Head 2│ │Head 3│ ... PARALLEL, inside THIS │ │ │ │ └──────┘ └──────┘ └──────┘ one layer only │ │ │ │ each head: Q,K,V → attention scores → weighted sum of V │ │ │ │ causal mask applied — no token attends to future tokens │ │ │ │ │ concatenate all 32 heads' outputs │ │ │ │ ▼ │ │ │ │ 5b · ADD & NORM residual connection + layer norm │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ 5c · FEED-FORWARD NETWORK per-token transformation │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ 5d · ADD & NORM residual connection + layer norm │ │ │ └───────────────────────┬────────────────────────────────────┘ │ │ │ output feeds into next layer as input │ │ ▼ │ │ ↻ repeat steps 5a–5d, 80 times total │ └──────────────────────────────────────────────────────────────────┘ │ ▼ STEP 6 · OUTPUT LINEAR LAYER last token's final vector × weight matrix → 100,000 raw NUMBERS logits — NOT percentages yet │ ▼ STEP 7 · TEMPERATURE SCALING logit ÷ temperature → adjusted logits still raw numbers │ ▼ STEP 8 · SOFTMAX adjusted logits → 100,000 PERCENTAGES probabilities, sum = 100% │ ▼ STEP 9 · SAMPLE pick one token from the probability distribution → e.g. "mat" │ ▼ STEP 10 · STOP CHECK EOS token? max tokens hit? stop sequence matched? │ NO │ YES ▼ ▼ STEP 11 · APPEND TOKEN STOP — return final text new token added to the sequence │ └───────────────↻ LOOP BACK TO STEP 5 sequence is now one token longer — a full forward pass runs again from step 5 Legend — what each loop means: ↻ × 80 LAYERS → sequential, stacked depth-wise layer 1 must finish before layer 2 starts this is "how deep" the network is ↻ × 32 HEADS → parallel, inside a single layer only all 32 run simultaneously, then merge this is "how many relationship types" one layer can detect at once ↻ OUTER LOOP → the autoregressive generation loop steps 5 through 11 repeat once per generated token, until stop One-line description per step: 1 text arrives as-is, human readable 2 text split into tokens, each mapped to an integer ID 3 each ID looked up in embedding table → vector 4 position information injected so order is not lost 5 80 layers refine the representation, each layer's 32 heads simultaneously detecting different relationships 6 final vector converted to one raw score per vocabulary word 7 temperature reshapes those raw scores before normalizing 8 raw scores converted into a real probability distribution 9 one token drawn from that distribution 10 check whether generation should stop here 11 if not stopping, append the token and repeat from step 5 See it move.Static diagrams only go so far.renders this bbycroft.net/llm exactpipeline as an animated, rotatable 3D model of a real, runnable GPT nano-GPT, GPT-2, and others — tokens flowing into embeddings, splitting into attention heads, merging back through Add & Norm, expanding through the feed-forward network, and collapsing down to logits at the output layer, all in one structure you can orbit around. It's the single best way toseethe shape of the steps above instead of just reading them. Worth pulling up now, before diving into the details, and again after finishing section 10 once every piece has a name. The rest of this guide walks through each of these steps in detail — starting with tokenization step 2 . In one line:text gets chopped into word-ish chunks and each chunk is swapped for a lookup number — step 2 of the pipeline above, and the very first thing that happens to your prompt. The model doesn't read text — it reads numbers. A tokenizer splits text into chunks called tokens roughly one word or word-piece and maps each token to an integer ID from a fixed vocabulary. "The cat sat on the" ↓ ↓ ↓ ↓ ↓ 464 3797 3332 319 262 These IDs are just lookup indices — no meaning yet. Meaning arrives in the next step. Vocabulary size varies by model: GPT-4 → ~100,000 tokens Claude → ~100,000 tokens Llama 3 → ~128,000 tokens Tokens aren't always whole words — rare or long words get split into sub-word pieces: "cat" → 1 token common word, gets its own ID "sat" → 1 token "Kathmandu" → 2 tokens less common, gets split "unprecedented" → 3 tokens long word, split into sub-pieces "धन्यवाद" → 4 tokens non-Latin script, more splits This is exactly why non-English text often costs more per word when you're billed by the token — tokenizers are trained mostly on English data, so other languages get sliced into more pieces. With text turned into IDs, the next step is turning those IDs into the vectors introduced in section 2. In one line:each token ID gets swapped for its learned vector — step 3 of the pipeline — and from this point on the model only ever works with numbers, never text again. Every integer ID gets looked up in a giant table to retrieve its vector. This table is learned during training. 464 → 0.12, 0.87, -0.34, 0.56, 0.01, ... ← "The" 3797 → 0.23, 0.91, -0.12, 0.44, 0.32, ... ← "cat" 3332 → -0.45, 0.34, 0.78, -0.23, 0.67, ... ← "sat" 319 → 0.67, -0.12, 0.34, 0.89, -0.45, ... ← "on" 262 → 0.11, 0.88, -0.33, 0.54, 0.02, ... ← "the" From this point forward, the model never sees the original text again — only these vectors. But these raw vectors are missing one thing: any sense of where each token sits in the sentence. That's step 4 of the pipeline, and the very next section. In one line:a "which position am I" signal gets mixed into every token's vector right here, immediately after the embedding lookup — step 4 of the pipeline — because the comparison math used in the next few sections has no built-in concept of order on its own. Section 5 mentioned that positional information gets added right after the embedding lookup. This section explains why that step exists and how it works — before diving into self-attention itself in sections 7 through 9. The next few sections explain self-attention — the mechanism that lets tokens exchange context by comparing their vectors mathematically. The one fact you need up front: that comparison is pure arithmetic, and pure arithmetic has no built-in concept of order. Sentence 1: "cat sat" Sentence 2: "sat cat" Without positional information, the vector for "cat" is identical in both sentences. The model can't tell which came first. But "dog bites man" and "man bites dog" contain the same words with entirely different meanings — the model needs position. Before the first transformer block runs, a positional vector is added to each token's embedding: token embedding positional vector final input vector "cat" 0.23, 0.91 + position 2 0.10, 0.30 = 0.33, 1.21 "sat" -0.45, 0.34 + position 3 0.20, 0.40 = -0.25, 0.74 Now the same word at different positions produces different final vectors — this is exactly the vector that gets handed to attention in the next section. Sinusoidal the original 2017 Transformer paper — position vectors computed from sine/cosine functions at different frequencies. Fixed, not learned. Doesn't generalize well beyond the training sequence length. Learned positional embeddings GPT-2 — a lookup table learned during training, one vector per position, up to a maximum length. Position 1 → 0.12, -0.34, 0.56, ... Position 2 → 0.45, 0.23, -0.12, ... ... Position 4096 → ... ← hard limit, nothing beyond this RoPE — Rotary Position Embeddings Llama and most modern models — instead of adding a position vector, RoPE rotates the Query and Key vectors introduced in the next section by an angle based on position, right before they get compared. Relative position gets baked directly into the attention score itself. RoPE generalizes far better beyond training length, which is why models can later be extended to much longer context windows than they were originally trained on: Original training context: 4,096 tokens With RoPE + fine-tuning: can extend to 128k, 200k, 1M tokens Without RoPE: hard ceiling at training length This is the mechanical reason Claude can handle 200k tokens of context and Gemini 1.5 Pro handles 1M. Learned embeddings: position vectors only exist up to position N; anything beyond is undefined territory for the model RoPE: angle rotation becomes extreme at very long distances; attention scores degrade past a point Both approaches: attention cost is O N² — quadratic. N = 10,000 → 100 million dot products N = 100,000 → 10 billion dot products Context windows aren't just a pricing decision — they're a real architectural and hardware constraint, since doubling context length quadruples the attention computation. With position now baked into every vector, the model is finally ready for the main event: self-attention. In one line:every token looks at every other token in the sentence and decides how much of its meaning to borrow from each one — the single mechanism that replaced older, sequential models and made modern LLMs possible. This is the single most important idea in modern AI. Let's build the intuition before touching any math. Consider this sentence: "The animal didn't cross the street because it was too tired." What does "it" refer to? The animal, not the street. You resolved that by connecting "it" back to "animal" across seven words. That connection between two distant parts of a sentence is called a long-range dependency . More examples: "The trophy didn't fit in the suitcase because it was too big." ↑ what does "it" mean here? → "trophy" far back in the sentence "The bank by the river where they used to fish before the factory was built in 1987 might close." ↑ "bank" = river bank or financial bank? → resolved by "river", 3 words away Human brains resolve these instantly. A model needs an explicit mechanism to do it — that mechanism is self-attention. Before transformers, language models used RNNs Recurrent Neural Networks , which process text one word at a time, carrying forward a single "memory" vector: token 1 → hidden state h1 ↓ token 2 → hidden state h2 ← h2 is computed from h1 + token 2 ↓ token 3 → hidden state h3 ← h3 is computed from h2 + token 3 ↓ ... and so on That hidden state is a single fixed-size vector, overwritten at every step. Imagine reading a 300-page book but you're only allowed one sticky note — every new sentence forces you to erase your note and rewrite a fresh summary. By page 300, almost nothing from page 1 survives. "The cat that had been sitting on the mat every morning since it was a kitten and had never left the house was hungry." RNN at "was hungry": hidden state ≈ ...stuff from recent tokens... nothing from "cat"... By token 501, information from token 1 has been diluted across 500 rewrites. The connection is essentially lost. Self-attention abandons the "carry a summary forward" idea entirely. Instead, every token looks directly at every other token, simultaneously , and decides: "how relevant is each other token to understanding me right now?" "The cat sat on the" T1 T2 T3 T4 T5 When processing T2 "cat" : - How much should I look at T1 "The" ? → some - How much should I look at T3 "sat" ? → a lot cat is doing the sitting - How much should I look at T4 "on" ? → a little - How much should I look at T5 "the" ? → almost nothing Result: T2's vector gets updated with relevant information from all other tokens This is direct, parallel communication between any two tokens — regardless of distance. Token 1 and token 500 communicate with equal ease. No bottleneck, no dilution. The exact mechanics of how a token decides "how much to look at" another token is the subject of the next section. In one line:every token computes three vectors — a question Query , an advertisement Key , and a contribution Value — and attention is just "compare every Query to every Key, then blend Values by how well they matched." Section 7 gave the intuition — every token looks at every other token. This section gives the actual mechanism behind that: how does a token decide how much to borrow from each other token? SELF-ATTENTION INPUT: 5 vectors, one per token already carrying position info from section 6 "The" → 0.12, 0.87, -0.34, 0.56 "cat" → 0.23, 0.91, -0.12, 0.44 "sat" → -0.45, 0.34, 0.78, -0.23 "on" → 0.67, -0.12, 0.34, 0.89 "the" → 0.11, 0.88, -0.33, 0.54 using 4 numbers per vector here for simplicity — real models use up to 4096 SELF-ATTENTION OUTPUT: 5 vectors, one per token — same count, same size — but now each vector carries information borrowed from other tokens "The" → updated — now knows it precedes "cat sat on the mat" "cat" → updated — now knows cat is the one doing the sitting "sat" → updated — now knows who sat and where "on" → updated — knows it connects action to location "the" → updated — knows it precedes a surface/location Same number of vectors in and out, same dimensions — but richer content. The question is: how does each token decide how much to borrow from each other token? That's exactly what Q, K, V computes. Q, K, V come straight out of information retrieval. Think of a search engine: You type a search query: "best hiking trails Nepal" Each webpage has metadata: title, tags, description Each webpage has content: the actual article text QUERY = what you are searching for KEY = the metadata compared against your query VALUE = the actual content you retrieve if the match is good The search engine compares your Query against every Key, scores relevance, then returns a weighted mix of Values. Self-attention does exactly this — except every token is simultaneously a searcher and a page being searched : Every token asks a question Query : "what context do I need?" Every token advertises itself Key : "here is what I contain" Every token offers content Value : "here is what I give if selected" They aren't separate pieces of data — they're three different transformations of the same token vector , produced by multiplying it by three learned weight matrices . A weight matrix is a grid of numbers the model learns during training. Multiplying a vector by it produces a new vector viewed "through a different lens." There are three such matrices — Wq,Wk,Wv— one each for producing the Query, Key, and Value. For token "cat" with embedding vector E cat: E cat × Wq = Q cat cat's question: "what context do I need?" E cat × Wk = K cat cat's advertisement: "here is what I am about" E cat × Wv = V cat cat's offering: "here is what I contribute" This happens for every token, giving us three sets of five vectors. Step 1 — Score every token against every other token. Focus on token "cat." Its Query gets compared against every token's Key using a dot product — a single number measuring how aligned two vectors are higher = more similar . Q cat = 1.0, 0.5 K The = 0.2, 0.1 → dot product = 1.0×0.2 + 0.5×0.1 = 0.25 K cat = 0.9, 0.8 → dot product = 1.0×0.9 + 0.5×0.8 = 1.30 K sat = 0.8, 0.6 → dot product = 1.0×0.8 + 0.5×0.6 = 1.10 K on = 0.1, 0.3 → dot product = 1.0×0.1 + 0.5×0.3 = 0.25 K the = 0.2, 0.2 → dot product = 1.0×0.2 + 0.5×0.2 = 0.30 Raw attention scores for "cat": The: 0.25, cat: 1.30, sat: 1.10, on: 0.25, the: 0.30 This tracks intuition — "cat" is most relevant to itself, and second most relevant to "sat" the action it performs . Step 2 — Scale the scores. Raw scores are divided by √d k the square root of the Key vector's dimension . With dimensions in the thousands, raw dot products can get huge, which destabilizes the next step. Dividing keeps numbers in a sane range — this is the "scaled" in "scaled dot-product attention." Step 3 — Softmax converts scores into probabilities. Softmax takes a list of numbers and converts them into probabilities — all between 0 and 1, all summing to 1. Raw scores after scaling : 0.25, 1.30, 1.10, 0.25, 0.30 After softmax: 0.08, 0.42, 0.36, 0.08, 0.10 The cat sat on the Reading this: "cat" puts 42% of its attention on itself, 36% on "sat", 10% on "the", 8% each on "The" and "on" Step 4 — Weighted sum of Values. Use those probabilities to blend the Value vectors: Output for "cat" = 0.08 × V The + 0.42 × V cat + 0.36 × V sat + 0.08 × V on + 0.10 × V the The result is a new vector for "cat" that says, in effect: I am cat, mostly defined by what I am and what I do sat , with minor contributions from surrounding context. This computation happens for every token, all at once: HOW MUCH DOES EACH ROW TOKEN ATTEND TO EACH COLUMN TOKEN? "The" "cat" "sat" "on" "the" "The" 0.60 0.20 0.10 0.05 0.05 "cat" 0.08 0.42 0.36 0.08 0.10 "sat" 0.05 0.30 0.45 0.15 0.05 "on" 0.05 0.10 0.35 0.40 0.10 "the" 0.05 0.10 0.25 0.35 0.25 Each row sums to 1.0 softmax guarantees this . Each row is a different token's view of what matters. Self-attention never adds or removes tokens — 5 in, 5 out, always. What changes is the content of each vector: BEFORE self-attention: "cat" vector = 0.23, 0.91, -0.12, 0.44 only the meaning of "cat" in isolation AFTER self-attention: "cat" vector = 0.67, 1.23, 0.45, 0.89 "cat" + borrowed context from "sat" — the model now knows this cat is sitting, not sleeping Real-world analogy: five people enter a meeting room, each carrying their own notes. They talk for an hour. Five people leave — same headcount, but each person's understanding has been updated by what they heard from the others. Nothing was added or removed from the room; what changed is what's now in each person's head. One set of Q, K, V, though, can only pick up on one kind of relationship at a time — the next section explains why models run many of these side by side. In one line:instead of computing attention once per layer, the model computes it 32 or more times in parallel, each copy free to specialize in a different kind of relationship — like several editors reading the same sentence, each looking for something different. One set of Q, K, V can only learn one type of relationship at a time. But language has several relationship types happening simultaneously: "The cat sat on the mat" Relationship type 1 — grammatical subject: "cat" relates to "sat" Relationship type 2 — location: "sat" relates to "mat" Relationship type 3 — article: "The" relates to "cat" Relationship type 4 — preposition link: "on" relates to "mat" A single attention "head" tends to lock onto the most dominant relationship and miss the rest. Multi-head attention runs many independent Q/K/V computations in parallel — each with its own weight matrices, each free to specialize in a different kind of relationship. HEAD 1: Wq1, Wk1, Wv1 → learns grammatical subject-verb relations HEAD 2: Wq2, Wk2, Wv2 → learns positional/location relations HEAD 3: Wq3, Wk3, Wv3 → learns coreference what "it" refers to ... HEAD 32: Wq32, Wk32, Wv32 → learns some other pattern All 32 outputs are concatenated and projected back to the original dimension. GPT-3 uses 96 heads per layer; a typical 7-billion-parameter open model uses 32. Nobody assigns a head its job — specialization emerges from training. With 32+ heads per layer and 80 layers stacked, a model is running thousands of relationship detectors on every token, which is how it eventually learns to resolve something like the "trophy/suitcase/it" example from a handful of training examples that showed the same pattern. Worked example — two heads disagreeing on purpose: Sentence: "The trophy didn't fit in the suitcase because it was too big." HEAD 3 coreference specialist scores "it" against: "trophy" → 0.71 ← wins "suitcase" → 0.19 "big" → 0.05 learned: "it" tends to bind to the earlier physical object when the sentence structure implies containment HEAD 7 recency specialist scores "it" against: "trophy" → 0.15 "suitcase" → 0.62 ← wins it's the nearest noun "big" → 0.10 Final blended vector for "it" leans toward "trophy" because head 3's signal is stronger for THIS sentence structure — but if you swapped "big" for "small," head 3's own internal weighting would flip toward "suitcase" instead. Multiple heads vote; the training data decided which votes to trust in which context. This is also why models occasionally get coreference wrong on unusual sentence structures — if a relationship pattern was rare in training data, no head ever specialized in detecting it well. Attention one head or many is only step 5a of the transformer block from section 3's diagram. The next section covers the rest of that block — 5b through 5d — which is just as necessary for any of this to actually train. In one line:each layer is attention tokens talk to each other followed by a feed-forward step each token privately thinks alone , with a causal mask keeping generation honest and residual connections + layer norm making it possible to stack 80 of these without the network falling apart. Section 3's diagram labeled one layer as steps 5a through 5d: attention → Add & Norm → feed-forward → Add & Norm . Sections 7 through 9 only explained 5a. The other three pieces are just as load-bearing — without them, stacking 80 layers wouldn't work at all — so they get a full section here. Section 8 showed attention as every token looking at every other token. That's true for understanding text, but it would be cheating during generation . If token 3 could attend to token 7, the model would be predicting token 4 partly by looking at words that come after it — words that don't exist yet at generation time, and that the model is supposed to be predicting in the first place. The fix is a causal mask also called a look-ahead mask : before softmax runs, every attention score for a "future" token gets forced to negative infinity, so after softmax it becomes exactly 0%. Sentence: "The cat sat on the" T1 T2 T3 T4 T5 Attention scores for T3 "sat" , BEFORE masking: T1: 0.20 T2: 0.35 T3: 0.30 T4: 0.10 T5: 0.05 ↑ ↑ these are FUTURE tokens relative to T3 Attention scores for T3, AFTER causal mask applied: T1: 0.20 T2: 0.35 T3: 0.30 T4: -inf T5: -inf After softmax: T1: 29% T2: 51% T3: 20% T4: 0% T5: 0% T3 can now only ever borrow from T1, T2, and itself — never T4 or T5. Real-world analogy: reading a mystery novel with a bookmark that physically won't let you flip ahead. You can only use clues from pages you've already read — never a peek at chapter 12 while you're still on chapter 3 — which is exactly what keeps both training and generation honest. This single masking rule is why the same architecture can be trained on an entire document at once every token predicts the next token in parallel, each one only seeing what came before it and then used at inference time to generate one token at a time — the constraint is identical in both cases. It's also why this style of model is called decoder-only or causal : contrast this with an encoder like BERT , which is trained without a causal mask because it's meant to understand a whole passage at once, not generate text forward one token at a time. "Add & Norm" is actually two distinct ideas bundled into one step, and both exist to solve the same underlying problem: very deep networks are hard to train. Residual connection "Add" — instead of a layer's output replacing its input, the original input gets added back on top of whatever the layer computed: WITHOUT a residual connection: layer output = Attention x → if the attention layer computes something slightly wrong or unhelpful for a given token, that mistake fully overwrites whatever useful information was already in x WITH a residual connection: layer output = x + Attention x → even in the worst case, x itself always survives, untouched, as a "skip path" running straight through the layer Real-world analogy: imagine a report passing through 80 different editors, one after another. If each editor's markup fully replaced the previous draft, one careless editor on round 40 could wipe out everything good from rounds 1 through 39. A residual connection is like stapling the original page behind every editor's revision — each edit gets added as a note on top, but the original is always still there underneath, so nothing valuable is ever fully erased, only built upon. Stack 80 of these and the effect compounds: every layer has a direct, unobstructed path back to the original embedding. Without this, a network that deep tends to struggle to learn at all — the training signal the gradient computed from the loss function — see section 15 has to travel back through 80 sequential transformations during training, and it tends to shrink to nothing or blow up along the way. The residual path gives it a shortcut that doesn't decay, which is the main reason transformers can be stacked this deep in the first place. Layer normalization "Norm" — after the addition, every token's vector gets rescaled so its values sit in a consistent, stable range roughly mean 0, consistent spread , independent of whatever wild swings happened inside the layer: Before layer norm, a token's vector might drift to something like: 45.2, -103.7, 8.9, 210.4, ... ← values growing unpredictably after 40+ stacked layers, values like this cause training to become unstable — small input changes cause huge output swings After layer norm, the same vector is rescaled to something like: 0.31, -0.87, 0.12, 1.42, ... ← consistent, bounded range Every layer downstream now always receives input in a predictable range, regardless of what happened in the layers before it. Real-world analogy: a sound engineer riding the volume fader before every speaker steps up to the mic at a conference. Without it, one overly enthusiastic speaker could blow out the levels for everyone after them; layer norm resets things to a safe, consistent range before every new sub-layer gets its turn. Together, residuals and layer norm are why a model can be 80 layers deep instead of 4 or 5 — they're plumbing, not intelligence, but the intelligence doesn't work without them. Attention's whole job is letting tokens exchange information with each other . The feed-forward network's job is different: it processes each token's vector independently , with no communication between tokens at all, giving the model a place to actually "think about" what it just gathered from attention. Real-world analogy: attention is a group discussion where everyone talks and listens; the feed-forward step is what happens right after, when each person goes back to their own desk and privately thinks through what they just heard, updating their own notes with no more cross-talk. Attention is the meeting; the FFN is the reflection afterward. Structurally it's simple — two learned linear transformations with a nonlinear activation function in between, usually GELU: FFN x = Linear 2 GELU Linear 1 x Linear 1: expands the vector — e.g. 4096 dimensions → 16,384 dimensions GELU: a nonlinear "squashing" function applied elementwise Linear 2: compresses back down — 16,384 dimensions → 4096 dimensions Why the expand-then-compress shape? The wider middle layer gives the network more room to combine features in complex ways before compressing back to a size that fits into the next attention layer. Why the nonlinearity? Without it, stacking two linear layers back-to-back is mathematically equivalent to a single linear layer — the nonlinearity is what lets the network represent genuinely complex, non-straight-line relationships between inputs and outputs. One number worth internalizing: the FFN is where most of a model's parameters actually live — typically around two-thirds of the total, versus roughly one-third in attention the Wq/Wk/Wv/output matrices from section 8 . Attention gets most of the explanation in tutorials like this one because it's the more novel, more interesting mechanism — but by raw parameter count, the feed-forward layers are the bulk of what you're running and paying for on every forward pass. token vectors in │ ▼ ┌─────────────────────────────┐ │ 5a. Multi-head self-attention │ tokens exchange context │ causal mask applied │ sections 8–9, above └─────────────────────────────┘ │ ▼ x = x + attention output ← 5b: residual "Add" x = LayerNorm x ← "Norm" │ ▼ ┌─────────────────────────────┐ │ 5c. Feed-forward network │ each token "thinks" independently └─────────────────────────────┘ │ ▼ x = x + ffn output ← 5d: residual "Add" x = LayerNorm x ← "Norm" │ ▼ token vectors out → fed into the next layer repeat 80 times See it move.is the best match for this exact section — it visualizes a live GPT-2 block with attention, Add & Norm, and the FFN all animated side by side in the browser, running on text you type yourself. Pull it up here and watch a real vector's values change at each of the 5a–5d sub-steps above. poloclub.github.io/transformer-explainer With one full layer understood, stacking 80 of them is literally just repeating this block 80 times, each one refining the tokens' vectors a little further. Two consequences of that repetition are worth calling out before moving on to what comes after the last layer. In one line:softmax forces every token's attention to sum to 100%, so in a long prompt that 100% gets sliced so thin across thousands of competitors that anything sitting in the middle — with no positional signal marking it as special — tends to lose out to whatever sits at the very start or very end. A well-known failure mode: stuff a fact into the middle of a long prompt, and the model is far more likely to miss it than if that same fact were at the very start or very end. This is often called "lost in the middle." It isn't a bug — it's a direct, mechanical consequence of two things covered in section 8: how attention scores are computed, and how softmax forces them to compete. It's not that the middle is invisible. Attention technically lets every token look at every other token. The real cause is competition inside softmax . SHORT CONTEXT — 5 tokens: Raw scores: 0.25, 1.30, 1.10, 0.25, 0.30 After softmax: 0.08, 0.42, 0.36, 0.08, 0.10 Smallest slice is 0.08 — still meaningful. LONG CONTEXT — 10,000 tokens: Softmax's 1.0 must now be sliced across 10,000 competitors. Average slice ≈ 0.0001. The model has no advance knowledge of which middle token matters — it has to discover relevance purely through the dot-product comparison. Meanwhile: BEGINNING tokens: system prompt lives here → trained to be high priority END tokens: the current user question → recency is a strong signal MIDDLE tokens: no positional signal marking them as important — must win purely on content, diluted among thousands of competing tokens Why training makes this worse: the model learned from internet text, books, and code, where structure almost always looks like: Important instruction / title / topic sentence ... body content ... Conclusion / answer / punchline Over billions of examples, the weight matrices developed a bias: beginning and end positions produce stronger Key vectors, because that's statistically where important information lives in human-written text. Middle positions default to weaker Keys — not because they're unseen, but because the model learned they're usually supporting detail. Concrete failure case: System prompt — 200 tokens Retrieved document 1 — 800 tokens Retrieved document 2 — 800 tokens ← CRITICAL ANSWER IS HERE Retrieved document 3 — 800 tokens Retrieved document 4 — 800 tokens User question — 50 tokens The critical answer sits around position 1,000-1,800 out of ~3,450 tokens, with no positional signal marking it as special. Its Key vector has to outcompete thousands of others. The model often misses it, partially uses it, or under-weights it — even though it's technically "visible." What this means practically: WRONG — burying critical instructions in the middle: System prompt: general behavior rules Long background context Important constraint: never discuss pricing ← often missed More background context User question RIGHT — critical information at the boundaries: System prompt: general behavior rules Important constraint: never discuss pricing ← at the top Background context User question with key constraint repeated ← at the end This is exactly why production retrieval systems RAG rerank retrieved chunks and place the most relevant ones closest to the user's question, instead of just dumping everything in document order. With all 80 layers now explained end to end, the next question is what happens the moment the last layer finishes — how a refined vector actually turns into a word. In one line:the last token's final vector gets turned into a probability for every word in the vocabulary, one word gets picked, it's appended to the sequence, and the entire 80-layer pass runs again — this loop, one word at a time, is all "generation" ever is. Every layer from sections 7 through 11 refines a token's vector using context from the rest of the sequence. This section covers what happens after the last one of those 80 layers — steps 6 through 11 of the pipeline diagram in section 3 — where a refined vector finally turns into an actual word. THE GENERATION LOOP: You give: "The cat sat on the" │ ▼ full forward pass │ ▼ model produces a probability distribution over all words │ ▼ one word is selected: "mat" │ ▼ sequence is now: "The cat sat on the mat" │ ▼ full forward pass again │ ▼ one word is selected: "." │ ▼ continues until stop condition This loop — where the model's own output becomes its next input — is what autoregressive means. Auto = self. Regressive = predicting from previous values. After the final transformer block, only the last token's vector matters for generation — that's the one the next word gets predicted from. Input: "The cat sat on the" ↑ last token: "the" After 80 transformer blocks: "the" vector = 0.45, -0.23, 0.87, ..., 0.12 4096 numbers This vector passes through one final weight matrix sized 4096 × 100,000 that maps it to 100,000 raw scores — one per vocabulary word. These raw, unnormalized scores are called logits : "mat" → 8.42 high — strongly predicted "floor" → 6.31 "wall" → 5.10 "table" → 4.88 "ground" → 4.20 ... "democracy" → -2.30 very low — contextually absurd Softmax converts these 100,000 logits into probabilities summing to 1.0: "mat" → 34.2% "floor" → 18.1% "wall" → 9.3% "table" → 8.7% "ground" → 6.5% "democracy" → 0.0% The model isn't asserting "the answer is mat." It's saying: given everything I've seen, here's my probability estimate for every possible next word. A sorting algorithm always gives the same output for the same input. An LLM does not — a word is sampled from the distribution, like a weighted dice roll: "mat" ████████████████ 34.2% "floor" █████████ 18.1% "wall" ████ 9.3% "table" ████ 8.7% "ground" ███ 6.5% everything █████████████ 23.2% spread across 99,995 other tokens else Run 1: lands on "mat" Run 2: lands on "mat" Run 3: lands on "floor" Run 4: lands on "mat" Run 5: lands on "wall" This is why the same prompt can produce different outputs on different runs. It's deliberate: if the model always picked the single highest-probability word greedy decoding , every run of "write me a poem about mountains" would produce the exact same, statistically-average poem. Sampling turns the model's uncertainty into a creative feature. Greedy decoding — always pick the highest-probability token. Deterministic, fast, but can get stuck in repetitive loops because the locally-best word at each step doesn't guarantee the best overall sentence. Sampling — pick probabilistically from the distribution at each step. Non-deterministic, more varied, more natural. In production this is almost always paired with temperature and top-p controls see section 14 https://gist.github.com/starred.atom 14-temperature--sampling--controlling-randomness . ITERATION 1: ├── forward pass through all 80 transformer blocks ├── output layer produces 100,000 logits for the last token ├── softmax → probability distribution ├── sample → "mat" └── sequence becomes: "The cat sat on the mat" ITERATION 2: ├── forward pass now "mat" attends to all previous tokens too ├── new logits → new distribution ├── sample → "." └── sequence becomes: "The cat sat on the mat." ITERATION 3: ├── forward pass ├── logits show high probability for the end-of-sequence token ├── sample → <EOS └── generation stops Generation continues until one of three stop conditions : the model samples a special end-of-sequence token, you hit a max tokens limit, or a custom stop sequence appears in the output. Why this matters practically: Non-determinism is a first-class concern. You can't unit-test an LLM output with assertEqual . Testing strategy has to account for probabilistic output. Streaming exists because of this loop. Each token is available the instant it's sampled — the model doesn't wait to finish the whole response before sending anything.Every iteration is a full forward pass through every layer. Generating 1,000 tokens means 1,000 full forward passes; capping max tokens is a real cost lever. max tokens at 100 instead of 1,000 is a genuine 10x reduction in output-side compute. This loop works, but it repeats a lot of work every single iteration — the next section shows the one optimization that makes it practical to run at all. In one line:a past token's Key and Value vectors never change once computed, so instead of recomputing them on every loop iteration, the model stores them once and just adds the new token's — this is the single biggest efficiency trick in real-world LLM serving. Generation is a loop section 12 . Without any optimization, every single iteration recomputes Query, Key, and Value vectors for every token generated so far — including ones already processed in earlier iterations: ITERATION 1 — generate "mat": Input: The, cat, sat, on, the Compute Q, K, V for ALL 5 tokens ITERATION 2 — generate ".": Input: The, cat, sat, on, the, mat Compute Q, K, V for ALL 6 tokens → recomputes K, V for the first 5 — pure waste, they haven't changed ITERATION 3 — generate "The": Input: The, cat, sat, on, the, mat, . Compute Q, K, V for ALL 7 tokens → recomputes K, V for the first 6, again A past token's Key and Value never change once computed — "cat" at position 2 always produces the same K and V no matter what gets generated afterward. Recomputing them every time is wasted work that compounds badly over long generations. The KV cache stores every past token's K and V vectors at every layer . Each new iteration only computes K, V, and Q for the newest token, and reuses everything else from cache. Why cache K and V but not Q? Q → "what am I looking for?" Only the current token being generated asks a question. Past tokens' questions were already used — no need to keep them. K → "what do I contain?" advertised to everyone Every token, including past ones, must remain searchable by future tokens' queries. Must be cached. V → "what do I contribute when selected?" Every token must be able to contribute its content. Must be cached. ITERATION 2 — generate ".": Compute Q, K, V for "mat" only ← just the new token Retrieve all past K, V from cache Use "mat"'s Q against cached K's + its own K → generate "." Store K mat, V mat into the cache for future iterations Each iteration now only does the work of one new token — everything from the past is free. Large context windows need enormous GPU memory , because the cache has to store K and V for every token, at every layer, for every attention head: Cache size per token = 2 K and V × num layers × num heads × head dimension × 2 bytes float16 For a 70B model: ~2.5 MB per token in KV cache For a 128k-token context: 128,000 × 2.5 MB = 320 GB of cache alone This is a large part of why serving long-context requests requires multiple high-end GPUs just for the cache — not extra compute, extra memory. Prefill vs. decode are the two performance phases of generation: PREFILL — your entire input prompt processed in one parallel pass. Fast, benefits fully from GPU parallelism. Determines time-to-first-token TTFT . DECODE — the autoregressive loop, one token per step, strictly sequential. KV cache grows by one entry per step. Slower — determines total generation time. This is why an API call often feels fast at first, then slows as the response grows: prefill finishes quickly, decode runs sequentially after. Why output tokens cost more than input tokens: Input tokens → processed during prefill parallel, fast, cheaper Output tokens → generated during decode sequential, per-step, pricier Claude 3.5 Sonnet input: $3 per million tokens Claude 3.5 Sonnet output: $15 per million tokens 5x more Prompt caching is a direct product of how the KV cache works: if the start of your prompt e.g. a long system prompt is identical across requests, the provider can store and reuse that portion's KV cache across calls, charging a much lower rate on a cache hit — often around 10% of the normal price for that segment. The loop itself, and the cache that makes it efficient, are now fully covered. The one control still missing is how random each sampled word should be — that's temperature, next. In one line:temperature is one division applied to the logits before softmax — divide by a number less than 1 to make the model more confident and repetitive, divide by a number greater than 1 to make it more random and varied. Section 12 introduced logits and softmax as steps 6 and 8 of the pipeline. Temperature step 7 sits in between them, and it's the main dial you actually get to turn: Forward pass │ ▼ 100,000 raw logits ← pure model output, no control yet │ ▼ DIVIDE BY TEMPERATURE ← this is where you intervene │ ▼ 100,000 adjusted logits │ ▼ SOFTMAX ← converts to probabilities │ ▼ TOP-P FILTER ← optional second control │ ▼ SAMPLE ← pick one token Temperature T is a single number. Every logit gets divided by it before softmax: adjusted logit = original logit / T Using five example tokens with original logits mat=8.42, floor=6.31, wall=5.10, table=4.88, other=3.00 : T = 1.0 default — unchanged : After softmax: "mat" 72.1% ████████████████████████████ "floor" 16.4% ██████ "wall" 5.8% ██ "table" 4.7% █ "other" 1.0% ▌ T = 0.5 low — sharpens the distribution : dividing by a number less than 1 amplifies the differences between logits. After softmax: "mat" 95.1% ████████████████████████████████████████ "floor" 3.9% █ "wall" 0.6% ▌ "table" 0.4% ▌ "other" 0.0% → nearly certain, very predictable T = 2.0 high — flattens the distribution : dividing by a number greater than 1 compresses the differences. After softmax: "mat" 41.2% ████████████████ "floor" 24.1% █████████ "wall" 16.2% ██████ "table" 14.8% █████ "other" 3.7% █ → much more variety, less predictable T = 0 special case — greedy : dividing by zero is undefined, so in practice this is implemented as argmax — always pick the single highest logit. Fully deterministic; same input always gives same output. FLAT SHARP ◄─────────────────────────────────────────────────► T = 2.0 T = 1.0 T = 0.5 T = 0.1 T = 0 random balanced confident near-certain deterministic creative natural focused repetitive always same It's worth being precise here, because it explains why top-p works at all. Softmax does not just divide by the count of tokens — it uses exponentials : P token i = exp logit i / sum of exp every logit exp x means e raised to the power x e ≈ 2.718 — and exponential growth is explosive. Small logit differences become enormous probability differences: Step 1 — raw logits: Step 2 — apply exp : "mat" 8.42 exp 8.42 = 4,526 "floor" 6.31 exp 6.31 = 550 "wall" 5.10 exp 5.10 = 164 "table" 4.88 exp 4.88 = 132 "other" 3.00 exp 3.00 = 20 Step 3 — sum = 5,392 Step 4 — divide each by the sum: "mat" 4,526 / 5,392 = 83.9% "floor" 550 / 5,392 = 10.2% "wall" 164 / 5,392 = 3.0% "table" 132 / 5,392 = 2.4% "other" 20 / 5,392 = 0.4% A logit gap of just 5.42 between "mat" and "other" becomes a 226x difference in probability after exponentiation. Across a real 100,000-token vocabulary, this means the top handful of tokens typically absorb 90%+ of the total probability mass before any filtering even happens — which is exactly why top-p, described next, works so well. Even with good temperature, the distribution has a long tail of thousands of tokens with tiny non-zero probabilities. Top-p cuts that tail off before sampling: sort tokens by probability, keep adding them to a "nucleus" until their cumulative probability reaches p , then sample only from that nucleus. p = 0.90: Token Probability Cumulative "mat" 34.2% 34.2% ← include "floor" 18.1% 52.3% ← include "wall" 9.3% 61.6% ← include "table" 8.7% 70.3% ← include "ground" 6.5% 76.8% ← include "roof" 3.2% 80.0% ← include "rug" 2.8% 82.8% ← include "bed" 2.1% 84.9% ← include "couch" 1.8% 86.7% ← include "chair" 1.5% 88.2% ← include "desk" 1.2% 89.4% ← include "box" 0.9% 90.3% ← include crosses threshold here ───────────────────────────────────────── everything else ~99,988 tokens : EXCLUDED — cut off entirely Top-k always keeps a fixed number of tokens, regardless of the shape of the distribution — which causes problems in both directions: CONTEXT A — model is very certain top-k=5 : "mat" 94.0%, "floor" 3.0%, "wall" 1.5%, "table" 0.9%, "ground" 0.3% → top-k=5 forces in that 0.3% token — basically noise CONTEXT B — model is genuinely uncertain top-k=5 : "mat" 12%, "floor" 11%, "wall" 10%, "table" 9%, "ground" 9%, "roof" 8% ← cut, "rug" 7% ← cut, "bed" 7% ← cut → top-k=5 is cutting off perfectly valid options Top-p adapts automatically to the shape of the distribution instead: CONTEXT A p=0.90 : "mat" alone is already 94% → nucleus = 1 token CONTEXT B p=0.90 : takes 12 tokens to reach 90% → nucleus = 12 tokens top-k → fixed count, blind to distribution shape top-p → variable count, adapts to distribution shape Top-p is the strictly better default in modern production systems. If a provider only exposes top-k, set it high 50+ to minimize the damage. USE CASE TEMPERATURE TOP-P ────────────────────────────────────────────────────── Structured data extraction 0 — Classification / routing 0 — Factual Q&A one right answer 0 – 0.2 — Code generation 0.1 – 0.3 0.95 Summarization 0.3 – 0.5 0.95 Conversational assistant 0.7 – 1.0 0.95 Creative writing 1.0 – 1.5 0.95 Brainstorming / ideation 1.0 – 1.5 0.95 ────────────────────────────────────────────────────── A common mistake: leaving default temperature often 1.0 on a structured-extraction task, then getting inconsistent JSON and random formatting. Temperature 0 plus schema enforcement is the right combination when consistency matters. Also note: temperature 0 doesn't guarantee byte-identical output across different providers — floating point implementations vary slightly — so true determinism usually still needs schema validation on top. That completes the inference side of this guide — everything a trained model does when you run it, from raw text in to a sampled word out. The next section switches focus entirely: how did those billions of weights get set in the first place? In one line:the model makes a prediction, gets scored on how wrong it was, and every single one of its billions of numbers gets nudged a tiny bit in whichever direction would have made that prediction less wrong — repeated trillions of times, that's the entire mechanism behind everything a model "knows." Every section so far described what a trained model does when you run it. This section is the missing piece: how do billions of random numbers turn into weights that actually produce good predictions in the first place? The next section hallucination leans on this directly, so it's worth having the mechanism in hand first. TRAINING LOOP repeated over trillions of tokens : 1. FORWARD PASS training text → tokenize → embed → 80 transformer layers → logits → softmax → a probability distribution over the next token exactly the same mechanics as inference — sections 4 through 12 2. COMPUTE LOSS compare the predicted distribution against the token that actually appeared next in the real training text loss = how wrong / "surprised" the model was 3. BACKWARD PASS backpropagation compute the gradient of the loss with respect to EVERY parameter — flowing backward from the output layer, through every transformer block, all the way back to the embedding table "if this specific number were slightly bigger or smaller, would the loss go down?" — answered for billions of numbers at once 4. UPDATE WEIGHTS gradient descent nudge every parameter a small step in the direction that reduces loss new weight = old weight − learning rate × gradient → repeat with the next training example, forever Every parameter in the model — every number inside Wq, Wk, Wv from section 8, every number inside the feed-forward matrices from section 10, every row of the embedding table from section 5 — starts as random noise and is shaped entirely by this four-step loop, repeated across trillions of tokens. The standard loss for next-token prediction is cross-entropy loss , and it has a simple form: loss = -log probability the model assigned to the correct token . Using this guide's running example — predicting "mat" after "The cat sat on the": If the model assigned "mat" a probability of 34.2% as in section 12 : loss = -log 0.342 ≈ 1.07 If the model had instead been well-calibrated and assigned 90%: loss = -log 0.90 ≈ 0.105 ← much lower loss, closer to ideal If the model had confidently assigned just 1% to the correct answer: loss = -log 0.01 ≈ 4.6 ← punished heavily for being confidently wrong Low loss means the model put high probability on the token that actually came next. High loss means it didn't — and the size of the loss is exactly what step 3 needs in order to know how hard to push, and in which direction, on every single parameter. Backpropagation uses the chain rule from calculus to answer one question for every parameter in the model, all at once: if I nudged this specific number, how much would the loss change? That answer is called the parameter's gradient . The computation runs in the opposite direction of the forward pass — starting from the loss at the output layer, and working backward through the output projection, back through transformer block 80, block 79, ... all the way to block 1, and finally back into the embedding table itself: FORWARD: embeddings → block 1 → block 2 → ... → block 80 → logits → loss → → → → BACKWARD: embeddings ← block 1 ← block 2 ← ... ← block 80 ← logits ← loss gradient flows this direction, layer by layer Real-world analogy: hiking down a mountain in thick fog. You can't see the valley floor, but you can feel the slope of the ground under your feet in every direction — that's the gradient. You take a small step downhill, re-check the slope, take another step, and repeat. Millions of small, locally-informed steps eventually walk you down to a low point, even though you never had a map of the whole mountain. Once every parameter's gradient is known, each one gets nudged a small step in the direction that reduces loss: new weight = old weight − learning rate × gradient The learning rate controls how big that step is. Too large, and updates overshoot and destabilize training. Too small, and training crawls and never converges in a reasonable amount of time — this is one of the most important numbers a training run has to get right. Section 10 explained that residual connections give the training signal a shortcut back to the original input, and this is exactly the training signal it meant: the gradient computed here has to travel backward through all 80 stacked layers before it can update the earliest ones. Without a residual shortcut, that gradient tends to shrink toward zero or blow up somewhere along the way — a problem serious enough that, before residual connections became standard, networks this deep simply couldn't be trained effectively at all. The word batch shows up in two different training/inference contexts and it's easy to conflate them: TRAINING BATCH: a group of training examples whose losses get averaged together before ONE gradient update is applied — done for training stability, not speed INFERENCE BATCH section 20 : a group of concurrent user requests processed together in one forward pass, purely to use GPU parallelism efficiently — no gradients, no loss, no weight updates involved at all Same word, two unrelated jobs — one is a training-stability concept, the other is a serving-efficiency concept. This is why pre-training is so expensive and fine-tuning is comparatively cheap. Fine-tuning section 17 is the exact same four-step loop — forward pass, loss, backward pass, update — just run on a much smaller, curated dataset starting from already-good weights instead of random noise. Same mechanism, far fewer iterations needed. This is why training needs dramatically more compute and memory than inference. Training runs a backward pass roughly 2x the compute of the forward pass alone for every single example, and has to hold onto intermediate values from the forward pass in memory so the backward pass can use them. Inference section 20 only ever runs the forward pass — no gradients, no backward pass, no loss — which is a large part of why serving a model is so much cheaper than training one. "The model learned X" always cashes out to this loop. Whenever this guide or anyone says a model "learned" a pattern, a bias, or a capability, this four-step loop — applied enough times, on enough examples showing that pattern — is the entire mechanism behind it. There is no other channel through which a transformer acquires anything. With training itself explained, the next three sections cover what training actually produces: a model that's fluent but not always truthful 16 , a process for turning that raw model into an assistant 17 , and what happens as all of this gets scaled up 18 . In one line:the model was trained to predict statistically plausible text, never to be truthful — on accurate training data those two mostly line up, but "mostly" is exactly where confident, fluent, completely fabricated answers come from. Section 15 explained that a model is trained purely to minimize a loss based on plausibility — how surprised it was by the next real token. This section follows that fact to its most consequential conclusion. WHAT MOST PEOPLE THINK THE MODEL DOES: "Who founded Apple?" → searches internal knowledge → finds fact: "Steve Jobs, Steve Wozniak, Ronald Wayne — 1976" → returns it WHAT THE MODEL ACTUALLY DOES: "Who founded Apple?" → "given these tokens, what tokens are most likely to come next?" → generates a statistically plausible continuation → returns that continuation There's no search, no lookup, no fact-retrieval step. The model generates what's statistically likely to follow — and it's usually correct simply because it trained on mostly-accurate text. Correctness is a side effect of the training data, not the objective the model was optimized for. The model was trained on one task at massive scale: given "The cat sat on the ", predict the next token. The loss function from section 15 — what the model was actually trying to minimize — measured only this: LOSS = how surprised was the model by the actual next token? Low surprise = high probability assigned to the correct token = good High surprise = low probability assigned to the correct token = bad Truth never appears in that objective. On a dataset of mostly-accurate human writing, predicting accurately and predicting plausibly mostly line up — but they are not the same thing, and the gap between them is exactly where hallucination lives. The model has no database, no index, no filesystem. All of it is compressed into parametric memory — billions of weight values adjusted during training section 15 . When the model "knows" Paris is the capital of France, it does NOT have a record like { country: "France", capital: "Paris" }. It has weight values that, when processing "What is the capital of France?", produce a high logit for "Paris" and low logits for everything else. The knowledge IS the pattern of weights — it has no other form. Critically, this means the model has no way to distinguish "things I know well" from "things I barely saw in training." Both are produced by the exact same generation mechanism. A database query either returns a row or returns null — a clean signal for "I don't have this." The generation mechanism has no equivalent. It always produces something : "What is 2 + 2?" → high confidence → "4" → correct "What did an obscure private individual eat for breakfast on March 3rd 2019?" → never seen this in training → generation mechanism still runs anyway → produces a plausible-sounding, completely fabricated answer → delivered with the exact same fluency as the correct answer above Why doesn't it just say "I don't know"? It can — the phrase exists in its vocabulary and training data — but the model also learned that, statistically, a confident answer is far more likely to follow a question than an admission of uncertainty. Wikipedia doesn't say "I don't know." Textbooks don't hedge. Instruction tuning and RLHF next section substantially improve calibrated uncertainty, but the underlying generation mechanism still has no native uncertainty detector — expressing doubt is itself a learned pattern, not a genuine internal check. 1. Factual confabulation — a fact was rare or absent in training data, so the model fills the gap with a plausible-sounding invention. "What papers did researcher X publish in 2019?" X is real but obscure → a list of realistic-looking paper titles, journals, co-authors — none of which exist. The "published papers" format is a strong pattern; the model fills it with statistically plausible content. 2. Knowledge cutoff confusion — confidently stating something that was true when training data was collected, but has since changed. "Who is the CEO of Twitter?" → depending on training cutoff, might answer Jack Dorsey, Parag Agrawal, or Elon Musk — with no awareness that time has passed since training ended. 3. Reasoning errors stated confidently — a wrong intermediate step in a chain of reasoning produces a plausible-looking continuation that compounds. "3 boxes with 4 apples each, give away half, then buy 7 more — how many apples?" Correct: 3×4=12, 12/2=6, 6+7=13 The model might land on 13 correct or on a wrong number like 15, stated with identical confidence either way — each token was locally plausible even if the overall chain went wrong. Chain-of-thought prompting having the model show its work step by step substantially reduces this category by giving it room to self-correct before committing to a final answer. 4. Source fabrication — asked for citations, the model produces realistic-looking but nonexistent references. "Cite three academic papers about transformer attention" → "Smith et al. 2021 . Attention Mechanisms in Modern NLP. Journal of Machine Learning Research, 22 4 , 112-134." Completely fabricated — but every field author, year, journal, volume, pages matches the format of a real citation perfectly, because that format is what the model actually learned. The architectural fix for hallucination is RAG Retrieval-Augmented Generation — instead of asking the model to recall facts from its parametric memory, you retrieve the actual relevant facts and place them directly in the prompt: WITHOUT RAG: "What is our refund policy?" → model generates a plausible-sounding policy from general training patterns — may be entirely wrong for your specific business WITH RAG: retrieve: actual refund policy text from your own database "Based on this policy: text , what is our refund policy?" → model summarizes the text you handed it → grounded in your real document, hallucination risk drops sharply Also worth internalizing: the model writes "the answer is X" with identical grammatical confidence whether X is correct or fabricated. Linguistic confidence is not a reliability signal — any system where factual accuracy actually matters legal, medical, financial needs grounding or independent verification, not just careful prompting. A raw, hallucination-prone next-token predictor is also nearly unusable as an assistant for a second reason: it doesn't even know it's supposed to act like one. The next section covers the training stages that fix that. In one line:a base model only knows how to continue text plausibly; two more rounds of training — supervised examples of good answers, then reinforcement learning from ranked preferences — are what turn it into something that behaves like ChatGPT or Claude. The model described so far — trained purely to predict the next token section 15 , and prone to hallucination as a direct result section 16 — is almost unusable as an assistant on its own. Here's the three-stage pipeline that turns it into something like ChatGPT or Claude. STAGE 1: PRE-TRAINING ───────────────────────────────────────────────── Objective: predict next token Data: entire internet + books + code trillions of tokens Duration: months, thousands of GPUs Cost: tens to hundreds of millions of dollars Result: BASE MODEL — extraordinarily capable but completely unaligned. Ask it a question and it might just continue your question rather than answer it. STAGE 2: SUPERVISED FINE-TUNING SFT ───────────────────────────────────────────────── Objective: teach the model the format of being an assistant Data: tens of thousands of human-written good question → good answer pairs Duration: days Cost: thousands of dollars Result: INSTRUCTION-TUNED MODEL — knows it should answer questions directly, but isn't yet well-calibrated on what "good" means across edge cases. STAGE 3: RLHF ───────────────────────────────────────────────── Objective: align behavior with human preferences Data: human rankings of multiple candidate model outputs Duration: weeks Cost: millions of dollars Result: CHAT MODEL — refuses harmful requests, expresses uncertainty appropriately, behaves the way the lab intended. You give the base model: "What is the capital of France?" It might output: "What is the capital of France? This is a question commonly asked in geography quizzes. The answer varies depending on..." It just continues the text like a document — it has no concept of "being an assistant" yet. Fine-tuning means continuing to train an already-trained model on new data — not training from scratch. It's the exact same forward-pass/loss/backward-pass/update loop from section 15, just run on a much smaller, curated dataset. The model keeps everything from pre-training and additionally learns new patterns from that dataset: Example 1: HUMAN: "What is the capital of France?" ASSISTANT: "Paris is the capital of France." Example 2: HUMAN: "Write a Python function to reverse a string." ASSISTANT: "Here is a Python function that reverses a string: def reverse string s : return s ::-1 " Example 3: HUMAN: "How do I make a bomb?" ASSISTANT: "I can't help with that." Tens of thousands of examples like these, written by human contractors, trained with the same next-token objective as pre-training — except now the "correct" next tokens are the human-written assistant responses. SFT teaches format and style : questions get answered directly, harmful requests get declined, instructions get followed. It doesn't yet teach the model to reliably tell a good answer from a mediocre one across the huge variety of real user requests — that's RLHF's job. Reinforcement learning is a training paradigm where a model the "policy" takes actions, gets a reward signal for how good those actions were, and learns to maximize that reward over time — as opposed to supervised learning, where you just show correct examples directly. RLHF runs in two sub-stages: A — Train a reward model. You can't have humans rate every output during training too slow, too expensive , so you first build an automated stand-in for human judgment: Step 1: Generate multiple responses to the same prompt: "Explain what a transformer is." A: clear, accurate, appropriately detailed B: vague, technically true but not actually explanatory C: about electrical transformers — wrong context entirely Step 2: Human raters rank them: A B C Step 3: Train a separate reward model on thousands of these prompt, response A, response B, ranking triplets. Step 4: The trained reward model can now score any response without a human in the loop: Response A → 0.91, Response B → 0.54, Response C → 0.12 B — RL training against the reward model: LOOP: 1. Give the SFT model a prompt 2. It generates a response 3. The reward model scores the response 4. Update the model's weights to produce higher-scoring responses 5. Repeat millions of times A KL divergence penalty keeps the model from drifting too far from the original SFT model during this process — without it, the model can find ways to get high reward scores that have nothing to do with what humans actually want reward hacking : Example: if the reward model was trained on data where longer, more detailed responses scored higher, an unconstrained model might learn to pad every answer with filler text to farm reward — great score, terrible for users. The KL penalty caps how far it can drift. Standard RLHF depends on human raters, who are expensive, inconsistent, and don't scale easily. Anthropic's alternative, Constitutional AI CAI — a form of RLAIF , Reinforcement Learning from AI Feedback — replaces much of the human ranking step with AI self-critique against an explicit set of written principles: 1. Define a "constitution": principles like "be helpful, harmless, and honest," "prefer responses that don't assist illegal activity," "choose the response least likely to contain misinformation." 2. Have the model critique its own candidate responses against these principles: "Which of these two responses better follows the principle of being helpful without causing harm?" 3. Train the reward model on these AI-generated rankings, instead of or alongside human rankings — which scales far more cheaply and consistently. This is why Claude has a distinct, consistent character — it was deliberately shaped by the specific principles written into its constitutional training. Direct Preference Optimization DPO achieves a similar result to RLHF without training a separate reward model at all — human preference data response A preferred over response B is used to update the model's weights directly, in one step: RLHF: SFT model → reward model → RL training loop → chat model three stages, more complex, can be unstable DPO: SFT model → direct preference training → chat model two stages, simpler, more reproducible Many recent open models Llama 3, Mistral use DPO instead of full RLHF. Base models and chat models are different products. Never use a raw base model in production directly — always use the instruction-tuned / RLHF or DPO variant. Model behavior can shift silently between versions. RLHF training isn't perfectly reproducible, so provider model updates can quietly change refusal thresholds, verbosity, or formatting. Pinning a specific model version e.g. claude-3-5-sonnet-20241022 rather than -latest protects a prompt that already works from breaking under you. Refusals are learned, not hardcoded. They come from the reward model assigning low scores to harmful responses across training — which means they're probabilistic. The same underlying request phrased differently can get a different outcome. Don't treat model refusal as your only safety layer in a real system. System prompts work because SFT taught the model to treat them specially — the fine-tuning data explicitly included examples of following system-prompt instructions, so the behavior is learned, not a special hardcoded mode. Everything so far has been about a single model of fixed size. The next section covers what changes — often surprisingly abruptly — as that size scales up. In one line:bigger models don't just get incrementally better at the same things — past certain parameter-count thresholds, entirely new capabilities multi-step reasoning, few-shot learning switch on that smaller models literally cannot do at all. GPT-2 2019 : 1.5 billion parameters → decent text completion GPT-3 2020 : 175 billion parameters → surprisingly good reasoning GPT-4 2023 : ~1 trillion parameters → passes bar exam, complex math The jump from GPT-2 to GPT-3 wasn't just "better at the same things." GPT-3 could do things GPT-2 literally could not do at all — few-shot learning, chain-of-thought reasoning, complex coding. These capabilities didn't improve gradually. They appeared suddenly past a scale threshold. That's emergence. A parameter is a single number inside one of the model's weight matrices like the Wq, Wk, Wv from section 8 , adjusted during training section 15 . A small weight matrix 4×4, for illustration : Wq = 0.23 -0.45 0.12 0.87 0.56 0.34 -0.23 0.45 -0.12 0.78 0.56 -0.34 0.89 -0.12 0.34 0.23 16 parameters in this one small matrix. A real model has thousands of matrices, each with millions of parameters — billions in total. Every parameter is a number nudged countless times during training to reduce prediction error. The entire collection of parameter values is what the model "knows" — its parametric memory section 16 . Model Parameters Approx GPU memory to run ────────────────────────────────────────────────────────────── Llama 3.2 1B 1 billion ~2 GB ← runs on a phone Llama 3.2 3B 3 billion ~6 GB ← runs on a laptop Llama 3.1 8B 8 billion ~16 GB ← runs on 1 consumer GPU Llama 3.1 70B 70 billion ~140 GB ← needs 2-4 datacenter GPUs Llama 3.1 405B 405 billion ~810 GB ← needs 8-16 datacenter GPUs Memory scales roughly linearly with parameter count, since each parameter is typically stored as 2 bytes float16 : 7B × 2 bytes = 14 GB , 70B × 2 bytes = 140 GB . This is exactly why you can't run GPT-4 on a laptop, and why API access exists at all. KNOB 1 — Depth number of layers : GPT-2: 48 layers GPT-3: 96 layers GPT-4: 120+ estimated KNOB 2 — Width model dimension, size of each token's vector : GPT-2: 1,600 numbers per vector GPT-3: 12,288 numbers per vector KNOB 3 — Number of attention heads: GPT-2: 25 heads per layer GPT-3: 96 heads per layer Scaling laws are empirically discovered relationships between model size, training data size, compute budget, and resulting performance. The key one, the Chinchilla scaling law DeepMind, 2022 : For a given compute budget: model parameters ≈ training tokens × 20 10 billion parameter model → needs ~200 billion training tokens 70 billion parameter model → needs ~1.4 trillion training tokens The original GPT-3 175B parameters was trained on only 300B tokens → significantly undertrained by this standard → a 70B model trained on 1.4T tokens can outperform it → this is why Llama 2 70B is competitive with early GPT-3 despite having fewer than half the parameters The practical lesson: a smaller model trained on more data can beat a bigger model trained on less. Parameter count alone doesn't predict capability — the ratio of parameters to training tokens does. Emergence : a capability that appears suddenly at a scale threshold, rather than improving gradually. Below the threshold, the model can't do it at all; above it, it can — closer to a phase transition water turning to ice at 0°C than a smooth curve. CAPABILITY APPEARS AROUND ────────────────────────────────────────────────────────── Few-shot learning learn from 10B+ parameters examples in the prompt, no retraining Chain-of-thought reasoning 60B+ parameters Instruction following without examples 60B+ parameters Code generation 60B+ parameters Calibrated uncertainty "I'm not sure" 100B+ parameters when genuinely uncertain Theory of mind basics reasoning about 175B+ parameters what others know or believe Why does this happen? Researchers don't have a fully settled answer. Two leading hypotheses, likely both partially true: HYPOTHESIS 1 — Capability composition: A complex capability like chain-of-thought reasoning actually requires several sub-skills following multi-step instructions, holding intermediate results, recognizing a completed step, moving on correctly . Each sub-skill improves gradually with scale. If all of them individually cross their own threshold around the same parameter count, the composite capability appears to "switch on" suddenly — but it's really the last sub-skill crossing its line. HYPOTHESIS 2 — Metric non-linearity: The benchmark itself may be scored as strictly right/wrong, while the underlying capability actually improves smoothly underneath. A 55B model might be "40% of the way" to solving a math problem correctly, but a binary correct/incorrect metric records that as a flat 0. At 65B it crosses into fully correct — the metric jumps from 0 to 1, which looks like emergence but may just be a measurement artifact on top of gradual improvement. Mixture of Experts : instead of every token passing through every parameter a "dense" model , the model has many specialized sub-networks "experts" , and a router activates only a small subset of them per token. DENSE MODEL: every token → passes through ALL parameters → expensive MOE MODEL: every token → router picks 2-4 relevant experts → token only passes through those → rest sit idle for it GPT-4 is believed to use MoE: Estimated total parameters: ~1.8 trillion Estimated active parameters: ~220 billion per token Meaning: GPT-4 has the knowledge of a 1.8T-parameter model but roughly the inference cost of a ~220B-parameter model, because most parameters are inactive for any given token. This is why bigger total parameter count doesn't automatically mean a proportionally bigger API bill — cost tracks active parameters, not total ones. Model selection is a capability question before it's a cost question. If a task genuinely needs chain-of-thought reasoning, a 7B model will fail no matter how cleverly you prompt it — the capability doesn't exist yet at that scale. If the task is simple classification, using a frontier model is just waste. Smaller, better-trained models keep closing the gap — reassess model tier choices periodically as training efficiency improves e.g. Llama 3.1 8B, trained on 15 trillion tokens, beats GPT-3 175B on many benchmarks . Public benchmark scores aren't your task's score. A model scoring 90% on a general benchmark can still fail badly on a narrow domain that was underrepresented in its training data — always evaluate against your actual use case. Everything up to this point has been about the LLM itself. The last two sections zoom out to two things built around the model in production: the embedding vectors that power search and retrieval, and the economics of actually serving all of this. In one line:the same "meaning becomes geometry" trick from section 2 also works at the sentence and document level, which is the entire foundation of semantic search, RAG, and every vector database. There are two distinct uses of the word "embedding" in this space, and conflating them is a common source of confusion: USE 1 — TOKEN EMBEDDINGS inside the LLM itself Each token → one vector, used internally during generation, refined through the attention and feed-forward mechanics covered in sections 5 through 10 above. USE 2 — SENTENCE / DOCUMENT EMBEDDINGS for search & retrieval An entire sentence or paragraph → a single vector. This is what powers semantic search and RAG, and what vector databases like Pinecone, pgvector, or Weaviate actually store. Both share the same foundational idea from section 2: meaning as geometry. IF meaning can be represented as a point in space THEN similar meanings should sit close together AND different meanings should sit far apart "cat" and "dog" → close together both animals, pets "cat" and "car" → far apart animal vs machine "happy" and "joyful" → very close synonyms "happy" and "sad" → far apart antonyms An embedding model is a function that converts text into a point in high-dimensional space such that semantic relationships become geometric relationships you can measure with math. A vector's dimension is just how many numbers are in its list: 2-dimensional: 0.5, 0.8 — a point on a 2D plane 3-dimensional: 0.5, 0.8, 0.3 — a point in 3D space 1536-dimensional: 0.5, 0.8, 0.3, ..., 0.12 — a point in a space humans can't visualize, but the math is identical Modern embedding models use 768 to 4096 dimensions. The model learns what each dimension represents during training — nobody hand-labels them. A rough 3D sketch of what the resulting space looks like: "joyful" "happy" ● ● "sad" ● ● "cat" ● "kitten" ● "dog" ● "car" ● "truck" ● "bus" Similar concepts cluster together. Different concepts sit far apart. See it move.This ASCII sketch is a cartoon of something you can explore for real, on actual trained embeddings, at. It loads real word-embedding datasets, projects them down to 3D so you can rotate and zoom the actual cluster structure, and lets you search for a word to see its true nearest neighbors in the embedding space — the best way to build real intuition for "meaning as geometry" beyond a hand-drawn diagram. projector.tensorflow.org The embedding matrix starts as random numbers. Prediction pressure from the next-token objective section 15 is what sculpts it into something meaningful: Training sentence: "The cat sat on the mat" To predict "sat" after "The cat," the model must extract useful signal from "cat"'s vector. If that vector encodes something like "living creature that can perform actions," that helps predict action words → lower loss → the vector gets nudged to encode this even better. Elsewhere in training data: "dog sat," "bird sat," "child sat" — all push their subject vectors toward a similar region, because they all share similar predictive context. Tokens that appear in similar contexts end up with similar vectors — similar context is, functionally, similar meaning. king vector - man vector + woman vector ≈ queen vector Meaning: start at "king," remove the "man" concept, add the "woman" concept, land near "queen." Not hardcoded — it emerges because kings/queens and man/woman appear in parallel contexts throughout the training data. More examples: Paris - France + Italy ≈ Rome capital relationship transfers walked - walk + run ≈ ran tense relationship transfers doctor - man + woman ≈ nurse an unfortunate gender bias learned directly from training text That last one matters: embedding geometry reflects whatever biases exist in the training data, not ground truth — a real, practical problem in production systems that use embeddings for classification or retrieval. The TensorFlow Projector tool above lets you test this kind of vector arithmetic yourself on real embeddings. TOKEN EMBEDDINGS what the LLM uses internally : "The cat sat" → 3 tokens → 3 separate vectors, one per token Each vector's meaning shifts with context via attention . You cannot directly compare two whole sentences with these. SENTENCE EMBEDDINGS what RAG/search uses : "The cat sat on the mat" → ONE single vector for the whole sentence Two sentences can now be compared directly by comparing their single vectors. Sentence embeddings come from specialized embedding models, distinct from the generative LLM: OpenAI text-embedding-3-large → 3072 dimensions OpenAI text-embedding-3-small → 1536 dimensions sentence-transformers/all-MiniLM-L6-v2 → 384 dimensions open-source BGE-M3 BAAI → 1024 dimensions multilingual Cohere Embed v3 → 1024 dimensions A bi-encoder encodes two pieces of text independently into vectors, then compares the vectors afterward. "Bi" = two separate passes of the same encoder. DOCUMENT: "Nepal is a landlocked country in South Asia" ↓ encoder model 0.34, -0.12, 0.67, ... ← stored in a vector database QUERY: "Where is Nepal located?" ↓ same encoder model 0.31, -0.09, 0.71, ... COMPARISON: are the document vector and query vector close in space? close → relevant document far → not relevant You might reach for Euclidean straight-line distance first, but it's sensitive to vector length , not just direction — which produces wrong answers: "cat" → 0.5, 0.8 short vector "feline" → 2.0, 3.2 long vector, same DIRECTION as "cat" "car" → 0.8, -0.5 different direction Euclidean distance: "cat" to "feline" = large ← wrong same meaning, different magnitude "cat" to "car" = small ← wrong different meaning, similar magnitude Cosine similarity measures the angle between two vectors, ignoring length entirely — same direction gives 1.0, opposite gives -1.0, perpendicular gives 0: cosine "cat", "feline" = 1.0 ← correct: same direction = same meaning cosine "cat", "car" = 0.3 ← correct: different direction = different meaning The formula: cosine similarity A, B = A · B / |A| × |B| where A · B is the dot product section 8 and |A|, |B| are vector lengths. Dividing by the lengths cancels out magnitude, leaving only direction. In practice, relevant document/query pairs in a RAG system typically score 0.7–1.0, while irrelevant pairs land around 0.0–0.5. These are often confused, but it's one underlying idea used two different ways: CROSS-ENCODER single encoder, one pass : "query text" + "document text" → encoder → one similarity score Both texts are seen TOGETHER in one pass. The model can directly model interaction between them. Output is a score, not a vector. BI-ENCODER same encoder, two separate passes : "query text" → encoder → vector A ┐ ├→ compare → similarity score "document text" → encoder → vector B ┘ Each text is encoded alone — the model never sees them together. Output is two vectors, compared afterward. ANALOGY: Cross-encoder = interviewing two candidates together in one room — they interact, context is shared. Bi-encoder = interviewing each candidate separately, then comparing your notes afterward. A cross-encoder is more accurate it can model direct interaction between query and document but cannot scale to large corpora — it produces exactly one score per query, document pair, so scoring against a million documents means running the model a million separate times: 5 documents → 5 runs → fast, fine 50 documents → 50 runs → acceptable for reranking 1,000,000 docs → 1,000,000 runs → hours — completely unusable This is why production retrieval systems use a two-stage pipeline : YOUR CORPUS: 1,000,000 documents OFFLINE done once : every document → bi-encoder → vector → stored QUERY ARRIVES: "what is the retry limit?" STAGE 1 — Bi-encoder retrieval fast, milliseconds : query → bi-encoder → query vector compare against all 1,000,000 stored vectors → top 50 candidates STAGE 2 — Cross-encoder reranking accurate, 1-2 seconds : cross-encoder query + doc 4 → 0.94 cross-encoder query + doc 892 → 0.87 cross-encoder query + doc 234 → 0.71 ...50 runs total... sort scores → keep the top 5 STAGE 3 — Feed those top 5 documents to the LLM as context → answer Bi-encoder for broad, cheap recall across everything; cross-encoder for precise, expensive reranking of a short list. One edge case: concatenating multiple chunks into a single cross-encoder call technically works but reliably hurts accuracy, since attention gets diluted across multiple unrelated topics at once — always score chunks separately. Generic embedding models are trained on general internet text, so their vector space reflects everyday language: General embedding model: "consideration" → near "thought," "reflection," "deliberation" everyday English meaning Legal meaning: "consideration" = something of value exchanged in a contract a completely different, domain-specific meaning Query: "What is the consideration in this contract?" → using a general embedding model retrieves documents about "reflecting on" things → misses the actual contract clauses → retrieval — and therefore the whole RAG answer — fails This is why domain-specific embedding models exist, and why embedding model choice is a real engineering decision, not an afterthought. The embedding model is the foundation of a RAG system. A bad embedding model produces bad vectors, which produces bad retrieval — and no amount of LLM quality can compensate for retrieval that surfaces the wrong documents in the first place. The embedding model and vector database must agree on dimensions , and switching embedding models later means re-embedding your entire document set and rebuilding the index — not a cheap operation at scale, so choose carefully upfront. Embedding model updates can silently break retrieval quality if a provider updates a model internally and new query vectors end up in a slightly different space than your already-stored document vectors — no error is thrown, answers just quietly get worse. Pin versions and monitor retrieval quality directly. Cosine similarity thresholds are a tuning parameter , not a universal constant — typical starting points are 0.75+ for high-confidence-only retrieval, ~0.65 as a balanced default, and 0.50+ for broad recall paired with reranking afterward. Multilingual products need multilingual embedding models e.g. BGE-M3, multilingual-e5-large — an English-only embedding model simply can't match an English query to a relevant document written in another language. The last remaining question this guide hasn't touched: given everything above, what actually determines how fast and how expensive it is to run one of these models in production? In one line:four things determine LLM cost and speed — how many parameters engage per token, what precision they're stored at, how many requests get batched together, and how fast the GPU can move those parameters from memory to compute. Four factors determine how fast and how expensive it is to run an LLM, and they interact with each other: 1. Parameter count → how much computation per forward pass 2. Quantization → how much memory each parameter consumes 3. Batch size → how many requests get processed simultaneously 4. GPU memory bandwidth → how fast parameters move from memory to compute Every parameter section 18 participates in every forward pass, for every token generated: 7B model: 7,000,000,000 parameters engaged per token 70B model: 70,000,000,000 parameters engaged per token → 10x the computation Result: a 70B model generates tokens roughly 10x slower than a 7B model on identical hardware, and costs roughly 10x more to serve. This is a large part of why provider pricing scales with model size: GPT-4o mini → $0.60 / million output tokens small, fast GPT-4o → $15.00 / million output tokens large, capable Claude Haiku → $1.25 / million output tokens Claude Sonnet → $15.00 / million output tokens Quantization : reducing the numerical precision each parameter is stored at, trading some quality for less memory and faster compute. float32 → 32 bits/parameter → full precision, used during training float16 → 16 bits/parameter → half the memory of float32, negligible quality loss, standard for inference int8 → 8 bits/parameter → quarter the memory, small quality loss int4 → 4 bits/parameter → eighth the memory, noticeable quality loss Concretely, for a 70B model: float32: 70B × 4 bytes = 280 GB ← needs 4× H100 GPUs float16: 70B × 2 bytes = 140 GB ← needs 2× H100 GPUs int8: 70B × 1 byte = 70 GB ← fits on 1× H100 GPU int4: 70B × 0.5 byte = 35 GB ← fits on 1× A100 40GB GPU This is exactly why you can run a 70B open-weight model locally on two consumer GPUs using int4 — something completely impossible at full float32 precision. Lower precision also speeds things up: less data to move means faster loading into compute units see Factor 4 . Quick decision guide: float16 for anything quality-sensitive and running on real GPU infrastructure; int8 as a solid middle ground for local deployment; int4 only when memory is the hard constraint and you've verified the quality drop is acceptable for your specific task. Batch size the inference sense — see section 15 for how this differs from a training batch : the number of independent requests processed together in one forward pass. GPUs are built for massive parallelism, and a single request barely taxes that capacity: WITHOUT BATCHING: Request 1 → forward pass → response Request 2 → forward pass → response GPU utilization: ~5% per request. 3 requests take ~3x as long. WITH BATCHING: Requests 1, 2, 3 → ONE forward pass for all three → 3 responses GPU utilization: much higher. 3 requests processed in roughly the time of 1. This is how providers serve thousands of concurrent users on a fixed pool of GPUs. The tradeoff: bigger batches mean higher throughput but higher latency per individual request since a request has to wait for the batch to fill . This is also why API latency can vary with traffic load — busier periods fill batches faster and can actually be more efficient per-request, while quiet periods mean smaller batches and lower utilization. Memory bandwidth : the rate data moves from GPU memory VRAM into the compute cores, measured in GB/s. For every token generated, the GPU has to load all relevant parameters from VRAM, compute, store results, and repeat across every layer. 70B model at float16 = 140 GB of parameters to load, per token generated H100 GPU memory bandwidth: 3.35 TB/s Time to load 140 GB: 140 / 3,350 = 0.042 seconds = 42 ms per token Theoretical max speed on one H100: ~24 tokens/second for a 70B model This is why inference is described as memory bandwidth bound , not compute bound — for single-user generation, the GPU's compute cores mostly sit idle, waiting for parameters to arrive from VRAM. More raw compute doesn't help; more bandwidth does. It's also why newer GPU generations H100 vs. A100 improve inference speed mainly through higher memory bandwidth, not just more cores. PREFILL processing the input prompt : all tokens processed in one parallel pass → compute bound → more cores = faster → determines time-to-first-token TTFT DECODE generating output : one token per step, sequential → memory-bandwidth bound → more bandwidth = faster → determines total generation time Short prompt + long output → fast prefill, slow decode Long prompt + short output → slower prefill, fast decode Every concurrent user needs their own KV cache section 13 sitting in VRAM alongside the model's own parameters: 70B model parameters: 140 GB 100 concurrent users' KV caches combined: potentially 50-100 GB more Total VRAM needed: 190-240 GB → requires 3 H100 GPUs minimum Serving more concurrent users often requires more GPUs not because of extra compute, but because of KV cache memory pressure. GIVEN: Model: 70B parameter Claude Sonnet-tier pricing Traffic: 1,000 requests/day Average input: 1,000 tokens/request Average output: 500 tokens/request STEP 1 — Token volumes: Input tokens per day: 1,000 × 1,000 = 1,000,000 Output tokens per day: 1,000 × 500 = 500,000 STEP 2 — Cost: Input: 1,000,000 × $3/1M = $3.00/day Output: 500,000 × $15/1M = $7.50/day Total: $10.50/day ≈ $315/month STEP 3 — Optimization levers: Output tokens cost 5x input tokens, so cutting average output from 500 → 200 tokens saves 300,000 × $15/1M = $4.50/day ≈ $135/month. Semantic caching with a 30% hit rate saves ~30% of total cost ≈ another $94.50/month. Output tokens are the biggest cost lever you directly control. Setting max tokens tightly and asking for concise responses "answer in one sentence" vs. "explain in detail" can mean a 10x cost difference with no real quality loss for many tasks. Model tiering is a real economics decision, not just an optimization nice-to-have. Routing simple classification or yes/no decisions to a small, cheap model and reserving the expensive model for genuinely complex requests can cut costs 60-70% with no quality loss where it matters — if 70% of your traffic is simple, running 100% of it through the expensive model is pure waste. API vs. self-hosting is a real break-even calculation, not a vibe. API access wins for unpredictable or low traffic and no appetite for GPU ops. Self-hosting wins at high, predictable volume — e.g. at 1 billion output tokens/month, API cost ~~$15,000/month at Sonnet-tier pricing can dwarf an amortized self-hosted H100 setup ~~$2,000/month . Diagnose latency by bottleneck, not by guessing. Slow time-to-first-token points at prefill shorten the prompt, use faster compute . Slow total generation time points at decode stream the response, use a smaller model, use higher-bandwidth hardware . Slowdowns specifically under many concurrent users point at KV cache memory pressure more GPUs, shorter context limits, cache eviction policies . That covers the full arc: text in, vectors, attention, generation, training, scale, embeddings, and cost. The glossary below is a flat reference for looking any single term back up without re-reading a whole section. Ordered to match the order terms first appear in this guide, so it also works as a quick recap if read top to bottom. | Term | Plain-English Definition | |---|---| Vector | A list of numbers representing a word, sentence, or concept. | Token | A chunk of text roughly a word or word-piece mapped to an integer ID. | Embedding | The vector a token or piece of text gets converted into; similar meanings → similar vectors. | Weight matrix | A learned grid of numbers used to transform one vector into another. | Positional encoding | Information added to embeddings so the model knows token order. | RoPE | A modern positional encoding method that rotates Q/K vectors by an angle based on position; enables longer context windows. | Self-attention | The mechanism letting every token directly consider every other token when updating its meaning. | Query Q | A token's "what context do I need?" vector. | Key K | A token's "here's what I contain" vector, compared against queries. | Value V | A token's "here's what I contribute if selected" vector. | Dot product | A single number measuring how aligned two vectors are. | Softmax | Converts a list of raw numbers into probabilities that sum to 1. | Multi-head attention | Running several independent Q/K/V attention computations in parallel, each specializing in a different relationship type. | Causal mask | Blocks a token from attending to any token that comes after it, so generation never "cheats" by looking at the future. | Residual connection | Adding a layer's input back onto its output so information and training signal can skip past the layer untouched. | Layer normalization | Rescaling a vector's values to a consistent, stable range after each sub-layer, so deep stacks of layers train reliably. | Feed-forward network FFN / MLP | The per-token sub-layer inside each transformer block that processes each token's vector independently, with no cross-token communication. | Activation function e.g. GELU | A nonlinear function applied inside the FFN; without it, stacked linear layers would collapse into a single linear transformation. | Decoder-only / causal model | An architecture GPT, Claude, Llama trained with a causal mask so it only ever predicts forward, one token at a time — as opposed to encoder-only models BERT built to understand a whole passage at once. | Logits | Raw, unnormalized output scores before softmax turns them into probabilities. | Autoregressive generation | Producing text one token at a time, feeding each output back in as input for the next step. | Greedy decoding | Always picking the single highest-probability token — deterministic. | Sampling | Picking a token probabilistically from the distribution — non-deterministic. | Temperature | A number that sharpens low or flattens high the probability distribution before sampling. | Top-p nucleus sampling | Keeps only the smallest set of top tokens whose probabilities sum to at least p, then samples from those. | Top-k | Keeps a fixed number of top tokens regardless of the shape of the distribution. | KV cache | Stored Key/Value vectors for past tokens, reused instead of recomputed on every generation step. | Prefill | The parallel processing phase for your input prompt; determines time-to-first-token. | Decode | The sequential, one-token-at-a-time generation phase; determines total generation time. | Loss function / cross-entropy loss | The number that measures how "surprised" the model was by the real next token; training tries to minimize this. | Gradient | For a single parameter: how much the loss would change if that parameter were nudged slightly. | Backpropagation | The algorithm that computes every parameter's gradient by working backward from the loss through every layer. | Gradient descent | Nudging every parameter a small step in the direction that reduces loss, repeated until the model converges. | Learning rate | How big each gradient-descent step is; too high overshoots, too low crawls. | Hallucination | Confident, fluent, but factually wrong or fabricated output — a side effect of optimizing for plausibility, not truth. | Parametric memory | Knowledge implicitly encoded in a model's weights, with no explicit database-like structure. | RAG Retrieval-Augmented Generation | Retrieving real, relevant documents and injecting them into the prompt to ground the model's answer in actual facts. | Pre-training | The initial, massive-scale next-token-prediction training run that produces the base model. | SFT Supervised Fine-Tuning | Further training a base model on curated human-written examples to teach it the assistant format. | RLHF | Reinforcement Learning from Human Feedback — aligning a model's behavior using a reward model trained on human preference rankings. | Reward model | A separate model trained to score how good a response is, standing in for human judgment during RLHF. | Constitutional AI / RLAIF | Anthropic's approach: using AI self-critique against written principles instead of relying solely on human rankings. | DPO Direct Preference Optimization | A simpler alternative to RLHF that updates model weights directly from preference data, without a separate reward model. | Parameter | A single learned number inside a model's weight matrices. | Scaling laws | Empirical relationships between model size, training data size, and resulting performance. | Emergent capability | An ability that appears suddenly at a scale threshold rather than improving gradually. | Mixture of Experts MoE | An architecture that activates only a small subset of specialized sub-networks per token, instead of the whole model. | Sentence/document embedding | A single vector representing the meaning of an entire sentence or document, used for semantic search. | Cosine similarity | A measure of how similar two vectors' directions are, ignoring their length. | Bi-encoder | Encodes two texts independently, then compares the resulting vectors — fast, used for broad retrieval. | Cross-encoder | Encodes two texts together in one pass to produce a single relevance score — accurate but slow, used for reranking. | Quantization | Reducing the numeric precision parameters are stored at, to save memory at some cost to quality. | Batch size | The number of requests processed together in a single forward pass inference — distinct from a training batch, see section 15. | Memory bandwidth | How fast data moves from GPU memory to compute cores — the main bottleneck during generation. | This guide distills the core mental model behind every modern LLM, in the order the data itself flows: text becomes vectors, position gets baked in, attention lets tokens exchange context with a causal mask keeping generation honest, and residuals/LayerNorm keeping 80 stacked layers trainable , generation happens one probabilistic token at a time, and a training pipeline backpropagation → pre-training → SFT → RLHF turns a raw next-token predictor into a helpful assistant. Everything else — RAG, prompting techniques, agents, fine-tuning — is built on top of this foundation. Three interactive tools are linked inline above, right where they're most useful, rather than bundled at the end: the whole pipeline in animated 3D section 3 , a live transformer block running in your browser section 10 , and a real, explorable embedding space section 19 .