Nobody sat down in 2017 and designed the Transformer from first principles.
What actually happened is messier and much more interesting: someone built a thing, it broke in a specific and annoying way, someone else patched the break, and that patch broke somewhere new. Twenty years of this and you get ChatGPT.
Which means the history of language models reads less like a research timeline and more like a changelog. So let's read it that way.
Upfront: I'm learning this stuff, not teaching it from experience. This is the map I drew for myself while trying to understand how we got from word-counting to RAG, written down partly because explaining something is how I find out whether I actually understood it. Corrections very welcome.
Approach: Hire linguists. Write grammar rules. "English adjective before noun → French adjective after noun." Thousands of them.
🐛 Bug: Language has infinite exceptions, and rules can't represent "probably."
"I saw the man with the telescope."
Did you use a telescope, or did he have one? No grammar rule settles this. You need to know what's likely in context. Rules don't do likely.
Takeaway that stuck: stop specifying language by hand. Learn it from examples.
Approach: Take a mountain of text. Count what follows what.
Saw "the cat sat on the"
a hundred times, and "mat"
came next forty times? Cool: P(mat) = 0.4
. This is an n-gram model, and it ran speech recognition and machine translation for two solid decades.
🐛 Bug: Every word is an unrelated ID number.
Show this model "The cat is walking in the bedroom"
a thousand times. Then ask it about:
"A dog was running in a room."
It has learned nothing. Zero transfer. To this model, cat
and dog
are as related as cat
and bulldozer
- just distinct labels with no insides. Every synonym, plural, and tense had to be learned from scratch, separately, forever.
This bug is the one that mattered.
Approach: Stop using IDs. Give every word a list of numbers, a vector and let those numbers be learned.
cat → [ 0.21, -0.85, 0.44, ... ]
dog → [ 0.19, -0.79, 0.51, ... ] # neighbors
car → [-0.63, 0.22, -0.90, ... ] # different neighborhood
Picture a map where similar meanings sit near each other. Because cat
and dog
show up in similar sentences across the training data, training quietly pulls their coordinates together. Now learning about cats does teach you something about dogs, for free.
Then word2vec (2013) revealed something nobody programmed in:
king − man + woman ≈ queen
Paris − France + Italy ≈ Rome
Subtract "man" from "king," you're left with something like royalty. Meaning turned out to have arithmetic.
🐛 Bug: One vector per word. Forever.
"I sat on the river bank." ┐
"I got a loan from the bank." ┘ same vector
One set of coordinates has to serve both meanings, so it lands in a mush that fits neither. Same story for play, run, light, right — a big chunk of common English.
What you want is a vector for "bank" that depends on the rest of the sentence. Which requires a model that actually reads the sentence.
Approach: Loop over words one at a time, carrying a scratchpad of numbers that summarizes what you've read so far.
🐛 Bug: It's a game of telephone.
The scratchpad gets rewritten at every step. Thirty steps in, word 1 is a rumor. And training is worse: to learn that word 30 depends on word 2, the learning signal has to travel back through 28 steps, getting multiplied each time. If those multipliers are around 0.7:
0.7 ^ 28 ≈ 0.0000002
That's the vanishing gradient problem, and it's not "learns long-range patterns badly." It's never learns them at all.
Patch (LSTM, 1997): Add a second pathway — a memory lane where information is updated by addition instead of being overwritten, guarded by three learned dimmer switches (what to forget, what to store, what to expose).
The difference is whispering a message down a line of thirty people versus handing them a sealed envelope to pass along. Now the model can write "the subject is plural," seal it, carry it eight words, and open it when it's time to pick are over is.
🐛 Bug (still): It's sequential by construction. Word 1 before word 2 before word 3. Buy a hundred GPUs; they'll all sit around waiting for each other.
🐛 Bug (new): For translation, the standard design squeezed the entire source sentence through one fixed-size vector. Same suitcase whether you're packing for a weekend or a year. Quality fell off a cliff on long sentences, exactly as you'd expect.
Approach (2014): Why compress at all? The encoder already produced a scratchpad for every input word — they're all still sitting in memory. Let the decoder look at all of them and learn which ones matter right now.
Translating "le chat dort" → "the cat sleeps", while writing "cat":
le ▍ 4%
chat ████████████████ 85%
dort ██ 11%
Then the focus slides over to dort
for "sleeps". A custom summary, rebuilt for every single output word.
Long sentences stopped degrading. You could plot the percentages and see what the model was using. Both great.
But the accidental discovery was the important one: attention connects any two positions in a single hop. In an RNN, getting from word 3 to word 90 means trudging through 87 steps and losing the signal. With attention it's one step, and nothing fades.
Which raises an awkward question. If attention gives you unlimited range for free, and recurrence is the thing making everything slow and forgetful - why are we keeping the recurrence?
Approach (2017): Throw away the loop. Keep only the attention part.
The paper is called "Attention Is All You Need." I assumed that was a cheeky title. It isn't, it's a literal description of what they deleted.
Here's the bit that took me longest to get. I assumed the Transformer was a big deal because it gave better answers. It does, but that's not really the point. The point is that it's fast in a way the old models could never be.
Think about how an LSTM reads. Word 1, then word 2, then word 3, strictly in that order because each step needs the answer from the step before it.
A Transformer looks at every word at the same time. Not in sequence — all at once. And "do thousands of things simultaneously".
Every word now looks at every other word in the same sentence. The mechanism is a lookup. Each word carries three things:
| It has | Meaning |
|---|---|
| A question (Query) | |
| "here's what I'm trying to work out" | |
| A label (Key) | |
| "here's what I am" | |
| Some content (Value) | |
| "here's what you get from me, if I turn out to be useful" |
Each word writes a search query, checks it against everyone's labels, and collects a blend of the contents from whoever matched. Take:
"The animal didn't cross the street because
itwas too tired."
Is it
the animal or the street? Watch the query for it
come back:
animal ████████████████████ 71%
tired ██████ 12%
street ██ 5%
cross █ 4%
...
The new representation of it
is now 71% animal. It has absorbed the thing it refers to. One step, in parallel with every other word doing the same thing.
And notice — this also kills the v0.3 bug completely. Every word's vector is now built from its actual surroundings, so river bank
and bank account
come out entirely different.
Run several of these in parallel (attention heads) and they specialize on their own during training. People poking at trained models have found heads that track subject-verb agreement, heads that follow pronouns, heads that match quotation marks. A committee of specialists that hired itself.
🐛 Bug: Every word compares itself to every word. 100 words → 10,000 comparisons. 1,000 words → 1,000,000. 10× the text, 100× the work. That's why context windows are a scarce resource and why an entire subfield exists to soften the curve.
Approach (2020): GPT-3 was not a new architecture. It was GPT-2 at roughly 100× scale.
Then something weird showed up:
English: sea otter → French: loutre de mer
English: cheese → French: fromage
English: peppermint → French: ___
It fills that in correctly. With frozen weights. No training, no gradient step, nothing — it learned the task from the prompt and forgets it afterward. This is in-context learning, and it flipped how anyone uses ML. Before: gather labeled data, train, deploy, repeat per task. After: write a prompt.
🐛 Bug: It's an autocomplete engine, not an assistant.
Ask a raw pretrained model:
You:What are three good exercises for keeping focus?
Model:What stretches help with sciatica? How do I fix my posture at a desk?
It isn't being difficult. It correctly noticed that documents containing your question usually contain more questions as you've handed it an FAQ heading. It's modeling text. It has no idea it's supposed to help you.
Approach (2022): Show it thousands of examples of good answers (fine-tuning), then teach it human taste:
That's RLHF - reinforcement learning from human feedback. Step 3 exists purely because of cost: you can't pay humans to rate millions of answers, but you can pay them to rate a few thousand and then train something to imitate their taste.
The number that should recalibrate your instincts: a 1.3B-parameter aligned model was preferred by human raters over 175B unaligned GPT-3. Alignment beat a 100× size increase
Approach: Stop asking the model to remember. Look the information up when the question arrives and paste it into the prompt.
It's a closed-book exam becoming an open-book exam. A student sitting closed-book has to have memorized everything, will misremember some of it, can't tell you which page a fact came from, and has never seen your company handbook. Hand them the relevant pages and all four problems improve at once while the thing they're actually good at, reasoning and writing clearly, still gets used.
INDEX (once, then keep updated)
docs → split into chunks → embed each chunk → searchable index
ANSWER (per question)
question
→ search the index (grab ~50 candidates)
→ rerank (narrow to ~5)
→ build prompt: [instructions] + [chunks] + [question]
→ LLM answers, citing sources
Those chunk embeddings are the exact same idea from v0.3 - just grown up. Back then we mapped single words so similar words landed near each other. Now we map whole passages so similar meanings land near each other. You're not matching keywords; you're matching positions on a map of meaning.
One thing I'd love from the comments. If I've got something wrong above, please say so - I'd much rather be corrected here than discover it three weeks into building something.