In 2017, a paper from Google proposed dropping recurrence from sequence models entirely. The argument fits in a sentence: *a self-attention layer connects any two positions in * A recurrent network needs n steps to carry a signal from the first word of a sentence to the last. A convolutional one needs a stack of layers, log of n deep. Attention needs one. The parallel training, the speed, the long-range dependencies these models actually learned: all of it follows from that.
one hop.
This page works through the paper in order. Most of the figures below are running a real transformer, a small one, in your browser. You can grab almost everything you see.
§ 1One hop Start with the fact itself. Below are three ways to move information from one end of a sequence to the other. The recurrent lane passes a signal from each position to the next, one step at a time, and the signal decays a little at every hand-off. The convolutional lane lets each position see a window of k neighbors, so reach grows by a factor of k per layer. The attention lane connects every position to every other position directly.
Set n to a long sentence and watch the step counters. That is the whole thesis. The rest of the page is the price paid for the one hop, and the evidence that the price is low.
§ 2The bottleneck Why not just keep recurrence? Because the hidden state at position t depends on the one at t−1. That dependency is a chain through the whole sequence, and nothing near the end can be computed until everything ahead of it is done. Within one training example there is no parallelism to find, and at long sequence lengths, memory limits how many examples you can batch together to compensate.
§ 3Scaled dot-product attention What does the one-hop mechanism actually do? From here on, the figures are live. A small transformer is running in your browser: the paper's architecture with two layers instead of six and 48 dimensions instead of 512, trained to read a date in any messy format and write it back as yyyy-mm-dd. It is not the paper's model, and this page will always say which numbers come from the toy and which come from the paper. But nothing the toy shows you is faked. Every tensor is computed from the actual weights, on your machine, as you watch.
Type a date. Pick a character. That character's query vector is dotted against every position's key; the scores go through a softmax; the resulting weights blend every position's value vector into one output. Every stage is exposed below.
[§ 4](#sec-sqrtdk)Why √dk
The formula divides the dot products by the square root of the key dimension before the softmax, and the paper spends a footnote on why. If the components of a query and a key are independent with mean 0 and variance 1, their dot product has variance d_k, which grows with the dimension. Push large scores through a softmax and it saturates: one weight goes to 1, the rest to 0, and the gradients through the softmax go with them.
Raise d_k and watch the distribution widen and the gradient gauge fall. Then turn the scaling on and watch it snap back.
§ 5Multi-head One attention pattern per layer is a bottleneck of its own, since averaging inhibits it. So the model runs several small attentions in parallel: each head projects the input down, attends in its own subspace, and the results are concatenated and mixed back together. The four heads below are the toy model's real heads, and they have learned different jobs.
Turn a head off and watch the prediction shift, or fall apart entirely. The weights are real, so the damage is too.
§ 6Positional encoding Attention is a set operation: shuffle the inputs and the outputs shuffle with them. Order has to be put in explicitly, so the model adds a position signal to each embedding: sinusoids whose wavelengths run from 2π up to 10000·2π, one pair of dimensions per frequency. Think of a bank of clock hands spinning at geometrically spaced rates.
Drag the offset. Every hand advances by a fixed rotation. That is the paper's reason for choosing sinusoids: the encoding of position pos+k is a linear function of the encoding of pos, so relative offsets are easy for the model to represent.
§ 7The stack Here is the whole machine. Six identical encoder layers, each self-attention followed by a position-wise feed-forward network; six decoder layers with a third sub-layer attending over the encoder's output. Every sub-layer is wrapped in a residual connection and layer normalization. The output embedding is offset by one position, for a reason the next section gets to.
§ 8The mask The decoder generates left to right, and during training it must not see the future it is supposed to predict. The fix is blunt: before the softmax, every score from a position to any later position is set to −∞, which the softmax turns into exactly zero weight.
Toggle the mask off and watch the prediction row underneath. This decoder was trained with the mask in place, so showing it the future doesn't help it cheat. It breaks it.
§ 9The feed-forward sub-layer Between attentions, each position is pushed through the same small two-layer network: expand to 2048 dimensions, rectify, project back. The same weights at every position within a layer, different weights across layers. Equivalently: two convolutions with kernel size 1.
§ 10What the hop costs Self-attention's one hop costs n²·d work per layer, against recurrence's n·d². The question is where the crossover sits. Below, drag d and run along n yourself. At the paper's d = 512, a sentence has to get very long before attention's quadratic term catches up, and sentence-level n is usually far below that.
§ 11Training The recipe is spare: Adam with a learning rate that ramps up linearly for 4000 steps and then decays as the inverse square root of the step count, dropout on every sub-layer output, and label smoothing that hurts perplexity but helps BLEU. The base model trained in about twelve hours on eight P100s.
§ 12Cost against quality The results, plotted as quality against training compute. On English-to-German the transformer sits up and to the left of the whole field: the base model beats every earlier single model and every ensemble, at roughly 3× less compute than the cheapest of them and 55× less than the dearest. English-to-French is the less tidy panel — there the base model trails four earlier systems, and it takes the big model to pass them all, still at less than a quarter of the previous state of the art's training cost.
§ 13Ablations Which parts matter? The paper varied one thing at a time. A single head costs 0.9 BLEU, and piling on too many heads costs something as well. Shrinking the key dimension hurts. Bigger models help. Dropout matters a great deal.
§ 14What the heads learned The paper ends by looking inside. Individual heads in the trained model turn out to do legible work: one follows a long-distance dependency seven positions ahead; two others resolve the pronoun “its” onto both “Law” and “application.” The architecture also generalized on the first try: a four-layer transformer got within reach of the best constituency parsers of the time.
That is the paper. One idea — every position one hop from every other — and the engineering that makes the idea trainable: the scaling, the heads, the sinusoids, the mask. You have handled each piece now. The title is an inventory of them.
The toy model on this page is 2 encoder and 2 decoder layers, dmodel 48, 4 heads, 109,376 parameters, trained once on synthetic dates and shipped as 214 KB of fp16 weights. Every live figure computes from those weights in plain JavaScript. Read the forward pass and check the arithmetic against the paper's section numbers.