How the Transformer Paper Came About

The Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need,' was designed primarily to reduce training time by enabling parallelization, not to improve translation quality. The paper's key contribution was replacing recurrent and convolutional layers with self-attention, which requires a constant number of sequential operations regardless of sequence length. However, the quadratic complexity of self-attention in sequence length later became a central challenge for scaling context windows in modern large language models.

Attention Is All You Need appeared on arXiv on 12 June 2017 and at NIPS that December. It is a machine translation paper, its central argument is about training time rather than about quality, and the architecture it describes is not the one used by any current large language model. Almost everything that made transformers matter was added by other people within the following eighteen months. By 2017 the state of the art in machine translation https://multigrid.ai/learn/machine-translation-history was a recurrent encoder-decoder https://multigrid.ai/learn/encoder-decoder with attention: read the source sentence one token at a time with an LSTM or GRU https://multigrid.ai/learn/rnn-lstm-explained , and generate the target the same way, with an attention mechanism letting the decoder look back at the encoder’s states. It worked and it was unbearably slow to train, for a structural reason rather than an implementation one. A recurrent network’s hidden state at position t depends on the state at t−1 . That dependency chain cannot be parallelised along the sequence, so a training example of length 50 requires 50 sequential steps regardless of how many processors are available. On hardware whose entire advantage is doing thousands of things at once, this is the worst possible shape of computation. Several groups were attacking it by replacing recurrence with convolution — ByteNet and ConvS2S are both cited in the paper for exactly this. Convolution parallelises, but the number of operations needed to relate two positions grows with the distance between them: linearly for ConvS2S, logarithmically for ByteNet. The transformer paper’s pitch is that self-attention relates any two positions in a constant number of operations, which is the specific claim its table of complexities is there to support. Read the abstract and the case is not that attention produces better representations in some deep sense. It is that dispensing with recurrence and convolutions entirely gives a model that is more parallelisable and requires significantly less time to train, while being at least as good. The three-column comparison the paper's argument rests on per layer, for a sequence of length n and representation size d : complexity sequential max path per layer operations length self-attention O n² · d O 1 O 1 recurrent O n · d² O n O n convolutional O k · n · d² O 1 O log k n The middle column is the whole argument. Self-attention needs a constant number of sequential steps regardless of sequence length, so the work fits on parallel hardware. Note the first column, because it is the bill that came due later: self-attention is quadratic in sequence length. In 2017, with n ≈ 50 tokens of a sentence and d = 512, the n² term is negligible. It is why context length is expensive now. That last point deserves emphasis, because it explains why a design decision that was obviously correct in 2017 became the central engineering problem of the 2020s. At sentence length, quadratic attention https://multigrid.ai/learn/attention-complexity costs nothing. At a hundred thousand tokens it dominates everything, and an entire subfield exists to work around it. The paper’s other components are stated matter-of-factly and have all survived: multi-head attention, so that different heads can attend to different kinds of relationship simultaneously; scaled dot-product attention, with the division by the square root of the key dimension to stop the softmax saturating; sinusoidal positional encodings, since a set of vectors with no recurrence has no inherent order; residual connections and layer normalisation around each sub-layer; and a position-wise feed-forward network with an inner dimension four times the model dimension. That 4× ratio is still the default in models a thousand times larger. | Reported in the paper | Description | |---|---| | WMT 2014 English-to-German | 28.4 BLEU for the big model — stated in the abstract as an improvement of more than 2 BLEU over the previous best results, including ensembles. | | WMT 2014 English-to-French | 41.8 BLEU for the big model, described as a new single-model state of the art. | | Training cost | The big model trained for 3.5 days on eight NVIDIA P100 GPUs; the base model for about 12 hours on the same eight cards. The paper is explicit that this is a small fraction of the cost of the best published models it beats. | | Model sizes | About 65 million parameters for the base configuration and about 213 million for the big one. Both are smaller than a modern embedding model. | Eight GPUs for three and a half days. The paper that produced the architecture behind every frontier model was a result anybody with a single well-equipped machine could have reproduced in a week, and this is worth stating plainly against the current cost of a frontier training run https://multigrid.ai/learn/frontier-training-cost . The paper also reports an English constituency parsing experiment, included to show the architecture generalised beyond translation. It is a modest result and it is the only evidence in the paper that the model is good for anything other than the task it was built for. The paper contains a table of variations on the base model, and it is the section that best repays reading now, because it shows which of the design choices the authors themselves found load-bearing. The general lesson from that table is that the transformer as published is a set of choices, several of them made on modest evidence at one model size on one task, and that the field then inherited all of them at a thousand times the scale. Some survived scrutiny and some — the positional encoding in particular — did not. This is the section that separates the source from the folklore. The following are all routinely attributed to Attention Is All You Need and none of them is in it: The accurate one-line summary is: the paper contributed the block, not the recipe. That is not a diminishment — the block turned out to be the reusable part, and section-by-section it holds up remarkably well. A closer reading of the architecture itself is in the section-by-section walkthrough of the paper https://multigrid.ai/learn/attention-paper . Five papers in twenty months, only the first of which is the one everybody cites. If you want to know where any specific property of a modern model came from, the answer is almost never 2017. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. The paper carries a footnote stating that the contribution is equal and that the listing order is random — an unusual choice, and one that has complicated every attempt to describe who did what. The title is generally reported to be a nod to the Beatles song, and the suggestion is usually attributed to Llion Jones. All eight authors had left Google by the early 2020s, and between them founded or joined a large number of the companies now working on the technology — Cohere, Character.AI, Sakana AI, Inceptive, Adept and NEAR among them, with Kaiser going to OpenAI. That diaspora is a fair proxy for how the field valued the paper in retrospect, and it is a much better-attested fact than any of the anecdotes about how the work was done, which come from interviews given years afterwards and do not always agree with one another. This page deliberately does not retell the origin-story anecdotes about who suggested what in which corridor. They are entertaining, they come from retrospective interviews, and several published versions conflict. The paper, its date, its results and its citations are checkable; the corridor is not.