{"slug": "attention-is-all-you-need-with-a-real-transformer-running-in-the-page", "title": "Attention Is All You Need, with a real transformer running in the page", "summary": "An interactive web page by an unnamed author walks through Google's 2017 'Attention Is All You Need' paper, running a real small transformer in the browser to demonstrate how self-attention connects any two sequence positions in one hop, unlike recurrent or convolutional models. The page explains the architecture's components—scaled dot-product attention, multi-head attention, and positional encoding—using live, un-faked tensors from a toy model trained to normalize dates.", "body_md": "In 2017, [a paper from\nGoogle](https://arxiv.org/abs/1706.03762) proposed dropping recurrence from\nsequence models entirely. The argument fits in a sentence: *a\nself-attention layer connects any two positions in * A\nrecurrent network needs n steps to carry a signal\nfrom the first word of a sentence to the last. A convolutional one needs a\nstack of layers, log of n deep. Attention needs\none. The parallel training, the speed, the long-range dependencies these\nmodels actually learned: all of it follows from that.\n\n**one hop**.\n\nThis page works through the paper in order. Most of the figures below are running a real transformer, a small one, in your browser. You can grab almost everything you see.\n\n[§ 1](#sec-onehop)One hop\n\nStart with the fact itself. Below are three ways to move information from one end of a sequence to the other. The recurrent lane passes a signal from each position to the next, one step at a time, and the signal decays a little at every hand-off. The convolutional lane lets each position see a window of k neighbors, so reach grows by a factor of k per layer. The attention lane connects every position to every other position directly.\n\nSet n to a long sentence and watch the step counters. That is the whole thesis. The rest of the page is the price paid for the one hop, and the evidence that the price is low.\n\n[§ 2](#sec-bottleneck)The bottleneck\n\nWhy not just keep recurrence? Because the hidden state at position t depends on the one at t−1. That dependency is a chain through the whole sequence, and nothing near the end can be computed until everything ahead of it is done. Within one training example there is no parallelism to find, and at long sequence lengths, memory limits how many examples you can batch together to compensate.\n\n[§ 3](#sec-sdpa)Scaled dot-product attention\n\nWhat does the one-hop mechanism actually do? From here on, the figures are live. A small transformer is running in your browser: the paper's architecture with two layers instead of six and 48 dimensions instead of 512, trained to read a date in any messy format and write it back as yyyy-mm-dd. It is not the paper's model, and this page will always say which numbers come from the toy and which come from the paper. But nothing the toy shows you is faked. Every tensor is computed from the actual weights, on your machine, as you watch.\n\nType a date. Pick a character. That character's query vector is dotted against every position's key; the scores go through a softmax; the resulting weights blend every position's value vector into one output. Every stage is exposed below.\n\n[§ 4](#sec-sqrtdk)Why √dk\n\nThe formula divides the dot products by the square root of the key dimension before the softmax, and the paper spends a footnote on why. If the components of a query and a key are independent with mean 0 and variance 1, their dot product has variance d_k, which grows with the dimension. Push large scores through a softmax and it saturates: one weight goes to 1, the rest to 0, and the gradients through the softmax go with them.\n\nRaise d_k and watch the distribution widen and the gradient gauge fall. Then turn the scaling on and watch it snap back.\n\n[§ 5](#sec-heads)Multi-head\n\nOne attention pattern per layer is a bottleneck of its own, since averaging inhibits it. So the model runs several small attentions in parallel: each head projects the input down, attends in its own subspace, and the results are concatenated and mixed back together. The four heads below are the toy model's real heads, and they have learned different jobs.\n\nTurn a head off and watch the prediction shift, or fall apart entirely. The weights are real, so the damage is too.\n\n[§ 6](#sec-pe)Positional encoding\n\nAttention is a set operation: shuffle the inputs and the outputs shuffle with them. Order has to be put in explicitly, so the model adds a position signal to each embedding: sinusoids whose wavelengths run from 2π up to 10000·2π, one pair of dimensions per frequency. Think of a bank of clock hands spinning at geometrically spaced rates.\n\nDrag the offset. Every hand advances by a fixed rotation. That is the paper's reason for choosing sinusoids: the encoding of position pos+k is a linear function of the encoding of pos, so relative offsets are easy for the model to represent.\n\n[§ 7](#sec-stack)The stack\n\nHere is the whole machine. Six identical encoder layers, each self-attention followed by a position-wise feed-forward network; six decoder layers with a third sub-layer attending over the encoder's output. Every sub-layer is wrapped in a residual connection and layer normalization. The output embedding is offset by one position, for a reason the next section gets to.\n\n[§ 8](#sec-mask)The mask\n\nThe decoder generates left to right, and during training it must not see the future it is supposed to predict. The fix is blunt: before the softmax, every score from a position to any later position is set to −∞, which the softmax turns into exactly zero weight.\n\nToggle the mask off and watch the prediction row underneath. This decoder was trained with the mask in place, so showing it the future doesn't help it cheat. It breaks it.\n\n[§ 9](#sec-ffn)The feed-forward sub-layer\n\nBetween attentions, each position is pushed through the same small two-layer network: expand to 2048 dimensions, rectify, project back. The same weights at every position within a layer, different weights across layers. Equivalently: two convolutions with kernel size 1.\n\n[§ 10](#sec-crossover)What the hop costs\n\nSelf-attention's one hop costs n²·d work per layer, against recurrence's n·d². The question is where the crossover sits. Below, drag d and run along n yourself. At the paper's d = 512, a sentence has to get very long before attention's quadratic term catches up, and sentence-level n is usually far below that.\n\n[§ 11](#sec-training)Training\n\nThe recipe is spare: Adam with a learning rate that ramps up linearly for 4000 steps and then decays as the inverse square root of the step count, dropout on every sub-layer output, and label smoothing that hurts perplexity but helps BLEU. The base model trained in about twelve hours on eight P100s.\n\n[§ 12](#sec-results)Cost against quality\n\nThe results, plotted as quality against training compute. On English-to-German the transformer sits up and to the left of the whole field: the base model beats every earlier single model and every ensemble, at roughly 3× less compute than the cheapest of them and 55× less than the dearest. English-to-French is the less tidy panel — there the base model trails four earlier systems, and it takes the big model to pass them all, still at less than a quarter of the previous state of the art's training cost.\n\n[§ 13](#sec-ablations)Ablations\n\nWhich parts matter? The paper varied one thing at a time. A single head costs 0.9 BLEU, and piling on too many heads costs something as well. Shrinking the key dimension hurts. Bigger models help. Dropout matters a great deal.\n\n[§ 14](#sec-learned)What the heads learned\n\nThe paper ends by looking inside. Individual heads in the trained model turn out to do legible work: one follows a long-distance dependency seven positions ahead; two others resolve the pronoun “its” onto both “Law” and “application.” The architecture also generalized on the first try: a four-layer transformer got within reach of the best constituency parsers of the time.\n\nThat is the paper. One idea — every position one hop from every other — and the engineering that makes the idea trainable: the scaling, the heads, the sinusoids, the mask. You have handled each piece now. The title is an inventory of them.\n\nThe toy model on this page is 2 encoder and 2 decoder layers,\ndmodel 48, 4 heads, 109,376 parameters, trained once on\nsynthetic dates and shipped as 214 KB of fp16 weights. Every live\nfigure computes from those weights in plain JavaScript.\n[Read the forward pass](js/model.mjs) and check the arithmetic\nagainst the paper's section numbers.", "url": "https://wpnews.pro/news/attention-is-all-you-need-with-a-real-transformer-running-in-the-page", "canonical_source": "https://onehop.world/", "published_at": "2026-09-01 13:08:18+00:00", "updated_at": "2026-09-01 13:23:39.544084+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-tools"], "entities": ["Google", "Attention Is All You Need"], "alternates": {"html": "https://wpnews.pro/news/attention-is-all-you-need-with-a-real-transformer-running-in-the-page", "markdown": "https://wpnews.pro/news/attention-is-all-you-need-with-a-real-transformer-running-in-the-page.md", "text": "https://wpnews.pro/news/attention-is-all-you-need-with-a-real-transformer-running-in-the-page.txt", "jsonld": "https://wpnews.pro/news/attention-is-all-you-need-with-a-real-transformer-running-in-the-page.jsonld"}}