📏 Play with the LayerNorm vs BatchNorm visualizer: https://dev48v.infy.uk/dl/day27-layer-norm.html
Normalization inside a neural net is almost embarrassingly simple: take some numbers, subtract their mean, divide by their standard deviation, then rescale with two learnable knobs. Every normalization layer you have ever used does exactly that. The only thing separating BatchNorm from LayerNorm is one deceptively small decision: which numbers do you average over?
Get that decision right and the layer drops straight into a Transformer. Get it wrong and your model falls apart the moment you feed it a single example.
Both layers compute the same two lines:
xhat = (x - mean) / sqrt(var + eps)
y = gamma * xhat + beta
Standardize to mean 0 and variance 1, then apply a learnable scale (gamma) and shift (beta) so the network can rescale, or even undo, the normalization if that helps the task. The epsilon just stops a divide-by-zero. Identical maths. Different axis.
BatchNorm takes each feature and computes its mean and variance across every sample in the mini-batch. Picture your activations as a grid: rows are samples, columns are features. BatchNorm normalizes each column, using every row to do it.
That works beautifully for convolutional vision models and was a big reason very deep CNNs became trainable. But it has a catch that turns fatal for sequence models: the statistics depend on the rest of the batch.
LayerNorm (Ba, Kiros and Hinton, 2016) normalizes the other way: for each sample, it averages across that sample's own features. In grid terms, it normalizes each row using only the values in that row.
The consequence is the whole point: the result for one example never touches any other example. Batch size becomes irrelevant. A batch of 1000 or a batch of 1 gives the same maths. Training and inference give the same maths, with no running averages to maintain.
Flip the toggle in the visualizer and set the batch size to 1. BatchNorm collapses every cell to beta. LayerNorm keeps working, standardizing the lone sample across its six features to mean 0, variance 1. That single screen is the entire argument for why Transformers use LayerNorm.
A Transformer activation has shape (batch, seq_len, d_model)
. Each token is one sample of d_model
features. nn.LayerNorm(d_model)
normalizes every token independently over its own features, so batch size, sequence length and padding simply do not matter. Sequences of 7 tokens or 4000 tokens, a batch of 1 during generation or 256 during training: the layer behaves the same. BatchNorm can promise none of that.
There is a second choice: where the norm goes inside a residual block.
x = ln(x + sublayer(x)) # needs LR warmup, harder to train deep
x = x + sublayer(ln(x)) # clean residual highway, very stable
Post-norm was the original design, but pre-norm, which normalizes the input to each sublayer and leaves the residual add as a clean identity path, trains deep stacks far more reliably and is the default in most modern models.
Recent LLMs (LLaMA, Mistral and most 2023+ models) push it further with RMSNorm. It drops the mean-centering and the beta shift entirely, keeping only division by the root-mean-square of the features and a learnable scale:
rms = sqrt(mean(x^2))
y = gamma * x / rms
Fewer operations, no mean to compute, and in practice the quality matches LayerNorm. It turns out the re-centering was doing less work than everyone assumed.
Normalization is one recipe with one free choice. BatchNorm averages across the batch and pays for it with batch dependence. LayerNorm averages across features per sample and buys batch independence, train/eval consistency and clean per-token behavior: exactly the properties a Transformer needs. That is why the norm humming quietly inside every large language model is Layer Normalization, not Batch Normalization.
📏 Watch it flatten row by row: https://dev48v.infy.uk/dl/day27-layer-norm.html
Part of DeepLearningFromZero. 🌐 https://dev48v.infy.uk