{"slug": "rmsnorm", "title": "RMSNorm", "summary": "RMSNorm, a normalization technique proposed by Zhang and Sennrich in 2019, rescales neural network activations by dividing by the root mean square of each row, skipping the mean-centering step of LayerNorm. PyTorch 2.4 and later ships torch.nn.RMSNorm, which performs the same computation. The technique cuts layer normalization's runtime by 7-64% depending on the model, with minimal impact on quality, and is used in modern language models like Llama and Mistral.", "body_md": "# RMSNorm\n\nMost modern language models, like Llama and Mistral, normalize activations with **Root Mean Square Normalization (RMSNorm)**. Here is what it does, and a PyTorch module you can drop into a model.\n\n### The problem it solves\n\nInside a neural network, a layer’s output feeds the next layer as input. Those values can become too big or too small as they pass through layer after layer. A value that doubles at each of 40 layers is unusable by the end. Normalization rescales the values back to a steady range before they move on, so training stays stable.\n\n### What RMSNorm does\n\n$$ \\mathrm{RMS}(x) = \\sqrt{\\frac{1}{n}\\sum_{i=1}^{n} x_i^2 + \\epsilon} $$\n\n$$ y_i = \\frac{x_i}{\\mathrm{RMS}(x)} \\gamma_i $$\n\nTake a vector $x$ of $n$ values. Square each value, sum them, average the squares, take the square root. That gives the result of $\\mathrm{RMS}(x)$, one number for the row’s typical size.\n\n$\\epsilon$ is a tiny constant so $x_i$ never divides by zero. Divide every value in the row by $\\mathrm{RMS}(x)$, and the row’s own RMS becomes about 1. Multiply by $\\gamma$, a learned weight for each position, to get the output $y$.\n\nExample: row $x = (2, -4, 6, -8)$. $\\mathrm{RMS}(x) = \\sqrt{30} \\approx 5.48$. With weights $\\gamma = (1.5, 0.5, 1.0, 2.0)$, the output is $y = (0.55, -0.37, 1.10, -2.92)$.\n\n### The code\n\n``` python\nimport torch\nimport torch.nn as nn\n\nclass RMSNorm(nn.Module):\n    def __init__(self, dim: int, eps: float = 1e-6):\n        super().__init__()\n        self.eps = eps\n        self.weight = nn.Parameter(torch.ones(dim))\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)\n        return (x / rms) * self.weight\n```\n\n`dim`\n\nis the length of the row being normalized, the model’s hidden size in a transformer. `self.weight`\n\nstarts at all 1s, so a freshly initialized `RMSNorm`\n\npasses its input through unchanged (aside from the rescaling), and training moves each weight away from 1 as needed.\n\n`forward`\n\nruns as the same two formulas above.\n\nThe RMS division and the weight multiplication both broadcast: `rms`\n\nhas shape `(..., 1)`\n\nand `self.weight`\n\nhas shape `(dim,)`\n\n, so each applies to every row of `x`\n\nwithout a loop.\n\nRunning it on a random batch:\n\n```\ntorch.manual_seed(0)\nx = torch.randn(2, 8) * 10\nnorm = RMSNorm(8)\ny = norm(x)\nprint(y.pow(2).mean(dim=-1).sqrt())\ntensor([1.0000, 1.0000], grad_fn=<SqrtBackward0>)\n```\n\nTwo rows, each starting on a different scale, both land at an RMS of 1 after the call. [ torch.nn.RMSNorm](https://docs.pytorch.org/docs/2.13/generated/torch.nn.RMSNorm.html), shipped in PyTorch 2.4 and later, does the same computation.\n\n### RMSNorm vs. LayerNorm\n\nLayerNorm does one more step: before dividing, it subtracts the row’s average from every value. This centers the row on 0. RMSNorm skips that. It rescales, but never re-centers.\n\nZhang and Sennrich, who [proposed RMSNorm in 2019](https://arxiv.org/abs/1910.07467), measured how much the centering step in LayerNorm actually mattered. Across machine translation and language modeling, dropping it barely moved model quality, while removing the mean and variance calculation cut layer normalization’s runtime by 7-64%, depending on the model.\n\nFull code: [gist.github.com/soasme/rms_norm.py](https://gist.github.com/soasme/59975740c3130362e105b8112d42603a) (module in `rms_norm.py`\n\n, the demo above in `demo.py`\n\n).", "url": "https://wpnews.pro/news/rmsnorm", "canonical_source": "https://julin.ai/2026/09/02/rms-norm/", "published_at": "2026-09-01 12:00:00+00:00", "updated_at": "2026-09-01 22:22:30.741674+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "large-language-models"], "entities": ["RMSNorm", "PyTorch", "LayerNorm", "Zhang", "Sennrich", "Llama", "Mistral"], "alternates": {"html": "https://wpnews.pro/news/rmsnorm", "markdown": "https://wpnews.pro/news/rmsnorm.md", "text": "https://wpnews.pro/news/rmsnorm.txt", "jsonld": "https://wpnews.pro/news/rmsnorm.jsonld"}}