Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention

A new study from arXiv:2607.15449v1 applies renormalization group theory to Transformer attention, finding that attention's relevance depends on data correlation length. For long-correlation sequences, attention is a relevant operator that drives a phase transition in representation space, while for short-correlation sequences it is irrelevant. The first-layer head (L0H0) dominates the representational shift, accounting for over 4 times the shift of any subsequent head.

arXiv:2607.15449v1 Announce Type: new Abstract: Using the language of Wilsonian renormalization group theory RG , we treat the Transformer's attention mechanism as a perturbation of the trained MLP residual-stack fixed point and ask whether it constitutes a relevant, marginal, or irrelevant operator. We derive a fixed-point shift formula and obtain four testable predictions for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum. Testing these on synthetic Markov chain sequences with controlled correlation length, we find: 1 For large chains long correlation , attention is strongly relevant: it closes a residual loss gap the MLP cannot bridge and drives a phase transition in representation space, with effective rank jumping above input dimensionality at layer 1 and stabilizing at a high-dimensional plateau. 2 For short chains short correlation , attention is irrelevant: the Transformer converges to the same loss and fixed-point geometry as the MLP, though it contracts perturbations faster. 3 The transition is dominated by the first-layer head L0H0 , which accounts for more than 4 times the representational shift of any subsequent head, consistent with the prediction that the relevant operator acts before the MLP begins integrating out positional variation. 4 Perturbation decay experiments reveal a regime reversal: in the long correlation regime the Transformer selectively preserves slow Markov modes 5.4 times the dynamic range in decay length vs. 1.3 times for the MLP ; in the short correlation regime it suppresses all modes faster than the MLP, with no spectral selectivity. Together, these results show that the relevance of attention is not a property of the architecture but of the spectral structure of the data-generating process, and that a first-order RG perturbation framework provides a predictive account of that difference.