Stability of Transformers under Layer Normalization

A new study from arXiv (arXiv:2510.09904v2) provides a theoretical framework for understanding the stability of Transformers under different layer normalization placements, deriving explicit bounds on hidden state growth and analyzing gradient backpropagation to explain training dynamics. The findings guide scaling of residual steps to improve stability and performance, offering a principled method for sanity-checking future architectural modifications.

arXiv:2510.09904v2 Announce Type: replace Abstract: Despite their widespread use, training deep Transformers can be unstable. Layer normalization, a standard component, improves training stability, but its placement has often been ad-hoc. In this paper, we conduct a principled study on the forward hidden states and backward gradient stability of Transformers under different layer normalization placements. Our theory provides key insights into the training dynamics: whether training drives Transformers toward regular solutions or pathological behaviors. For forward stability, we derive explicit bounds on the growth of hidden states in trained Transformers. For backward stability, we analyze how layer normalization affects the backpropagation of gradients, thereby explaining the training dynamics of each layer normalization placement. Our analysis also guides the scaling of residual steps in Transformer blocks, where appropriate choices can further improve stability and performance. Our numerical results corroborate our theoretical findings. Beyond these results, our framework provides a principled way to sanity-check the stability of Transformers under new architectural modifications, offering guidance for future designs.