LayerNorm

A developer explains the difference between LayerNorm and BatchNorm, showing that LayerNorm normalizes per token rather than per batch, which makes it ideal for Transformers. The post includes an inte…

A new study proves the Spectral Alignment Decomposition, which explains why the curvature exponent $\alpha$ — governing how Hessian eigenvalues scale with gradient singular values — varies across neur…