The Hidden Battle for Rank in Transformers
Skip connections and normalization layers in Transformer architectures serve a critical function beyond gradient stabilization: they preserve the rank of representations across layers, preventing rank collapse that would…