attention

Architecture research aims to remove bottlenecks to scaling rather than merely tweaking loss, according to a post by Beren on beren.io. The transformer's attention mechanism and residual stream enable…

Transformer models develop outlier channels—feature dimensions with unusually large values in weights and activations—due to the softmax normalization in attention layers, which forces tokens to assig…