Geshkovski

Experiments building on a mathematical theory of transformers reveal that the architecture drives tokens to cluster and collapse through layers, but trained weights learn to resist this clustering, en…

A new analysis of metastable states in transformer activation spaces confirms that token representations cluster into metastable groups across layers in trained models, as predicted by a recent dynami…