Analysis of Metastable States in the Transformer Activation Space
type: NewsArticle
description: A new analysis of metastable states in transformer activation spaces confirms that token representations cluster into metastable groups across layers in trained models, as predicted by a recent dynamical systems theory, but falsifies the theory's proposed mechanism. Researchers found that the interaction energy driving the process fails to increase monotonically in every model and prompt tested, while collapse speed is governed by the spectral radius of the value matrix rather than model dimension. The findings, based on experiments with real transformers, validate three core predictions—clustering, persistence, and separate two-timescale dynamics—while revealing that the theory's assumptions about energy monotonicity and depth-driven collapse do not hold in practice.
date published: 2026-06-06 21:30:13+00:00
date modified: 2026-06-06 21:48:58.290928+00:00
language: en
is accessible for free: true
article body: This is the first entry in a sequence.Over about ten parts, this series will work through a few humble experiments that test a mathematical theory of attention against real trained transformers.A project summary:a recent paper by Geshkovski, Letrouit, Polyanskiy, and Rigollet models attention as a dynamical system on the sphere and proves that tokens cluster and drift toward consensus, with a metastable two-timescale structure along the way — under the assumption that Q, K, and V are all the identity. This sequence asks how much of that survives in trained models (most of it does), finds the one prediction that failsuniversally(the energy is not monotone), and then spends the rest of the series tracking that failure to its mechanism in the value matrix. Links:[all the code for every phase.][GitHub repository]:[Geshkovski et al., the theory this whole investigation is built on][Original paper]:[a video going over the original paper for those who prefer video][YouTube walkthrough]: This is the first in what will be a sequence of ~10 posts. Each post will be aligned with a phase from the github. **Metastability is real in trained transformers — but the mechanism the theory proposes for it is not.** Tokens cluster into metastable groups the way the idealized model predicts; the energy that theory says drives the process is violated in every model tested; and how fast a model collapses is set by its value matrix, not by depth or width as the theory claims. Three predictions of the idealized theory survive the move to trained models, and survive cleanly. Token representations **cluster** over layers (CONFIRMED, every model, every real prompt). The clusters **persist** across runs of layers — metastable plateaus — before reorganizing (CONFIRMED). And the two timescales of metastability — fast formation, slow merging — are **genuinely separate** above a depth threshold (CONFIRMED). Two predictions break. The interaction energy the theory requires to **increase monotonically…
publisher: Web Pulse
keywords: artificial-intelligence, machine-learning, large-language-models, neural-networks, ai-research
mentions: Geshkovski
mentions: Letrouit
mentions: Polyanskiy
mentions: Rigollet
citation: CreativeWork