{"slug": "geometric-and-behavioral-stratification-in-transformer-residual-streams", "title": "Geometric and Behavioral Stratification in Transformer Residual Streams", "summary": "A new arXiv preprint (2608.12447v1) reports that in all eighteen transformer models tested (dense and mixture-of-experts, 7B-120B, base and instruction-tuned), residual-stream variation is geometrically and behaviorally stratified by proximity to the prediction direction, which acts as a privileged anchor. The prediction-proximal region is highly structured and causally decisive, while the distal complement is flatter but load-bearing, and variance-based analyses recover this organization only partly.", "body_md": "arXiv:2608.12447v1 Announce Type: new\nAbstract: Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream. But what kind of direction does such a basis select? We investigate the prediction direction, the unembedding direction of the token a model currently predicts, and find that it functions as a content-defined privileged anchor. Measured with respect to this anchor, residual-stream variation is geometrically and behaviorally stratified by proximity to the prediction.\nThe stratification holds in all eighteen models tested (dense and mixture-of-experts, 7B-120B, base and instruction-tuned). A narrow, scale-invariant prediction interface concentrates readout-relevant structure, while the vast prediction-distal complement expands with model scale. Because the prediction direction sits nearly orthogonal to the principal variance axes, variance-based analyses recover this organization only partly, and the shortfall grows with prompt heterogeneity.\nAnchoring reveals a steep geometric gradient: prediction-proximal regions are highly structured and cluster related prompts, while the complement is flatter and anti-discriminates among prompt groups. The interface is a narrow slice but functionally decisive. Disrupting the variance directions closest to the prediction causes immediate divergence and frequent task-frame shifts; disrupting the next level down delays divergence and preserves framing. The complement is weakly readout-aligned per direction yet causally and temporally load-bearing, and behavior is driven by direction rather than magnitude.\nThese results establish the prediction direction as a privileged anchor distinct from previously described coordinate axes, and give a geometric account of how high-dimensional computation coexists with linear readout.", "url": "https://wpnews.pro/news/geometric-and-behavioral-stratification-in-transformer-residual-streams", "canonical_source": "https://arxiv.org/abs/2608.12447", "published_at": "2026-08-14 04:00:00+00:00", "updated_at": "2026-08-14 04:15:53.408764+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "neural-networks", "ai-research"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/geometric-and-behavioral-stratification-in-transformer-residual-streams", "markdown": "https://wpnews.pro/news/geometric-and-behavioral-stratification-in-transformer-residual-streams.md", "text": "https://wpnews.pro/news/geometric-and-behavioral-stratification-in-transformer-residual-streams.txt", "jsonld": "https://wpnews.pro/news/geometric-and-behavioral-stratification-in-transformer-residual-streams.jsonld"}}