{"slug": "the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization", "title": "The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers", "summary": "A new arXiv preprint (2608.25166v1) reports that the grammatical role of tokens, marked by part-of-speech tags, shapes the local geometry of Transformer representations, with closed-class items expanding earlier and collapsing sooner than open-class ones across layers. The study compares encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently, and shows that geometric features alone can recover a token's grammatical role.", "body_md": "arXiv:2608.25166v1 Announce Type: new\nAbstract: Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.", "url": "https://wpnews.pro/news/the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization", "canonical_source": "https://arxiv.org/abs/2608.25166", "published_at": "2026-08-27 04:00:00+00:00", "updated_at": "2026-08-27 04:20:22.882645+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv", "ModernBERT", "bigbird-roberta-large", "gemma-2-2B", "Llama-3.2-3B"], "alternates": {"html": "https://wpnews.pro/news/the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization", "markdown": "https://wpnews.pro/news/the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization.md", "text": "https://wpnews.pro/news/the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization.txt", "jsonld": "https://wpnews.pro/news/the-changing-geometry-of-grammar-dimensionality-and-neighborhood-reorganization.jsonld"}}