The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

A new arXiv preprint (2608.25166v1) reports that the grammatical role of tokens, marked by part-of-speech tags, shapes the local geometry of Transformer representations, with closed-class items expanding earlier and collapsing sooner than open-class ones across layers. The study compares encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently, and shows that geometric features alone can recover a token's grammatical role.

arXiv:2608.25166v1 Announce Type: new Abstract: Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality ID , the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech PoS tag, shapes the local geometry of this manifold. To this end: 1 We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; 2 We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; 3 We compare encoders ModernBERT, bigbird-roberta-large and decoders gemma-2-2B, Llama-3.2-3B , finding that the two families evolve differently across layers, consistently with how each integrates context; 4 We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.