{"slug": "an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via", "title": "An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars", "summary": "Researchers have theoretically analyzed how deep transformers represent hierarchical structures in language, constructing models whose depth grows linearly with grammar depth and whose neuron count scales with derivation-tree shapes and production rules. The work supports the linear representation hypothesis, showing transformers can encode abstract grammatical states into low-dimensional subspaces.", "body_md": "arXiv:2606.17522v1 Announce Type: new\nAbstract: Deep neural networks are widely believed to derive their expressive power from their ability to form \\textbf{hierarchical representations}, capturing progressively more abstract and compositional features across layers. In language modeling, \\textbf{transformers} have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies. While this intuition has shaped model design, there remains a lack of rigorous theoretical work demonstrating \\textbf{how} deep transformers represent such hierarchical structures. In this work, we analyze the expressiveness of deep transformer models through the formal lens of bounded-depth, non-recursive context-free grammars. For this class of grammars, we explicitly construct transformers with positional attention whose depth grows linearly with grammar depth, while the neuron count scales with the number of derivation-tree shapes and quadratically with the number of production rules. Our theoretical results support the linear representation hypothesis by demonstrating that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.", "url": "https://wpnews.pro/news/an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via", "canonical_source": "https://arxiv.org/abs/2606.17522", "published_at": "2026-06-17 04:00:00+00:00", "updated_at": "2026-06-17 04:28:06.358086+00:00", "lang": "en", "topics": ["large-language-models", "neural-networks", "natural-language-processing", "machine-learning"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via", "markdown": "https://wpnews.pro/news/an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via.md", "text": "https://wpnews.pro/news/an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via.txt", "jsonld": "https://wpnews.pro/news/an-expressivity-analysis-of-hierarchical-modelling-in-deep-transformers-via.jsonld"}}