{"slug": "representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware", "title": "Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy", "summary": "A new study on arXiv (2608.27813v1) finds that the accuracy of syntactic tree reconstruction from neural language model representations varies by syntactic relation, and that most of this variability is predicted by the linear distance between related words and the diversity of the relation's head. The results hold across model sizes and architectures, revealing how syntax is abstracted in language models.", "body_md": "arXiv:2608.27813v1 Announce Type: new\nAbstract: Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.", "url": "https://wpnews.pro/news/representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware", "canonical_source": "https://arxiv.org/abs/2608.27813", "published_at": "2026-08-31 04:00:00+00:00", "updated_at": "2026-08-31 04:24:36.509671+00:00", "lang": "en", "topics": ["natural-language-processing", "ai-research"], "entities": ["arXiv", "Hewitt and Manning"], "alternates": {"html": "https://wpnews.pro/news/representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware", "markdown": "https://wpnews.pro/news/representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware.md", "text": "https://wpnews.pro/news/representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware.txt", "jsonld": "https://wpnews.pro/news/representation-of-syntax-in-llms-through-the-lens-of-linear-distance-and-aware.jsonld"}}