Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

A new arXiv paper (2610.33441v1) introduces SCAPE-T, an attention-derived metric built on encoder-only transformers that estimates semantic complexity by combining inverse-weighted self-focus, CLS focus, and inter-head redundancy with layer-weighted aggregation. Under word-length control, SCAPE-T reached 78.5% pairwise accuracy (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons), 12.7 percentage points above the strongest baseline, though readability heuristics overtook it without that control. The authors report SCAPE-T also led on a short-terms abstractness dataset, with ablations showing each submetric contributes materially, and say the method generalizes across two held-out domains.

arXiv:2610.33441v1 Announce Type: new Abstract: Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort readability, lexical difficulty , which correlates with surface features word length, syntax , or quantify surface features directly type-token ratio, word frequency . SCAPE-T targets semantic integration directly by combining three submetrics: i inverse-weighted self-focus token-to-self , ii CLS focus token-to-CLS , and iii inter-head redundancy cosine similarity across heads , using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% 95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline ; without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.