A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers

João Pimentel, Joana Amorim, and Frank Rudzicz published a memory-efficient method for constructing token graphs from Transformer attention coefficients in Computational Linguistics Volume 52, Issue 3 (September 2026), pages 1157–1206. The method uses a chunk-and-stride mechanism to fully encode documents longer than the Transformer's context window while reducing memory usage, and it outperforms fixed-size sliding-window graph construction on longer documents while matching it on shorter ones. The authors report the approach also surpasses GNN-based state-of-the-art techniques, and while it trails fine-tuning marginally on documents that fit inside the pre-trained model's context window, it beats fine-tuning on documents that do not fit.

Abstract Even though Transformers are the standard to extract information from text, they have considerable limitations that make them sub-optimal for certain tasks. Graph neural networks GNNs are an alternative that mitigate some of these limitations, with recent work focusing on combining information from pre-trained language models PLMs with graph structures. However, most existing methods use fixed-size sliding windows to construct graphs, ignoring long-distance relationships between words. We propose a memory-efficient method to construct token graphs that exploits semantic and relational information inside Transformers via their attention coefficients, mitigating the limitations of using sliding windows. Additionally, each graph can fully encode documents longer than the Transformer’s context window via a chunk-and-stride mechanism, while also reducing memory usage. Our method surpasses the performance of sliding windows when both approaches are compared under the same model architecture, particularly in longer documents, and reaches statistically identical performance in shorter ones. It also surpasses GNN-based state-of-the-art techniques. When compared with fine-tuning, our method reaches marginally lower performance in documents that fit inside the pre-trained model’s context window. However, our method surpasses fine-tuning in documents that do not fit inside these windows, which comes as a result of fully encoding each document using our chunk-and-stride mechanism. - Anthology ID: - 2026.cl-3.7 - Volume: - Computational Linguistics, Volume 52, Issue 3 - September 2026 https://aclanthology.org/volumes/2026.cl-3/ - Month: - September - Year: - 2026 - Address: - Cambridge, MA - Venue: - CL https://aclanthology.org/venues/cl/ - SIG: - Publisher: - MIT Press - Note: - Pages: - 1157–1206 - Language: - URL: - https://aclanthology.org/2026.cl-3.7/ https://aclanthology.org/2026.cl-3.7/ - DOI: - 10.1162/coli.a.618 https://doi.org/10.1162/coli.a.618 - Cite ACL : - João Pimentel, Joana Amorim, and Frank Rudzicz. 2026. A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers https://aclanthology.org/2026.cl-3.7/ . Computational Linguistics , 52 3 :1157–1206. - Cite Informal : - A Straightforward Approach to Construct “Lightweight” Token Graphs from Transformers https://aclanthology.org/2026.cl-3.7/ Pimentel et al., CL 2026 - PDF: - https://aclanthology.org/2026.cl-3.7.pdf https://aclanthology.org/2026.cl-3.7.pdf