Abstract
This paper investigates the evaluation of compositional generalization in Transformer models on the ReCOGS benchmark. The problem addressed is that ReCOGS relies on Semantic Exact Match, a binary metric that assigns the same penalty to minor local mismatches and severe structural errors, limiting diagnostic interpretation. To address this, the study introduces Compositional Graph Similarity (CGS), a graph-based metric that compares predicted and reference semantic structures through explicit edit operations, providing graded and interpretable structural evaluation. The work also uses controlled synthetic datasets to test whether low-scoring ReCOGS categories reflect true model limitations or weaknesses in dataset coverage. Empirical results show that CGS satisfies all seven quality criteria adopted for graph similarity and identifies the lowest-scoring ReCOGS categories as cp recursion (45.0%), obj pp to subj pp (65.4%), and prim to inf arg (66.7%). Follow-up experiments showed 0% Semantic Exact Match under depth extrapolation and constituent-role relocation, but 99.9% Semantic Exact Match for prim to inf arg in isolation. These findings support the conclusion that CGS is more informative than Semantic Exact Match and that Transformer limitations in ReCOGS are partly structural and partly induced by dataset distribution.- Anthology ID:
- 2026.brigap-1.4
- Volume:
[Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics (BriGap-3)](/volumes/2026.brigap-1/)- Month:
- July
- Year:
- 2026
- Address:
- Paris, France
- Editors:
[Timothée Bernard](/people/timothee-bernard/),[Emmanuele Chersoni](/people/emmanuele-chersoni/),[Giulia Rambelli](/people/giulia-rambelli/unverified/)- Venues:
[BriGap](/venues/brigap/)|[WS](/venues/ws/)- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 31–39
- Language:
- URL:
[https://aclanthology.org/2026.brigap-1.4/](https://aclanthology.org/2026.brigap-1.4/)- DOI:
- Cite (ACL):
- Bruno Franco and Edson Scalabrin. 2026. Diagnosing Compositional Generalization in Transformers on ReCOGS with Compositional Graph Similarity. InProceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics (BriGap-3), pages 31–39, Paris, France. Association for Computational Linguistics. - Cite (Informal):
[Diagnosing Compositional Generalization in Transformers on ReCOGS with Compositional Graph Similarity](https://aclanthology.org/2026.brigap-1.4/)(Franco & Scalabrin, BriGap 2026)- PDF:
[https://aclanthology.org/2026.brigap-1.4.pdf](https://aclanthology.org/2026.brigap-1.4.pdf)