GPT makes a poor AMR parser

GPT-3.5 and GPT-4o fall well short of state-of-the-art Abstract Meaning Representation (AMR) parsers, topping out at a maximum Smatch score of 60 with GPT-4o in a 5-shot setting, according to a study by Yanming Li and Meaghan Fowlie published in the Journal for Language Technology and Computational Linguistics (Vol. 38 No. 2, pages 43–76, July 2025). The researchers evaluated prompt-based strategies including 0-shot, few-shot, Chain-of-Thought (CoT), and a two-step approach splitting core arguments from non-core roles, finding that CoT prompting aided interpretability but did not improve performance. Fine-grained evaluations showed GPT's limited handling of AMR-specific linguistic structures and complex semantic roles, leading the authors to conclude GPT models are not yet suitable as standalone AMR parsers.

Abstract This paper evaluates GPT models as out-of-the-box Abstract Meaning Representation AMR parsers using prompt-based strategies, including 0-shot, few-shot, Chain-of-Thought CoT , and a two-step approach in which core arguments and non-core roles are handled separately. Our results show that GPT-3.5 and GPT-4o fall well short of state-of-the-art parsers, with a maximum Smatch score of 60 using GPT-4o in a 5-shot setting. While CoT prompting provides some interpretability, it does not improve performance. We further conduct fine-grained evaluations, revealing GPT’s limited ability to handle AMR-specific linguistic structures and complex semantic roles. Our findings suggest that, despite recent advances, GPT models are not yet suitable as standalone AMR parsers. - Anthology ID: - 2025.jlcl-2.4 - Volume: - Journal for Language Technology and Computational Linguistics, Vol. 38 No. 2: LLM Fails - Failed experiments with generative AI and what we can learn from them https://aclanthology.org/volumes/2025.jlcl-2/ - Month: - July - Year: - 2025 - Address: - Editors: - Ngoc Duyen Tanja Tu https://aclanthology.org/people/ngoc-duyen-tanja-tu/unverified/ , Annelen Brunner https://aclanthology.org/people/annelen-brunner/unverified/ , Christian Lang https://aclanthology.org/people/christian-lang/unverified/ - Venue: - JLCL https://aclanthology.org/venues/jlcl/ - SIG: - Publisher: - German Society for Computational Linguistics and Language Technology - Note: - Pages: - 43–76 - Language: - URL: - https://aclanthology.org/2025.jlcl-2.4/ https://aclanthology.org/2025.jlcl-2.4/ - DOI: - 10.21248/jlcl.38.2025.285 https://doi.org/10.21248/jlcl.38.2025.285 - Cite ACL : - Yanming Li and Meaghan Fowlie. 2025. GPT makes a poor AMR parser https://aclanthology.org/2025.jlcl-2.4/ . Journal for Language Technology and Computational Linguistics , 38 2 :43–76. - Cite Informal : - GPT makes a poor AMR parser https://aclanthology.org/2025.jlcl-2.4/ Li & Fowlie, JLCL 2025 - PDF: - https://aclanthology.org/2025.jlcl-2.4.pdf https://aclanthology.org/2025.jlcl-2.4.pdf