GPT makes a poor AMR parser

Abstract

This paper evaluates GPT models as out-of-the-box Abstract Meaning Representation (AMR) parsers using prompt-based strategies, including 0-shot, few-shot, Chain-of-Thought (CoT), and a two-step approach in which core arguments and non-core roles are handled separately. Our results show that GPT-3.5 and GPT-4o fall well short of state-of-the-art parsers, with a maximum Smatch score of 60 using GPT-4o in a 5-shot setting. While CoT prompting provides some interpretability, it does not improve performance. We further conduct fine-grained evaluations, revealing GPT’s limited ability to handle AMR-specific linguistic structures and complex semantic roles. Our findings suggest that, despite recent advances, GPT models are not yet suitable as standalone AMR parsers.

- Anthology ID:
- 2025.jlcl-2.4
- Volume:

• Journal for Language Technology and Computational Linguistics, Vol. 38 No. 2: LLM Fails - Failed experiments with generative AI and what we can learn from them
• Month:
• July
• Year:
• 2025

- Address:
- Editors:

• Ngoc Duyen Tanja Tu ,Annelen Brunner ,Christian Lang

- Venue:
- [JLCL](https://aclanthology.org/venues/jlcl/)
- SIG:
- Publisher:

• German Society for Computational Linguistics and Language Technology

- Note:
- Pages:

• 43–76

- Language:
- URL:
- [https://aclanthology.org/2025.jlcl-2.4/](https://aclanthology.org/2025.jlcl-2.4/)
- DOI:
- [10.21248/jlcl.38.2025.285](https://doi.org/10.21248/jlcl.38.2025.285)
- Cite (ACL):
- Yanming Li and Meaghan Fowlie. 2025. [GPT makes a poor AMR parser](https://aclanthology.org/2025.jlcl-2.4/) .*Journal for Language Technology and Computational Linguistics* , 38(2):43–76.
- Cite (Informal):
- [GPT makes a poor AMR parser](https://aclanthology.org/2025.jlcl-2.4/) (Li & Fowlie, JLCL 2025)
- PDF:
- [https://aclanthology.org/2025.jlcl-2.4.pdf](https://aclanthology.org/2025.jlcl-2.4.pdf)