{"slug": "gpt-makes-a-poor-amr-parser", "title": "GPT makes a poor AMR parser", "summary": "GPT-3.5 and GPT-4o fall well short of state-of-the-art Abstract Meaning Representation (AMR) parsers, topping out at a maximum Smatch score of 60 with GPT-4o in a 5-shot setting, according to a study by Yanming Li and Meaghan Fowlie published in the Journal for Language Technology and Computational Linguistics (Vol. 38 No. 2, pages 43–76, July 2025). The researchers evaluated prompt-based strategies including 0-shot, few-shot, Chain-of-Thought (CoT), and a two-step approach splitting core arguments from non-core roles, finding that CoT prompting aided interpretability but did not improve performance. Fine-grained evaluations showed GPT's limited handling of AMR-specific linguistic structures and complex semantic roles, leading the authors to conclude GPT models are not yet suitable as standalone AMR parsers.", "body_md": "##### Abstract\n\nThis paper evaluates GPT models as out-of-the-box Abstract Meaning Representation (AMR) parsers using prompt-based strategies, including 0-shot, few-shot, Chain-of-Thought (CoT), and a two-step approach in which core arguments and non-core roles are handled separately. Our results show that GPT-3.5 and GPT-4o fall well short of state-of-the-art parsers, with a maximum Smatch score of 60 using GPT-4o in a 5-shot setting. While CoT prompting provides some interpretability, it does not improve performance. We further conduct fine-grained evaluations, revealing GPT’s limited ability to handle AMR-specific linguistic structures and complex semantic roles. Our findings suggest that, despite recent advances, GPT models are not yet suitable as standalone AMR parsers.\n- Anthology ID:\n- 2025.jlcl-2.4\n- Volume:\n- [Journal for Language Technology and Computational Linguistics, Vol. 38 No. 2: LLM Fails - Failed experiments with generative AI and what we can learn from them](https://aclanthology.org/volumes/2025.jlcl-2/)\n- Month:\n- July\n- Year:\n- 2025\n- Address:\n- Editors:\n- [Ngoc Duyen Tanja Tu](https://aclanthology.org/people/ngoc-duyen-tanja-tu/unverified/) ,[Annelen Brunner](https://aclanthology.org/people/annelen-brunner/unverified/) ,[Christian Lang](https://aclanthology.org/people/christian-lang/unverified/)\n- Venue:\n- [JLCL](https://aclanthology.org/venues/jlcl/)\n- SIG:\n- Publisher:\n- German Society for Computational Linguistics and Language Technology\n- Note:\n- Pages:\n- 43–76\n- Language:\n- URL:\n- [https://aclanthology.org/2025.jlcl-2.4/](https://aclanthology.org/2025.jlcl-2.4/)\n- DOI:\n- [10.21248/jlcl.38.2025.285](https://doi.org/10.21248/jlcl.38.2025.285)\n- Cite (ACL):\n- Yanming Li and Meaghan Fowlie. 2025. [GPT makes a poor AMR parser](https://aclanthology.org/2025.jlcl-2.4/) .*Journal for Language Technology and Computational Linguistics* , 38(2):43–76.\n- Cite (Informal):\n- [GPT makes a poor AMR parser](https://aclanthology.org/2025.jlcl-2.4/) (Li & Fowlie, JLCL 2025)\n- PDF:\n- [https://aclanthology.org/2025.jlcl-2.4.pdf](https://aclanthology.org/2025.jlcl-2.4.pdf)", "url": "https://wpnews.pro/news/gpt-makes-a-poor-amr-parser", "canonical_source": "https://aclanthology.org/2025.jlcl-2.4/", "published_at": "2026-10-14 00:00:00+00:00", "updated_at": "2026-10-16 12:47:30.932668+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research", "machine-learning"], "entities": ["GPT-3.5", "GPT-4o", "Yanming Li", "Meaghan Fowlie", "Journal for Language Technology and Computational Linguistics", "German Society for Computational Linguistics and Language Technology"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gpt-makes-a-poor-amr-parser", "markdown": "https://wpnews.pro/news/gpt-makes-a-poor-amr-parser.md", "text": "https://wpnews.pro/news/gpt-makes-a-poor-amr-parser.txt", "jsonld": "https://wpnews.pro/news/gpt-makes-a-poor-amr-parser.jsonld"}}