A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition

Elena Volkanovska published a paper in the Journal for Language Technology and Computational Linguistics (Vol. 38 No. 2, pages 31–42, July 2025) proposing an error classification framework for analyzing large language model output in domain-specific few-shot named entity recognition. The framework is presented as an exploratory analysis complementary to established NER metrics such as F1 score, categorizing and quantitatively assessing incorrect named entity predictions to support cross-model and cross-prompt performance comparison and a roadmap for guided qualitative error analysis.

Abstract This paper proposes an error classification framework for a comprehensive analysis of the output that large language models LLMs generate in a few-shot named entity recognition NER task in a specialised domain. The framework should be seen as an exploratory analysis complementary to established performance metrics for NER classifiers, such as F1 score, as it accounts for outcomes possible in a few-shot, LLM-based NER task. By categorising and assessing incorrect named entity predictions quantitatively, the paper shows how the proposed error classification could support a deeper cross-model and cross-prompt performance comparison, alongside a roadmap for a guided qualitative error analysis. - Anthology ID: - 2025.jlcl-2.3 - Volume: - Journal for Language Technology and Computational Linguistics, Vol. 38 No. 2: LLM Fails - Failed experiments with generative AI and what we can learn from them https://aclanthology.org/volumes/2025.jlcl-2/ - Month: - July - Year: - 2025 - Address: - Editors: - Ngoc Duyen Tanja Tu https://aclanthology.org/people/ngoc-duyen-tanja-tu/unverified/ , Annelen Brunner https://aclanthology.org/people/annelen-brunner/unverified/ , Christian Lang https://aclanthology.org/people/christian-lang/unverified/ - Venue: - JLCL https://aclanthology.org/venues/jlcl/ - SIG: - Publisher: - German Society for Computational Linguistics and Language Technology - Note: - Pages: - 31–42 - Language: - URL: - https://aclanthology.org/2025.jlcl-2.3/ https://aclanthology.org/2025.jlcl-2.3/ - DOI: - 10.21248/jlcl.38.2025.281 https://doi.org/10.21248/jlcl.38.2025.281 - Cite ACL : - Elena Volkanovska. 2025. A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition https://aclanthology.org/2025.jlcl-2.3/ . Journal for Language Technology and Computational Linguistics , 38 2 :31–42. - Cite Informal : - A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition https://aclanthology.org/2025.jlcl-2.3/ Volkanovska, JLCL 2025 - PDF: - https://aclanthology.org/2025.jlcl-2.3.pdf https://aclanthology.org/2025.jlcl-2.3.pdf