A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition

Abstract

This paper proposes an error classification framework for a comprehensive analysis of the output that large language models (LLMs) generate in a few-shot named entity recognition (NER) task in a specialised domain. The framework should be seen as an exploratory analysis complementary to established performance metrics for NER classifiers, such as F1 score, as it accounts for outcomes possible in a few-shot, LLM-based NER task. By categorising and assessing incorrect named entity predictions quantitatively, the paper shows how the proposed error classification could support a deeper cross-model and cross-prompt performance comparison, alongside a roadmap for a guided qualitative error analysis.

- Anthology ID:
- 2025.jlcl-2.3
- Volume:

• Journal for Language Technology and Computational Linguistics, Vol. 38 No. 2: LLM Fails - Failed experiments with generative AI and what we can learn from them
• Month:
• July
• Year:
• 2025

- Address:
- Editors:

• Ngoc Duyen Tanja Tu ,Annelen Brunner ,Christian Lang

- Venue:
- [JLCL](https://aclanthology.org/venues/jlcl/)
- SIG:
- Publisher:

• German Society for Computational Linguistics and Language Technology

- Note:
- Pages:

• 31–42

- Language:
- URL:
- [https://aclanthology.org/2025.jlcl-2.3/](https://aclanthology.org/2025.jlcl-2.3/)
- DOI:
- [10.21248/jlcl.38.2025.281](https://doi.org/10.21248/jlcl.38.2025.281)
- Cite (ACL):

• Elena Volkanovska. 2025. A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition .Journal for Language Technology and Computational Linguistics , 38(2):31–42.

- Cite (Informal):
- [A Study of Errors in the Output of Large Language Models for Domain-Specific Few-Shot Named Entity Recognition](https://aclanthology.org/2025.jlcl-2.3/) (Volkanovska, JLCL 2025)
- PDF:
- [https://aclanthology.org/2025.jlcl-2.3.pdf](https://aclanthology.org/2025.jlcl-2.3.pdf)