Using an LLM to Tackle Difficult Cases in NER for Nepali Tweets

A study posted to arXiv (2611.08967v1) found that an LLM surpasses a fine-tuned pretrained language model on the hard subset of a Nepali-tweet named entity recognition test set, even though the LLM underperforms the PLM on the full and easy test sets. Splitting the test set into "easy" and "hard" cases using token probabilities from the fine-tuned PLM, the researchers evaluated the LLM across 16 prompt configurations and found that selecting examples by semantic similarity and increasing the number of examples raised F1 by almost 10 points over the baseline configuration. Combining the two models produced an improved score over either alone.

arXiv:2611.08967v1 Announce Type: new Abstract: Named entity recognition NER is challenging for social media data in a low-resource language. LLMs often lag behind pretrained language models PLMs in NER performance, but they show promise in handling cases that PLMs struggle with. Using a dataset of Nepali tweets, we split the test set into "easy" and "hard" cases using token probabilities from a fine-tuned pretrained language model PLM . We then evaluate an LLM on the hard test cases using 16 different prompt configurations. We find that selecting examples based on semantic similarity and increasing the number of examples improves F1 score by almost 10 points from the baseline configuration. Although the LLM underperforms the PLM on the full and easy test sets, it surpasses the PLM on the hard test set, resulting in an improved score when the two models are combined.