{"slug": "consensus-measures-for-unstructured-biomedical-text-annotations", "title": "Consensus Measures for Unstructured Biomedical Text Annotations", "summary": "A new arXiv preprint (2608.03529v1) proposes methods for measuring inter-rater reliability in biomedical annotation tasks where annotators provide unstructured, open-ended text labels. The study finds that soft reliability can be quantified using semantic equivalence measures, with embeddings being scalable but limited in differentiating similar concepts, large language models promising but limited by scalability, and natural language inference measures offering a sensible compromise.", "body_md": "arXiv:2608.03529v1 Announce Type: new\nAbstract: Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.", "url": "https://wpnews.pro/news/consensus-measures-for-unstructured-biomedical-text-annotations", "canonical_source": "https://www.machinebrief.com/news/consensus-measures-for-unstructured-biomedical-text-annotati-7g2m", "published_at": "2026-08-05 04:00:00+00:00", "updated_at": "2026-08-05 06:35:43.871641+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "artificial-intelligence"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/consensus-measures-for-unstructured-biomedical-text-annotations", "markdown": "https://wpnews.pro/news/consensus-measures-for-unstructured-biomedical-text-annotations.md", "text": "https://wpnews.pro/news/consensus-measures-for-unstructured-biomedical-text-annotations.txt", "jsonld": "https://wpnews.pro/news/consensus-measures-for-unstructured-biomedical-text-annotations.jsonld"}}