{"slug": "improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai", "title": "Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation", "summary": "A new AI-driven workflow that uses detailed \"constitutions\" to define labeling categories and a frontier LLM to interpret them has reduced cross-model inconsistency by up to 57 times compared to standard paragraph definitions in content moderation tasks. The approach, tested on harassment, hate speech, and non-violent crime categories, shifts human responsibility from individual labeling decisions to high-level oversight of category meaning. The system also introduces a dual-axis safety evaluation that scores intent and content independently across full conversations, enabling downstream consumers to act on either dimension.", "body_md": "arXiv:2605.24247v1 Announce Type: new\nAbstract: Many automated labeling pipelines classify inputs into categories defined by a written specification, content moderation being a prominent use case. Simple category definitions are not detailed enough for labelers to produce the accurate, consistent golden labels these pipelines require. One solution is to write a prescriptive definition that settles enough real boundary cases that labelers cannot disagree with the written interpretation. In practice, definitions at that level of detail exceed what a human annotator can hold in working memory, so annotators fall back on intuition and the labels drift from the written rules, regressing on accuracy and consistency.\nWe propose and demonstrate the efficacy of an AI-driven workflow in which AI helps write a per-category constitution that defines the label in enough detail to cover edge cases, and a frontier LLM interprets it on each input to produce the golden label more consistently and accurately than humans reading the same document. We evaluate on three content moderation categories (harassment, hate speech, non-violent crime) and show that the approach reduces cross-model inconsistency by up to 57x compared to paragraph definitions, with cross-model disagreement diagnosing specification gaps and the human responsible for high-level decisions about what each category should mean rather than individual labeling calls. For the safety evaluation, we introduce a dual-axis formulation scoring intent and content independently over the full conversation, so downstream consumers can act on either axis or both.", "url": "https://wpnews.pro/news/improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai", "canonical_source": "https://arxiv.org/abs/2605.24247", "published_at": "2026-05-26 04:00:00+00:00", "updated_at": "2026-05-26 04:15:46.485452+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-safety", "ai-ethics"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai", "markdown": "https://wpnews.pro/news/improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai.md", "text": "https://wpnews.pro/news/improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai.txt", "jsonld": "https://wpnews.pro/news/improving-labeling-consistency-with-detailed-constitutional-definitions-and-ai.jsonld"}}