{"slug": "understanding-annotator-safety-policy-with-interpretability", "title": "Understanding Annotator Safety Policy with Interpretability", "summary": "Researchers at Apple introduced Annotator Policy Models (APMs), interpretable models that learn annotators' internal safety policies from labeling behavior alone, enabling diagnosis of disagreement sources such as operational failures, policy ambiguity, and value pluralism without additional annotation effort. The models achieve over 80% accuracy in modeling annotator safety policy and can surface systematic differences in safety priorities across demographic groups, supporting more targeted and inclusive AI safety policy design.", "body_md": "[content type paper](/research/)published July 2026\n\nUnderstanding Annotator Safety Policy with Interpretability\n\nAuthorsAlex Oesterling†**, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys‡, Fred Hohman‡\n\nUnderstanding Annotator Safety Policy with Interpretability\n\nAuthorsAlex Oesterling†**, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys‡, Fred Hohman‡\n\nSafety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these sources matters. For example, operational failures call for quality control, ambiguity calls for policy clarification, and pluralism calls for deliberation about incorporating diverse perspectives. Yet understanding why annotators disagree is difficult. Directly asking annotators for their reasoning is costly, substantially increasing annotation burden, and can be unreliable for both human and LLM annotators as self-reported reasoning often fails to reflect actual decision processes. We introduce Annotator Policy Models (APMs), interpretable models that learn annotators’ internal safety policies from labeling behavior alone, making annotator reasoning visible and comparable without additional annotation effort. We validate that APMs accurately model annotator safety policy (>80% accuracy), faithfully predict responses to counterfactual edits, and recover known policy differences in controlled settings. Applying APMs to LLM and human annotations, we demonstrate two core applications: (1) surfacing policy ambiguity by revealing how annotators interpret safety instructions differently, and (2) surfacing value pluralism by uncovering systematic differences in safety priorities across demographic groups. Together, these capabilities support more targeted, transparent, and inclusive safety policy design.\n\n**Figure 1:** Annotator Policy Models (APMs) learn interpretable representations of individual annotator safety policies. APMs are trained on annotation behavior to reveal how different annotators operationalize safety, enabling diagnosis of disagreement sources. By mapping annotators to a shared feature space, APMs make systematic comparison possible: identifying where annotators may have misunderstood the task itself (identifying operational failures), where they interpret instructions differently (surfacing policy ambiguity), or where they systematically differ by demographic group (surfacing value pluralism).\n\nSafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation\n\nMarch 24, 2026[research area Computer Vision](/research/?domain=Computer%20Vision), [research area Speech and Natural Language Processing](/research/?domain=Speech%20and%20Natural%20Language%20Processing)[Workshop at ICLR](/research/?event=ICLR%20Workshop)\n\nThis paper was accepted at the Principled Design for Trustworthy AI — Interpretability, Robustness, and Safety across Modalities Workshop at ICLR 2026.\n\nWhat exactly makes a particular image unsafe? Systematically differentiating between benign and problematic images is a challenging problem, as subtle changes to an image, such as an insulting gesture or symbol, can drastically alter its safety implications. However, existing image safety…\n\nPolicy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors\n\nNovember 3, 2025[research area Data Science and Annotation](/research/?domain=Data%20Science%20and%20Annotation), [research area Human-Computer Interaction](/research/?domain=Human-Computer%20Interaction)[conference UIST](/research/?event=UIST)\n\nAI policy sets boundaries on acceptable behavior for AI models, but this is challenging in the context of large language models (LLMs): how do you ensure coverage over a vast behavior space? We introduce policy maps, an approach to AI policy design inspired by the practice of physical mapmaking. Instead of aiming for full coverage, policy maps aid effective navigation through intentional design choices about which aspects to capture and which to…", "url": "https://wpnews.pro/news/understanding-annotator-safety-policy-with-interpretability", "canonical_source": "https://machinelearning.apple.com/research/annotator-safety-policy-interpretability", "published_at": "2026-07-06 00:00:00+00:00", "updated_at": "2026-07-07 00:36:16.886324+00:00", "lang": "en", "topics": ["ai-safety", "ai-ethics", "machine-learning"], "entities": ["Apple", "Alex Oesterling", "Donghao Ren", "Yannick Assogba", "Dominik Moritz", "Sunnie S. Y. Kim", "Leon Gatys", "Fred Hohman"], "alternates": {"html": "https://wpnews.pro/news/understanding-annotator-safety-policy-with-interpretability", "markdown": "https://wpnews.pro/news/understanding-annotator-safety-policy-with-interpretability.md", "text": "https://wpnews.pro/news/understanding-annotator-safety-policy-with-interpretability.txt", "jsonld": "https://wpnews.pro/news/understanding-annotator-safety-policy-with-interpretability.jsonld"}}