content type paperpublished July 2026 Understanding Annotator Safety Policy with Interpretability
AuthorsAlex Oesterling†**, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys‡, Fred Hohman‡
Understanding Annotator Safety Policy with Interpretability
AuthorsAlex Oesterling†**, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys‡, Fred Hohman‡
Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these sources matters. For example, operational failures call for quality control, ambiguity calls for policy clarification, and pluralism calls for deliberation about incorporating diverse perspectives. Yet understanding why annotators disagree is difficult. Directly asking annotators for their reasoning is costly, substantially increasing annotation burden, and can be unreliable for both human and LLM annotators as self-reported reasoning often fails to reflect actual decision processes. We introduce Annotator Policy Models (APMs), interpretable models that learn annotators’ internal safety policies from labeling behavior alone, making annotator reasoning visible and comparable without additional annotation effort. We validate that APMs accurately model annotator safety policy (>80% accuracy), faithfully predict responses to counterfactual edits, and recover known policy differences in controlled settings. Applying APMs to LLM and human annotations, we demonstrate two core applications: (1) surfacing policy ambiguity by revealing how annotators interpret safety instructions differently, and (2) surfacing value pluralism by uncovering systematic differences in safety priorities across demographic groups. Together, these capabilities support more targeted, transparent, and inclusive safety policy design.
Figure 1: Annotator Policy Models (APMs) learn interpretable representations of individual annotator safety policies. APMs are trained on annotation behavior to reveal how different annotators operationalize safety, enabling diagnosis of disagreement sources. By mapping annotators to a shared feature space, APMs make systematic comparison possible: identifying where annotators may have misunderstood the task itself (identifying operational failures), where they interpret instructions differently (surfacing policy ambiguity), or where they systematically differ by demographic group (surfacing value pluralism).
SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generation
March 24, 2026research area Computer Vision, research area Speech and Natural Language ProcessingWorkshop at ICLR
This paper was accepted at the Principled Design for Trustworthy AI — Interpretability, Robustness, and Safety across Modalities Workshop at ICLR 2026.
What exactly makes a particular image unsafe? Systematically differentiating between benign and problematic images is a challenging problem, as subtle changes to an image, such as an insulting gesture or symbol, can drastically alter its safety implications. However, existing image safety…
Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
November 3, 2025research area Data Science and Annotation, research area Human-Computer Interactionconference UIST
AI policy sets boundaries on acceptable behavior for AI models, but this is challenging in the context of large language models (LLMs): how do you ensure coverage over a vast behavior space? We introduce policy maps, an approach to AI policy design inspired by the practice of physical mapmaking. Instead of aiming for full coverage, policy maps aid effective navigation through intentional design choices about which aspects to capture and which to…