{"slug": "learning-heterogeneous-preferences", "title": "Learning Heterogeneous Preferences", "summary": "A new arXiv paper (2609.17847v1) introduces \"individuated utility\" functions that condition reward models on both the individual and their decision context, arguing that annotator disagreement in subjective tasks reflects systematic preference heterogeneity rather than stochastic noise. Evaluated on a newly collected dataset of more than 575,000 pairwise aesthetic judgments from 2,398 participants comparing automotive wheel designs, the individuated utility models substantially outperformed universal utility models, including foundation model baselines. The authors conclude that collecting annotator attributes and learning individuated utility functions enables reward models that explicitly account for whose preferences they represent.", "body_md": "arXiv:2609.17847v1 Announce Type: new \nAbstract: Learning from human feedback has become a central paradigm for training modern AI systems, where models of human utility are used as reward models in policy learning. Existing methods typically assume a \\emph{universal utility} function shared across a population and treat disagreement between annotators as stochastic variation. While suitable for objective tasks, this assumption breaks down in subjective domains where preferences vary systematically across individuals. We study the problem of subjective preference learning, in which observed choices arise from heterogeneous but internally consistent utility functions. Drawing upon rational choice theory, RCT \\parencite{tversky1981framing}, we introduce \\emph{individuated utility} functions conditioned on both the individual and their decision context, and propose a novel multi-stage architecture for estimating them from multi-modal data. We evaluate our framework on a newly collected dataset of more than $575{,}000$ pairwise aesthetic judgments from $2{,}398$ participants comparing automotive wheel designs. Our experiments show that individuated utility models substantially outperform universal utility models including foundation model baselines. Our results demonstrate that disagreement reflects meaningful preference heterogeneity rather than annotation noise. More broadly, our findings highlight the importance of collecting annotator attributes and learning individuated utility functions, enabling reward models that explicitly account for whose preferences they represent and faithfully capture human decision diversity.", "url": "https://wpnews.pro/news/learning-heterogeneous-preferences", "canonical_source": "https://arxiv.org/abs/2609.17847", "published_at": "2026-09-17 04:00:00+00:00", "updated_at": "2026-09-17 04:24:42.830685+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety"], "entities": ["arXiv", "rational choice theory", "Tversky"], "alternates": {"html": "https://wpnews.pro/news/learning-heterogeneous-preferences", "markdown": "https://wpnews.pro/news/learning-heterogeneous-preferences.md", "text": "https://wpnews.pro/news/learning-heterogeneous-preferences.txt", "jsonld": "https://wpnews.pro/news/learning-heterogeneous-preferences.jsonld"}}