Beyond Right and Wrong: Evaluating Second-order Social Reasoning in Large Language Models

A new arXiv paper (2609.05437v1) introduces NormReact, a dataset of 450 hand-annotated norm violation scenarios, to evaluate second-order social reasoning — metanorms — in large language models. Across six models, the researchers found LLMs overpredict negative sanctions where humans expect inaction, and alignment with human judgments deteriorates as social distance increases. The authors warn that AI systems used in norm-sensitive domains such as conflict mediation and policy simulation may produce a distorted picture of social regulation that over-represents punishment and under-represents tolerance and restraint.

arXiv:2609.05437v1 Announce Type: new Abstract: Previous AI alignment efforts have focused primarily on first-order social norms -- teaching models what is socially acceptable or unacceptable e.g., do not steal' . However, social intelligence depends not only on norm recognition, but also on anticipating who will enforce it and how e.g., public shame or even imprisonment . These second-order expectations, known as metanorms, govern how people respond when social rules are broken. We introduce a novel framework for evaluating metanorm reasoning in Large Language Models LLMs along two dimensions: emotional appraisal and behavioral response, and propose new classification tasks, namely, predicting self-regulation in violators, and other-regulation in observers. We release a multi-perspective dataset, NormReact, of 450 norm violation scenarios, hand-annotated for emotions and behavioral responses across norm violators' gender and observers' social closeness. Current LLMs portray a harsher social world: across six models, they overpredict negative sanctions where humans would expect inaction, and alignment with human judgments deteriorates as social distance increases. These findings suggest that AI systems in norm-sensitive domains from conflict mediation to policy simulation, may risk producing a distorted picture of social regulation: one that over-represents punishment and under-represents the tolerance, restraint, and relational calibration that characterize actual norm enforcement in real world.