The Social World Model Evaluation Gap

An audit of 113 works published from 1999 to 2026 found that only 9 primary evaluations measure a real person's response to an action selected by a system or experimenter, and only 2 of those also compare an explicit response forecast with that response, according to the arXiv paper 2610.33014v1. A separate full-text check of 25 language-model works found no experiment meeting those conditions, with the remaining evaluations testing current-state inference, task success, or prediction from recorded actions. The authors propose a 1-step negotiation experiment to separate these questions and note the findings describe the audited sample, not the prevalence of the gap across the field.

arXiv:2610.33014v1 Announce Type: new Abstract: A negotiation assistant can reach a good deal without accurately predicting the other person's response. We ask whether evaluations of social world models test that prediction directly. We audit 113 works published from 1999 to 2026. 9 primary evaluations measure a real person's response to an action selected by the system or experimenter. Only 2 also compare an explicit response forecast with that response. Their predictors use information available when the action is chosen, and the evaluated responses were excluded from fitting; both randomize the intervention. A separate full-text check of 25 language-model works finds no experiment meeting these conditions. The other evaluations test current-state inference, task success, or prediction from recorded actions. We propose a 1-step negotiation experiment that separates these questions. The findings describe the audited sample, not the prevalence of this evaluation gap across the field.