{"slug": "the-social-world-model-evaluation-gap", "title": "The Social World Model Evaluation Gap", "summary": "An audit of 113 works published from 1999 to 2026 found that only 9 primary evaluations measure a real person's response to an action selected by a system or experimenter, and only 2 of those also compare an explicit response forecast with that response, according to the arXiv paper 2610.33014v1. A separate full-text check of 25 language-model works found no experiment meeting those conditions, with the remaining evaluations testing current-state inference, task success, or prediction from recorded actions. The authors propose a 1-step negotiation experiment to separate these questions and note the findings describe the audited sample, not the prevalence of the gap across the field.", "body_md": "arXiv:2610.33014v1 Announce Type: new \nAbstract: A negotiation assistant can reach a good deal without accurately predicting the other person's response. We ask whether evaluations of social world models test that prediction directly. We audit 113 works published from 1999 to 2026. 9 primary evaluations measure a real person's response to an action selected by the system or experimenter. Only 2 also compare an explicit response forecast with that response. Their predictors use information available when the action is chosen, and the evaluated responses were excluded from fitting; both randomize the intervention. A separate full-text check of 25 language-model works finds no experiment meeting these conditions. The other evaluations test current-state inference, task success, or prediction from recorded actions. We propose a 1-step negotiation experiment that separates these questions. The findings describe the audited sample, not the prevalence of this evaluation gap across the field.", "url": "https://wpnews.pro/news/the-social-world-model-evaluation-gap", "canonical_source": "https://arxiv.org/abs/2610.33014", "published_at": "2026-10-27 04:00:00+00:00", "updated_at": "2026-10-27 04:12:08.612009+00:00", "lang": "en", "topics": ["ai-research", "ai-safety", "large-language-models", "artificial-intelligence"], "entities": ["arXiv", "2610.33014v1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-social-world-model-evaluation-gap", "markdown": "https://wpnews.pro/news/the-social-world-model-evaluation-gap.md", "text": "https://wpnews.pro/news/the-social-world-model-evaluation-gap.txt", "jsonld": "https://wpnews.pro/news/the-social-world-model-evaluation-gap.jsonld"}}