OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

Researchers have introduced OmniToM, a benchmark that evaluates large language models' theory of mind by requiring explicit modeling of belief structures for all actors in a narrative. The benchmark, built from 895 stories and 22,343 labeled belief propositions, tests models in two stages: belief extraction and belief labeling across seven dimensions. Zero-shot evaluations reveal that current LLMs struggle with transforming narrative facts into actors' beliefs and shared mental states, exposing a bottleneck in actor-specific belief tracking.

arXiv:2605.26322v1 Announce Type: new Abstract: Theory of Mind ToM , the ability to infer others' knowledge, intentions, and emotions, is commonly evaluated in large language models LLMs using end-point question answering, where performance is judged solely by the final answer to a social reasoning query. This paradigm obscures whether the model actually constructs the underlying mental-state representations required for robust reasoning, particularly in scenarios involving divergent, evolving, or mistaken beliefs. In order to address this research gap, we introduce OmniToM, a benchmark that directly evaluates these representations by requiring explicit modeling of belief structures for all relevant actors within a narrative. These structures are composed of belief propositions: minimal statements of what an actor takes to be true about the world or another actor's mental state, allowing knowledge, intentions, emotions, and false beliefs to be analyzed in a common format. Models are evaluated in two stages: Stage 1: Belief Extraction, which extracts from the story the beliefs relevant to its social dynamics, and Stage 2: Belief Labeling, which assigns each belief a seven-dimensional schema label covering recursive order, truth status, knowledge access, explicitness, content type, mental source, and context. Built from 895 stories from the existing ToMBench story corpus and augmented with 22,343 labeled belief propositions, OmniToM uses a human-calibrated LLM-assisted annotation pipeline. Across diverse models in zero-shot evaluation, OmniToM reveals an actor-specific belief-tracking bottleneck: current LLMs struggle with the knowledge-access and representational decisions required to transform narrative facts into actors' beliefs and shared mental states.