SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

Researchers introduced SOMA, a benchmark of 1,632 human-verified questions for fine-grained egocentric body-motion understanding, spanning body configuration, motion dynamics, and temporal reasoning. In evaluations of 37 closed-source, open-source, and specialized egocentric MLLMs, humans reached 68.5% accuracy from video alone while the strongest model, Qwen3.6-27B, reached 50.0% zero-shot, with error analysis showing models misread the wearer's own body, misjudge event timing, and favor the right hand or under-count repetitions. The team also released a training dataset of aligned video, body motion, and language supervision plus a multimodal baseline, with the full benchmark, training data, and code to be released at cnai.epfl.ch/soma.

arXiv:2610.36384v1 Announce Type: new Abstract: Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models MLLMs largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone 68.5% accuracy , yet clearly outperform the strongest MLLM Qwen3.6-27B , which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. Full benchmark, training data, and code will be released at: https://cnai.epfl.ch/soma