MindEdit-Bench

Researchers introduced MindEdit-Bench, a benchmark of six spatial reasoning tasks for vision-language models, built from smartphone triplets of 120 private indoor scenes. Across 15 VLMs on 1,003 human-verified questions, models achieved only 8%-31% accuracy versus 81%-97% human accuracy, revealing a 53 percentage point gap and systematic failures in counterfactual reasoning about object movement and visibility.

Benchmarks for vision-language models VLMs mostly test observational spatial reasoning: models describe relations already visible in the input. Existing what-if tasks typically vary the observer while keeping the scene fixed. Can VLMs instead predict the consequences of hypothetically moving or rotating an object? We introduce MindEdit-Bench, a benchmark of six spatial reasoning tasks built from three-photo smartphone triplets of newly captured indoor scenes via an automatic in-the-wild 3D scene-graph extraction pipeline. Four tasks probe perception and perspective transformation over observed structure; two new tasks, L4 spatial editing and L5 cross-view visibility editing , probe object-level counterfactual reasoning, where correct answers are absent from all input images. Each question provides 8-24 structured answer choices, enabling answer-letter-level diagnosis of spatial and fallback errors. The benchmark covers 120 private indoor scenes not drawn from public datasets, reducing public-data pretraining-overlap risk. Across 15 VLMs on 1,003 human-verified questions, task-wise mean VLM accuracy is only 8%-31%, versus 81%-97% human majority-vote accuracy. The pooled human--best-VLM gap is 53 pp, with at least 39 pp on every task. The structured answer space further reveals non-uniform failures, including weaker camera-depth-axis inference and fallback behavior on difficult visibility-editing cases. Category: Spatial Reasoning. Imported rows: 15. Top imported result: Gemini-3.1-Pro-Preview, rank 1, 34.60.