Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

Researchers at an undisclosed institution audited 20 video large language models and found that benchmark accuracy often does not reflect genuine visual understanding. They introduced the Visual Dependency Gap (VDG) metric, showing that models can achieve high accuracy without relying on video content, and that temporal order contributes almost nothing to performance. The findings challenge the validity of existing video benchmarks for measuring visually grounded reasoning.

arXiv:2607.13305v1 Announce Type: new Abstract: Benchmark accuracy in video large language models LLMs is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap VDG , the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video p = 0.0003 but not on black screens p = 0.53 . Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.