Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

A new benchmark called ImagingBench reveals that leading vision-language models and agentic AI systems, including Gemini, GPT, and Qwen, perform poorly on computational imaging tasks involving physics and inverse problems, despite strong performance on semantic visual tasks. The benchmark of 20 tasks shows agentic models consistently underperform specialized methods, especially in computational sensing areas like lensless imaging and holography, highlighting a gap between semantic understanding and physically grounded imaging.

arXiv:2607.07189v1 Announce Type: new Abstract: Vision-language models VLMs and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the physics and inverse problems that underlie computational imaging. We present ImagingBench, a benchmark of 20 computational imaging tasks spanning five categories: ray and wave optics, image signal processing, inverse reconstruction, computational sensing, and calibration. ImagingBench evaluates three complementary settings: Expert, fixed expert-guided inverse reconstruction; Planner, planner-guided inverse reconstruction; and Forward, forward-system simulation for consistency checking. We benchmark leading proprietary and open-source image-centric multimodal systems, including Gemini, GPT, and Qwen, and compare them with representative task-specific non-agentic baselines. Across tasks, agentic models remain consistently weaker than specialized methods, especially on computational sensing problems such as lensless imaging, event-based reconstruction, time-of-flight imaging, and holography. Planner guidance provides only modest and inconsistent gains over the fixed-prompt Expert baseline. Although the models often generate visually plausible outputs, their reference-based fidelity remains poor, revealing a substantial gap between semantic visual competence and physically grounded imaging performance. ImagingBench provides a unified testbed for measuring this gap and tracking progress in agentic AI for computational imaging.