ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

A new diagnostic framework called ChartProbe, introduced in an arXiv paper (arXiv:2608.13766v1), shows that vision-language models (VLMs) can improve complex visual reasoning by training on simpler skills—perception, grounding, and simple reasoning—without any complex-reasoning data. Across three open-weight VLMs, fine-tuning on these simpler skills alone produced large gains on held-out complex-reasoning questions, with improvements holding across unseen chart types, a human-written benchmark (ChartQA), and a non-chart visual domain (CLEVR).

arXiv:2608.13766v1 Announce Type: new Abstract: Vision-language models VLMs remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements \emph{perception} , locating them and binding them to their labels \emph{grounding} , and performing single-step computations such as ranking, totals, and differences \emph{simple reasoning} . We introduce \textbf{ChartProbe}, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type pie charts , a human-written benchmark disjoint from our images and templates ChartQA , and a non-chart visual domain CLEVR . Complex visual reasoning can therefore improve without complex-reasoning supervision.