{"slug": "probe-manipulation-grounded-visual-question-answering-with-vlm-agents", "title": "PROBE: Manipulation-Grounded Visual Question Answering with VLM Agents", "summary": "Researchers introduced PROBE, a framework for benchmarking and finetuning vision-language model (VLM) agents on Manipulation-Grounded Visual Question Answering (MG-VQA), where robots must physically move objects to answer questions in cluttered scenes. In PROBE-Bench, a suite of 150 tasks across 6 question types, agentic tool-based methods outperformed perception-only baselines by 8.0% on average across all frontier VLMs. The finetuned PROBE-Agent models surpassed off-the-shelf agent baselines by 11.5% on average and showed positive transfer to unseen objects and a held-out task, with successful sim-to-real transfer in real-world tabletop environments.", "body_md": "arXiv:2608.17129v1 Announce Type: new\nAbstract: Vision-language Models (VLMs) excel at 2D grounding, spatial reasoning and agentic tool-based planning in static scenes. However, consider asking a home robot \"Is my medication still in the cabinet?\" The answer may be physically hidden behind a row of containers that must first be moved aside. Answering such questions in real-world cluttered environments requires reasoning in dynamic scenes: distractors must be manipulated to reveal occluded objects, and each action changes the scene the model must reason over. We formalize this setting as Manipulation-Grounded Visual Question Answering (MG-VQA) and introduce PROBE, a framework for benchmarking and finetuning VLM agents on such tasks. We first develop PROBE-Sim, a high-fidelity tabletop simulator with everyday objects and a robot manipulator equipped with grasping and pushing tools. PROBE-Sim is used to create PROBE-Bench: an evaluation suite of 150 tasks across 6 question types on cluttered tabletop scenes, where a VLM perceives, picks up or pushes objects before answering. We observe consistent trend across all frontier VLMs: agentic tool-based methods outperform their perception-only baselines (8.0% on average) across all task types. We further design PROBE-Agent, a finetuning recipe to distill successful trajectories from a powerful teacher foundation model to a smaller open-weight model using a mixed data recipe that encourages manipulation-efficient question answering. PROBE Agent finetuned models outperform their off-the-shelf agent baseline (11.5% on average) and demonstrate positive transfer to unseen objects and a held-out task. We validate sim-to-real transfer by deploying PROBE-Agent finetuned policies in real-world tabletop environments.", "url": "https://wpnews.pro/news/probe-manipulation-grounded-visual-question-answering-with-vlm-agents", "canonical_source": "https://arxiv.org/abs/2608.17129", "published_at": "2026-08-19 04:00:00+00:00", "updated_at": "2026-08-19 04:12:46.590087+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision", "robotics", "ai-agents"], "entities": ["PROBE", "PROBE-Sim", "PROBE-Bench", "PROBE-Agent", "Vision-language Models (VLMs)"], "alternates": {"html": "https://wpnews.pro/news/probe-manipulation-grounded-visual-question-answering-with-vlm-agents", "markdown": "https://wpnews.pro/news/probe-manipulation-grounded-visual-question-answering-with-vlm-agents.md", "text": "https://wpnews.pro/news/probe-manipulation-grounded-visual-question-answering-with-vlm-agents.txt", "jsonld": "https://wpnews.pro/news/probe-manipulation-grounded-visual-question-answering-with-vlm-agents.jsonld"}}