PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

Researchers posted arXiv paper 2611.07408v1 introducing PSGAgent, an agentic framework that lets multimodal large language models dynamically refine visual perception for panoptic scene graph generation through a think-act-observe loop. PSGAgent equips MLLMs with two tools — a Region Zoom-In Tool for local exploration of small or boundary relations and a Visual Highlight Tool for relation-focused segmentation that filters irrelevant contextual noise — and is trained via a two-stage pipeline combining an error-aware Multimodal Chain-of-Thought dataset for cold-start learning with a hierarchical reward mechanism integrated with Group Sequence Policy Optimization (GSPO). The authors report that experiments on the PSG benchmark show PSGAgent achieves state-of-the-art performance in holistic and fine-grained scene understanding.

arXiv:2611.07408v1 Announce Type: new Abstract: Panoptic Scene Graph generation PSG requires holistic yet fine-grained reasoning over complex relations among objects. However, current Multimodal Large Language Models MLLMs applied to PSG suffer from a perception bottleneck: they function as static observers limited by fixed-resolution views. As a result, models often overlook critical relations involving small or peripheral objects and struggle to differentiate ambiguous predicates in complex scenes. To address these limitations, we propose PSGAgent, an agentic framework that enhances visual perception for PSG through a think-act-observe loop. PSGAgent allows MLLMs to dynamically refine their visual perception using two tools: a Region Zoom-In Tool for local exploration of small or boundary relations and a Visual Highlight Tool for relation-focused segmentation that filters out irrelevant contextual noise. To optimize decision-making, we construct a two-stage training pipeline. First, we create a high-quality, error-aware Multimodal Chain-of-Thought M-CoT dataset that provides robust supervision for cold-start learning. Then, we design a novel hierarchical reward mechanism integrated with Group Sequence Policy Optimization GSPO to enable strategic and adaptive tool usage. Extensive experiments on the PSG benchmark show that PSGAgent achieves SOTA performance, demonstrating strong capability in both holistic and fine-grained scene understanding.