AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Researchers introduced AVTrack, a new human-centric audio-visual instance segmentation dataset designed for dynamic real-world scenarios. The dataset features challenging conditions such as camera motion, visual occlusions, and position changes, revealing substantial performance degradation in existing methods. AVTrack establishes a rigorous benchmark for robust audio-visual speaker tracking in complex environments, with applications in intelligent video editing, surveillance, and human-computer interaction.

arXiv:2606.02724v1 Announce Type: new Abstract: Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human-computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio-visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack, a human-centric audio-visual instance segmentation AVIS dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/