content type paperpublished July 2026 Incentivizing Temporal-Awareness in Egocentric Video Understanding Models
AuthorsZhiyang Xu†, Tian Qin‡, Bowen Jin§, Zhengfeng Lai¶, Meng Cao, Lifu Huang¶, Peng Zhang
Incentivizing Temporal-Awareness in Egocentric Video Understanding Models
AuthorsZhiyang Xu†, Tian Qin‡, Bowen Jin§, Zhengfeng Lai¶, Meng Cao, Lifu Huang¶, Peng Zhang
Multimodal large language models (MLLMs) have recently shown strong performance in visual understanding, yet they often lack temporal awareness, particularly in egocentric settings where reasoning depends on the correct ordering and evolution of events. This deficiency stems in part from training objectives that fail to explicitly reward temporal reasoning and instead rely on frame-level spatial shortcuts. To address this limitation, we propose Temporal Global Policy Optimization (TGPO), a reinforcement learning with verifiable rewards (RLVR) algorithm designed to incentivize temporal awareness in MLLMs. TGPO contrasts model outputs generated from temporally ordered versus shuffled video frames to derive calibrated, globally normalized reward signals that explicitly favor temporally coherent reasoning. Integrated with GRPO and GSPO, TGPO supports cold-start RL training and effectively suppresses spatial shortcut behaviors learned by existing MLLMs. Experiments across five egocentric video benchmarks demonstrate that TGPO consistently improves temporal grounding and causal coherence, outperforming prior RL-based video reasoning approaches. Our results suggest that TGPO offers a simple and scalable pathway toward temporally robust MLLMs for egocentric video understanding.
NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding
January 6, 2026research area Computer Vision, research area Data Science and Annotationconference ECCV
Multimodal large language models (MLLMs) have achieved impressive progress in vision-language reasoning, yet their ability to understand temporally unfolding narratives in videos remains underexplored. True narrative understanding requires grounding who is doing what, when, and where, maintaining coherent entity representations across dynamic visual and temporal contexts. We introduce NarrativeTrack, the first benchmark to evaluate narrative…
MM-Ego: Towards Building Egocentric Multimodal LLMs
April 11, 2025research area Computer Vision, research area Speech and Natural Language Processingconference ICLR
This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data. This is one of the largest egocentric QA datasets…