{"slug": "understanding-and-mitigating-the-video-action-generalization-gap-via-temporal", "title": "Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio", "summary": "Researchers at MIT introduced the Temporal Ratio (TR), an attention-based metric that predicts how well video-action models generalize to new tasks after finetuning on robotic data. They found that TR measures a model's reliance on future-predictive latents and fluctuates naturally between planning and manipulation phases. Based on this, they developed an inference-time adaptive guidance method that dynamically amplifies compositional video conditioning signals, reducing the compositional generalization gap on the LIBERO benchmark and real-world tasks.", "body_md": "arXiv:2607.08127v1 Announce Type: new\nAbstract: Generative video foundation models exhibit strong compositional priors, yet world-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data. We refer to this discrepancy as the video-action generalization gap. In this paper, we systematically investigate this gap by evaluating a comprehensive design space of VAMs, demonstrating that standard design choices yield no emergent explanation pattern. To explain this behavior, we introduce the Temporal Ratio (TR), an attention-based measure of how strongly the action head relies on future latent rollouts relative to the anchored current frame. TR has two key properties: first, a model's structural reliance on future-predictive latents, measured via TR, acts as a predictor of its compositional generalization capacity; second, it natively fluctuates based on task phase, shifting attention to future frames during planning and reverting to the present frame for precise manipulation. Finally, based on these findings, we propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern to dynamically amplify compositional video conditioning signals precisely when the policy relies on future rollouts. Evaluated on the LIBERO benchmark and real-world tasks, our approach mitigates the OOD-ID compositional generalization gap. More details: https://umishra.me/temporal-ratio/", "url": "https://wpnews.pro/news/understanding-and-mitigating-the-video-action-generalization-gap-via-temporal", "canonical_source": "https://arxiv.org/abs/2607.08127", "published_at": "2026-07-10 04:00:00+00:00", "updated_at": "2026-07-10 04:22:12.819020+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "computer-vision", "robotics", "ai-research"], "entities": ["MIT", "LIBERO"], "alternates": {"html": "https://wpnews.pro/news/understanding-and-mitigating-the-video-action-generalization-gap-via-temporal", "markdown": "https://wpnews.pro/news/understanding-and-mitigating-the-video-action-generalization-gap-via-temporal.md", "text": "https://wpnews.pro/news/understanding-and-mitigating-the-video-action-generalization-gap-via-temporal.txt", "jsonld": "https://wpnews.pro/news/understanding-and-mitigating-the-video-action-generalization-gap-via-temporal.jsonld"}}