Gen4U: Unifying Video Generation and Understanding via Diffusion

Researchers demonstrate that state-of-the-art video diffusion models possess a highly structured latent space capable of both generation and understanding. They introduce Gen4U, a framework that repurposes frozen generative representations for tasks like video classification, depth estimation, and captioning without fine-tuning. The work unifies video generation and understanding, achieving strong perception performance while preserving generative capabilities.

arXiv:2607.06856v1 Announce Type: new Abstract: Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U Generation for Understanding , a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives video classification, depth estimation, camera pose estimation, image and video captioning . Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.