{"slug": "comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video", "title": "CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration", "summary": "Researchers have developed CoMoGen, a controllable video generation framework that creates realistic motion dynamics and interactions from a single binary mask sequence and an input image. The system uses a lightweight MaskAdapter to encode mask sequences into latent signals injected into a Multi Modal Diffusion Transformer model, with selective fine-tuning of \"Motion Layers\" via Low-Rank Adaptation to focus on motion-critical components. CoMoGen achieves state-of-the-art performance in motion fidelity and perceptual realism, enabling precise subject motion and plausible interactions with surrounding humans, objects, and scenes.", "body_md": "arXiv:2605.22996v1 Announce Type: new\nAbstract: We present CoMoGen, a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image. CoMoGen introduces a lightweight MaskAdapter that encodes binary mask sequences into a latent residual signal, injected into the Multi Modal Diffusion Transformer (MMDiT) model through a cosine-weighted schedule. Unlike the hierarchical coarse-to-fine design of UNet architectures, MMDiT operates as a sequence of uniform transformer blocks, making it difficult to identify which layers are responsible for the motion generation. Therefore, we propose a novel way to determine \"Motion Layers\" operating in the attention space of MMDiT. We fine-tune the model by using Low-Rank Adaptation (LoRA) to the Motion Layers, without requiring any architecture change in the MMDiT. This selective adaptation enables our method to focus on motion-critical components, yielding reduced computational cost. Despite its simplicity, CoMoGen enables precise subject motion and plausible interactions with surrounding humans, objects, and scenes. Comprehensive experiments on different datasets show that CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism. Project page: mericadil.github.io/CoMoGen.", "url": "https://wpnews.pro/news/comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video", "canonical_source": "https://arxiv.org/abs/2605.22996", "published_at": "2026-05-25 04:00:00+00:00", "updated_at": "2026-05-25 15:20:21.873710+00:00", "lang": "en", "topics": ["generative-ai", "computer-vision", "machine-learning", "artificial-intelligence", "neural-networks"], "entities": ["CoMoGen", "MaskAdapter", "Multi Modal Diffusion Transformer", "MMDiT", "Low-Rank Adaptation", "LoRA", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video", "markdown": "https://wpnews.pro/news/comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video.md", "text": "https://wpnews.pro/news/comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video.txt", "jsonld": "https://wpnews.pro/news/comogen-controllable-motion-dynamics-and-interactions-with-mask-guided-video.jsonld"}}