{"slug": "taming-text-to-sounding-video-generation-via-advanced-modality-condition-and", "title": "Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction", "summary": "Researchers propose a Cross-Referential Rewriter (CRR) caption framework to improve Text-to-Sounding-Video (T2SV) generation by addressing text conditioning bottlenecks and cross-modal fusion challenges. The method uses a dual-agent pipeline with a Semantic Checker and Cross-Modal Rewriter to generate disentangled caption pairs, eliminating modal interference and bridging the gap between training and inference.", "body_md": "[content type paper](/research/)published July 2026\n\nTaming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction\n\nAuthorsKaisi Guan†‡**, Xihua Wang†‡, Zhengfeng Lai, Xin Cheng†, Peng Zhang, Xiaojiang Liu, Ruihua Song†, Meng Cao\n\nTaming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction\n\nAuthorsKaisi Guan†‡**, Xihua Wang†‡, Zhengfeng Lai, Xin Cheng†, Peng Zhang, Xiaojiang Liu, Ruihua Song†, Meng Cao\n\nThis study focuses on Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text, with both modalities aligned to the text conditions. Despite progress in joint audio-video training, two critical challenges remain: (1) text conditioning is a bottleneck—shared captions (TV=TA) trigger modal interference, while a gap persists between dense training captions and concise inference user prompts, and (2) the optimal fusion mechanism for cross-modal feature interaction remains unclear. To address the first challenge, we first propose the Cross-Referential Rewriter (CRR) caption framework, a dual-agent pipeline where a Semantic Checker extracts grounded Semantic Anchors and a Cross-Modal Rewriter generates disentangled caption pairs (TV and TA), eliminating modal interference and bridging the training-inference gap.\n\nRevisit Large-Scale Image–Caption Data in Pre-training Multimodal Foundation Models\n\nApril 8, 2025[research area Computer Vision](/research/?domain=Computer%20Vision), [research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms)[conference ICLR](/research/?event=ICLR)\n\nRecent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. Notably, the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still unclear. Additionally, different multimodal foundation models may have distinct preferences for specific caption formats while the efforts of studying the optimal captions for each foundation…\n\nPromoting Cross-Modal Representations to Improve Multimodal Foundation Models for Physiological Signals\n\nOctober 28, 2024[research area Methods and Algorithms](/research/?domain=Methods%20and%20Algorithms)[conference NeurIPS](/research/?event=NeurIPS)\n\nMany healthcare applications are inherently multimodal, involving several physiological signals. As sensors for these signals become more common, improving machine learning methods for multimodal healthcare data is crucial. Pretraining foundation models is a promising avenue for success. However, methods for developing foundation models in healthcare are still in early exploration and it is unclear which pretraining strategies are most effective…", "url": "https://wpnews.pro/news/taming-text-to-sounding-video-generation-via-advanced-modality-condition-and", "canonical_source": "https://machinelearning.apple.com/research/sounding-video-generation", "published_at": "2026-07-07 00:00:00+00:00", "updated_at": "2026-07-07 14:59:11.573500+00:00", "lang": "en", "topics": ["generative-ai", "computer-vision", "natural-language-processing", "ai-research"], "entities": ["Kaisi Guan", "Xihua Wang", "Zhengfeng Lai", "Xin Cheng", "Peng Zhang", "Xiaojiang Liu", "Ruihua Song", "Meng Cao"], "alternates": {"html": "https://wpnews.pro/news/taming-text-to-sounding-video-generation-via-advanced-modality-condition-and", "markdown": "https://wpnews.pro/news/taming-text-to-sounding-video-generation-via-advanced-modality-condition-and.md", "text": "https://wpnews.pro/news/taming-text-to-sounding-video-generation-via-advanced-modality-condition-and.txt", "jsonld": "https://wpnews.pro/news/taming-text-to-sounding-video-generation-via-advanced-modality-condition-and.jsonld"}}