The Design Space of Tri-Modal Masked Diffusion Models

Researchers from the University of Cambridge, MIT, Google DeepMind and Apple introduced the first tri-modal Masked Diffusion Models pretrained from scratch on text, image-text and audio-text data, pretraining a preliminary 3B-parameter model on 6.4T tokens that achieves strong results in text generation, text-to-image (T2I) and text-to-speech (T2S) tasks. The work systematically analyzes multimodal scaling laws, modality mixing ratios, noise schedules and batch-size effects, and derives a stochastic differential equation (SDE) based reparameterization that decouples physical batch size from logical batch size, eliminating the need to tune an optimal batch size. The authors describe it as the largest-scale systematic open study of multimodal discrete diffusion models conducted to date.

Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and finetuning a base unimodal model for bi-modal generation. Diverging from previous approaches, we introduce the first tri-modal Masked Diffusion Models MDM pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects and provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation SDE based reparameterization, eliminating the need for tuning the optimal batch size as reported in recent work. This reparameterization decouples the physical batch size, often chosen based on compute GPU saturation, FLOP-efficiency, wall-clock time from the logical batch size, chosen to balance the variance of gradients during stochastic optimization. Finally, we pretrain a preliminary model showcasing the capabilities of a unified design, achieving strong results at 3B model scale 6.4T tokens , in both text generation, T2I tasks, and T2S tasks. Our work represents the largest scale systematic open study of multimodal discrete diffusion models conducted to date, providing valuable insights into scaling behaviors across multiple modalities. - † University of Cambridge - ‡ MIT - § Google DeepMind - Work done while at Apple