{"slug": "multi-mask-diffusion-language-models-for-few-step-generation", "title": "Multi-Mask Diffusion Language Models for Few-Step Generation", "summary": "Researchers propose Multi-Mask Diffusion Models (MultiMDM) for few-step language generation, preserving the masking structure to enable drafting and refinement. The model achieves effective few-step generation through a closed-form ELBO training objective and discrete-state consistency distillation, as demonstrated in pretraining and distillation experiments.", "body_md": "arXiv:2607.19686v1 Announce Type: new\nAbstract: Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.", "url": "https://wpnews.pro/news/multi-mask-diffusion-language-models-for-few-step-generation", "canonical_source": "https://www.machinebrief.com/news/multi-mask-diffusion-language-models-for-few-step-generation-i9s7", "published_at": "2026-07-23 04:00:00+00:00", "updated_at": "2026-07-23 04:04:25.757867+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "generative-ai", "large-language-models"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/multi-mask-diffusion-language-models-for-few-step-generation", "markdown": "https://wpnews.pro/news/multi-mask-diffusion-language-models-for-few-step-generation.md", "text": "https://wpnews.pro/news/multi-mask-diffusion-language-models-for-few-step-generation.txt", "jsonld": "https://wpnews.pro/news/multi-mask-diffusion-language-models-for-few-step-generation.jsonld"}}