Masked Language Flow Models

Researchers introduced Masked Language Flow Models (MLFMs), combining masked diffusion and flow-based methods for efficient language generation. MLFMs enable conditional generation via continuous flows and support multi-step reasoning through a novel alternating sampler. Evaluations on GSM8K and MT-Bench show flow-based language models can scale to reasoning and instruction-following tasks.

arXiv:2606.27617v1 Announce Type: new Abstract: Masked Diffusion Models MDMs promise fast, parallel language generation, but their reverse transition factorises across token positions -- an approximation that breaks down in the few-step sampling regime where parallel generation ought to provide the greatest efficiency gains. Flow Language Models FLMs sidestep this limitation by learning a continuous flow that transports noise toward clean sequences represented in Euclidean space, inducing a flow map that can be distilled for single-step generation. However, this makes complex tasks requiring multi-step reasoning problematic for FLMs, as FLMs are forced to decode every token during generation. To address this, we introduce Masked Language Flow Models MLFMs , which incorporate masking into FLMs using a continuous stochastic interpolant to bridge partially masked and clean sequences. This design enables conditional generation via continuous flows and allows pretrained MDMs to be converted into MLFMs through a simple, lightweight adaptation. Leveraging this flexibility, we propose a novel sampler that alternates continuous denoising with the discrete unmasking of confident tokens to better support multi-step reasoning. We evaluate our approach on GSM8K and MT-Bench and find, for the first time, that flow-based language models can be scaled to solve downstream reasoning and instruction-following tasks.