Improved Large Language Diffusion Models

Researchers introduced iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention, scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus. Compared to LLaDA, iLLaDA improved by 21.6 points on BBH and 14.9 points on ARC-Challenge, and remained competitive with Qwen2.5 7B on several benchmarks, demonstrating that fully bidirectional diffusion training is a viable path for strong language models.

arXiv:2606.25331v1 Announce Type: new Abstract: Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning SFT , scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.