{"slug": "improved-large-language-diffusion-models", "title": "Improved Large Language Diffusion Models", "summary": "Researchers introduced iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention, scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus. Compared to LLaDA, iLLaDA improved by 21.6 points on BBH and 14.9 points on ARC-Challenge, and remained competitive with Qwen2.5 7B on several benchmarks, demonstrating that fully bidirectional diffusion training is a viable path for strong language models.", "body_md": "arXiv:2606.25331v1 Announce Type: new\nAbstract: Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \\emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.", "url": "https://wpnews.pro/news/improved-large-language-diffusion-models", "canonical_source": "https://arxiv.org/abs/2606.25331", "published_at": "2026-06-25 04:00:00+00:00", "updated_at": "2026-06-25 04:15:41.074996+00:00", "lang": "en", "topics": ["large-language-models", "generative-ai", "ai-research", "natural-language-processing"], "entities": ["iLLaDA", "LLaDA", "Qwen2.5 7B", "arXiv", "ML-GSAI"], "alternates": {"html": "https://wpnews.pro/news/improved-large-language-diffusion-models", "markdown": "https://wpnews.pro/news/improved-large-language-diffusion-models.md", "text": "https://wpnews.pro/news/improved-large-language-diffusion-models.txt", "jsonld": "https://wpnews.pro/news/improved-large-language-diffusion-models.jsonld"}}