{"slug": "trace-based-on-policy-distillation-for-masked-diffusion-language-models", "title": "Trace-Based On-Policy Distillation for Masked Diffusion Language Models", "summary": "Researchers propose trace-based on-policy distillation (TOPD), a teacher-supervised framework that transfers reasoning ability to a diffusion large language model (dLLM) without reward estimation. TOPD enables SDAR-4B-Chat to match the MATH500 accuracy of its RL-trained counterpart TraDo-4B-Instruct, with gains of +5.7 under static evaluation and +4.5 under dynamic evaluation, while achieving a 96.0× model-compute speedup with 4× fewer rollout rounds.", "body_md": "arXiv:2607.16872v1 Announce Type: new\nAbstract: Diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. However, reasoning-oriented post-training for dLLMs remains challenging. Supervised fine-tuning (SFT) for dLLMs requires dense but often off-policy masked states, while reinforcement learning (RL) relies on sparse rewards or value modeling. This paper proposes \\textbf{trace-based on-policy distillation (TOPD)}, a teacher-supervised framework that transfers reasoning ability to a target dLLM without reward estimation. The key idea is to supervise a dLLM on its own denoising trajectory, focusing on the trace-aligned token decisions that form the final response. Specifically, TOPD samples on-policy diffusion trajectories from the target dLLM, obtains teacher token distributions from a teacher model on the corresponding partially denoised states, and updates the target dLLM with a token-level Reverse Kullback-Leibler (Reverse-KL) objective. This design preserves dense teacher supervision while aligning training with the model's own denoising states. On mathematical reasoning benchmarks, TOPD enables SDAR-4B-Chat to match the MATH500 accuracy of its RL-trained counterpart TraDo-4B-Instruct, with gains of +5.7 under static evaluation and +4.5 under dynamic evaluation. Compared with the RL-trained counterpart, TOPD achieves this with 4$\\times$ fewer rollout rounds, corresponding to an estimated 96.0$\\times$ to-accuracy model-compute speedup.", "url": "https://wpnews.pro/news/trace-based-on-policy-distillation-for-masked-diffusion-language-models", "canonical_source": "https://arxiv.org/abs/2607.16872", "published_at": "2026-07-21 04:00:00+00:00", "updated_at": "2026-07-21 04:23:23.505679+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "machine-learning", "ai-research"], "entities": ["TOPD", "SDAR-4B-Chat", "TraDo-4B-Instruct", "MATH500"], "alternates": {"html": "https://wpnews.pro/news/trace-based-on-policy-distillation-for-masked-diffusion-language-models", "markdown": "https://wpnews.pro/news/trace-based-on-policy-distillation-for-masked-diffusion-language-models.md", "text": "https://wpnews.pro/news/trace-based-on-policy-distillation-for-masked-diffusion-language-models.txt", "jsonld": "https://wpnews.pro/news/trace-based-on-policy-distillation-for-masked-diffusion-language-models.jsonld"}}