{"slug": "beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress", "title": "Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress", "summary": "Researchers propose R2-OPD, a method that filters on-policy distillation rewards by reasoning progress to improve language model post-training. The approach, detailed in arXiv:2608.19408v1, suppresses teacher-derived rewards that conflict with genuine reasoning advancement, showing consistent gains over standard on-policy distillation, particularly in reasoning tasks.", "body_md": "arXiv:2608.19408v1 Announce Type: new\nAbstract: On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.", "url": "https://wpnews.pro/news/beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress", "canonical_source": "https://arxiv.org/abs/2608.19408", "published_at": "2026-08-21 04:00:00+00:00", "updated_at": "2026-08-21 04:12:32.018626+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "artificial-intelligence"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress", "markdown": "https://wpnews.pro/news/beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress.md", "text": "https://wpnews.pro/news/beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress.txt", "jsonld": "https://wpnews.pro/news/beyond-imitation-filtering-on-policy-distillation-by-reasoning-progress.jsonld"}}