Negative Self-Distillation: Learning to Reason by Avoiding Flaws

A new arXiv paper (2609.11699v1) introduces Negative Self-Distillation (NSD), a framework that improves large language model reasoning by diverging from self-generated flawed reasoning rather than imitating privileged ground-truth solutions. The authors report that NSD consistently outperforms On-Policy Self-Distillation (OPSD) and other label-free, self-bootstrapping reinforcement learning baselines, addressing OPSD's tendency to suppress uncertainty and penalize exploratory, self-corrective behavior on complex reasoning tasks. NSD uses a dynamic gating mechanism to isolate reasoning-critical tokens so gradient updates target behavioral flaws without degrading the model's foundational language capabilities.

arXiv:2609.11699v1 Announce Type: cross Abstract: On-Policy Self-Distillation OPSD has emerged as a popular paradigm for large language model LLM self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation NSD , a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition eg, acting as a careless reasoner'' and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning RL baselines.