One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

A critical review of on-policy self-distillation for language models finds that the method, which trains a model on its own generations with token-level teacher scores, requires a second, larger teacher model, limiting its practicality. The review highlights the trade-offs between dense supervision and on-policy sampling inherent in the approach.

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Di