†and On-Policy Distill
†
Distillation forces a student to imitate teacher logits, inducing memorization on the training samples while degrading generalization on unseen samples. (Overfitting on dataset and teacher)
†: prompt replay buffer · all experiments run on Qwen3.5
Forhard questions, how can we transfer the teacher's knowledge to the student without imitating the teacher's logits or injecting the teacher's response directly into the student's gradient?How to make the student solve the hard question withoutpolicy drift(degrading generalization)?
Technically, we use a Replay Buffer to store hard questions, so the model revisits each hard question many times — not just once, as in GRPO. Repeated exposure strengthens the BCQ/NCQ effect on each hard question, which we expect to lift its rollout accuracy.
A question is admitted to the Replay Buffer when its rollout accuracy stays below 50%, and it graduates — leaving the buffer — once that accuracy reaches 50%. ZPPO graduates far more hard questions than GRPO, and the gap is widest where the initial accuracy starts near zero.