Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

A study of on-policy self-distillation (OPSD) across 16 privileged-information (PI) configurations, three multi-turn agentic benchmarks and three student models found that PI effectiveness depends on its content, generation source and the student model, with information-dense trajectories and PI from more capable generators generally yielding larger gains while thinking models benefit less consistently. The authors report that two KL-based metrics — sampled-token reverse KL (D) and error-step KL contrast (Δd⁻err) — track post-training performance more consistently than PI utility, which can be distorted by information leakage, and that selecting each task's PI by Δd⁻err improved post-training performance over random selection, most when PI candidates differ widely. Effective PI also remained beneficial when combined with GRPO, with the combined objective outperforming a strong GRPO baseline.

arXiv:2610.22512v1 Announce Type: new Abstract: On-policy self-distillation OPSD enables large language models LLMs to learn from privileged information PI without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL $D$ and error-step KL contrast $\Delta d^{-} {\mathrm{err}}$ . The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by $\Delta d^{-} {\mathrm{err}}$ also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.