How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

A new analysis examines how much training-inference mismatch on-policy distillation (OPD) can tolerate, given that OPD trains a student language model on its own responses under a fixed teacher and that quantized inference and asynchronous rollouts—used to cut the cost of generating those responses—both introduce mismatch between the rollout and training. The work frames the tradeoff between cheaper rollout generation and the divergence it creates between the inference and training distributions.

On-policy distillation OPD trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rol