Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

A new arXiv paper (2610.31918v1) introduces a hierarchical agent distillation framework that separates two levels of on-policiness in distilling interactive LLM agents: the policy that induces an interaction history and the policy that generates the token prefix of the current action. Across a three-stage controlled study spanning three models and four interactive environments, the authors find that student-induced histories paired with teacher-generated action prefixes generally give the strongest standalone training signal, while following the student at both levels performs worst; fully on-policy data still helps when mixed with other configurations. The authors also report that mixtures with identical teacher/student proportions at each level can yield substantially different performance, and that front-loading fully teacher-generated contexts while delaying fully on-policy contexts beats a static mixture.

arXiv:2610.31918v1 Announce Type: new Abstract: Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.