TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

Researchers propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon language agents. The method addresses inefficiencies in vanilla agent OPD by using adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting. Experiments on ALFWorld, WebShop, and Multi-Hop Search show TurnOPD achieves superior validation accuracy under equal wall-clock training budgets.

arXiv:2607.05804v1 Announce Type: new Abstract: On-policy distillation OPD trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: 1 full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and 2 trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.