{"slug": "turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon", "title": "TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training", "summary": "Researchers propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon language agents. The method addresses inefficiencies in vanilla agent OPD by using adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting. Experiments on ALFWorld, WebShop, and Multi-Hop Search show TurnOPD achieves superior validation accuracy under equal wall-clock training budgets.", "body_md": "arXiv:2607.05804v1 Announce Type: new\nAbstract: On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.", "url": "https://wpnews.pro/news/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon", "canonical_source": "https://arxiv.org/abs/2607.05804", "published_at": "2026-07-08 04:00:00+00:00", "updated_at": "2026-07-08 04:05:24.985008+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-agents"], "entities": ["TurnOPD", "ALFWorld", "WebShop", "Multi-Hop Search"], "alternates": {"html": "https://wpnews.pro/news/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon", "markdown": "https://wpnews.pro/news/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon.md", "text": "https://wpnews.pro/news/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon.txt", "jsonld": "https://wpnews.pro/news/turnopd-making-on-policy-distillation-turn-aware-for-efficient-long-horizon.jsonld"}}