{"slug": "beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the", "title": "Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?", "summary": "A new arXiv paper (2610.31918v1) introduces a hierarchical agent distillation framework that separates two levels of on-policiness in distilling interactive LLM agents: the policy that induces an interaction history and the policy that generates the token prefix of the current action. Across a three-stage controlled study spanning three models and four interactive environments, the authors find that student-induced histories paired with teacher-generated action prefixes generally give the strongest standalone training signal, while following the student at both levels performs worst; fully on-policy data still helps when mixed with other configurations. The authors also report that mixtures with identical teacher/student proportions at each level can yield substantially different performance, and that front-loading fully teacher-generated contexts while delaying fully on-policy contexts beats a static mixture.", "body_md": "arXiv:2610.31918v1 Announce Type: new \nAbstract: Distilling interactive LLM agents requires choosing between teacher-generated demonstrations and student-generated experience, corresponding broadly to off-policy and on-policy distillation. While both have proven effective, we argue that this distinction is inherently hierarchical: the policy that induces an interaction history need not be the same as the policy that generates the token prefix of the current action. We introduce a hierarchical agent distillation framework that formalizes these two levels of on-policiness and enables explicit control over their pairing, mixture proportions, and training schedule. We study this problem through self-distillation, where a frozen teacher has access to task guidance unavailable to the student, and conduct a three-stage controlled study across three models and four interactive environments. We find that student-induced histories paired with teacher-generated action prefixes generally provide the strongest standalone training signal, whereas following the student at both levels generally performs worst. Nevertheless, fully on-policy data becomes beneficial when incorporated into a mixture with other configurations. Moreover, mixtures with identical teacher/student proportions at each level can yield substantially different performance, showing that how the two levels are paired matters beyond the overall amount of on-policy data. Finally, front-loading fully teacher-generated contexts and delaying fully on-policy contexts improves over a static mixture. These findings provide practical principles for deciding where, how much, and when agent distillation should follow the student.", "url": "https://wpnews.pro/news/beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the", "canonical_source": "https://arxiv.org/abs/2610.31918", "published_at": "2026-10-26 04:00:00+00:00", "updated_at": "2026-10-26 04:14:30.309481+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research", "machine-learning"], "entities": ["arXiv", "LLM agents"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the", "markdown": "https://wpnews.pro/news/beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the.md", "text": "https://wpnews.pro/news/beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the.txt", "jsonld": "https://wpnews.pro/news/beyond-on-policy-vs-off-policy-at-what-level-should-llm-agent-distillation-the.jsonld"}}