{"slug": "from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes", "title": "From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation", "summary": "Researchers propose Prefix-GRPO, a reinforcement learning framework that improves small language model agents by decomposing teacher trajectories into replay-aligned prefix queries and online continuations, enabling more efficient learning in long-horizon environments. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO outperforms distillation and standard RL baselines, with implementation available on GitHub.", "body_md": "arXiv:2607.19395v1 Announce Type: new\nAbstract: Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at https://github.com/HappynessI/Prefix_GRPO.", "url": "https://wpnews.pro/news/from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes", "canonical_source": "https://arxiv.org/abs/2607.19395", "published_at": "2026-07-23 04:00:00+00:00", "updated_at": "2026-07-23 04:13:20.232537+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents"], "entities": ["Prefix-GRPO", "TextCraft", "BabyAI", "ALFWorld", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes", "markdown": "https://wpnews.pro/news/from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes.md", "text": "https://wpnews.pro/news/from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes.txt", "jsonld": "https://wpnews.pro/news/from-trajectories-to-prefixes-reusing-teacher-trajectories-via-replayed-prefixes.jsonld"}}