Which Successes Make Good References? Action-Sequence Alignment for Multi-Turn On-Policy Distillation

A new arXiv paper (2611.13135v1) proposes selecting successful reference trajectories for multi-turn on-policy distillation by action-sequence alignment, using longest common subsequence (LCS) as an observable proxy for trajectory compatibility. The authors cast successful-reference selection as a free-target optimal transport problem that reduces to assigning each failed trajectory its most compatible successful reference. Experiments on ALFWorld and the airline domain of τ³-bench show LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics, and airline-trained models transferred to the retail domain with LCS improving all three metrics.

arXiv:2611.13135v1 Announce Type: new Abstract: On-policy distillation OPD reduces train-inference distribution mismatch by supervising a student on its own trajectories. For multi-turn tool-use agents, conditioning the teacher on a validated same-task successful trajectory as a reference can provide useful hindsight, but different successes may reach incompatible interaction states and offer misleading supervision. We cast successful-reference selection as a free-target optimal transport problem, which reduces to assigning each failed trajectory its most compatible successful reference. Because latent state paths are hidden, we use action-sequence alignment as an observable proxy for compatibility and instantiate it with longest common subsequence LCS , which identifies a shared ordered backbone of actions while allowing recoverable detours. This perspective makes trajectory compatibility the central criterion for reference selection in multi-turn OPD. Experiments on ALFWorld and the airline domain of $\tau^3$-bench show that LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics. Without further optimization, the airline-trained models also transfer to the retail domain, where LCS improves all three metrics.