Adapting to Changes in Agent Behavior via Finite-Depth Policy Sensitivity

A new arXiv paper (2610.07475v1) presents a finite-depth framework for estimating policy sensitivity in reinforcement learning, approximating the policy Hessian and mixed derivative from a reference environment to predict how a locally optimal policy shifts when another agent's behavior changes. The authors derive truncation-error bounds for the approximated derivatives and policy sensitivity that are nonincreasing with propagation depth and vanish at full-horizon propagation. In a belief-driven pursuit-evasion game, the method generally reduced derivative-estimation errors as propagation depth increased, outperformed baseline methods in estimation accuracy and policy adaptation, and its sensitivity-based initialization improved zero-shot return over direct transfer while aiding subsequent fine-tuning in the target environment.

arXiv:2610.07475v1 Announce Type: new Abstract: Adapting a reinforcement learning policy to changes in another agent's behavior typically requires a large amount of new interaction data. Policy sensitivity provides a first-order prediction of how a locally optimal policy changes with a behavioral parameter, but its computation requires second-order derivatives whose effects propagate across future interactions. We develop a finite-depth framework to estimate this sensitivity by approximating the policy Hessian and mixed derivative using information from a reference environment. The method features an adjustable propagation depth which determines where derivative propagation along the trajectory is truncated. We characterize the derivative contributions omitted by finite-depth propagation and derive truncation-error bounds for the approximated derivatives and resulting policy sensitivity. The bounds are nonincreasing with propagation depth and vanish at full-horizon propagation. Using a belief-driven pursuit-evasion game as a validation scenario, the proposed method generally achieves lower derivative-estimation errors as the propagation depth increases and outperforms the baseline methods in both estimation accuracy and policy adaptation. The sensitivity-based initialization improves zero-shot return over direct transfer, and also shows advantages for the subsequent fine-tuning in the target environment.