IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

Researchers introduced Influence-Aware Policy Optimization (IAPO), a method that uses typed influence-dependency graphs to redistribute trajectory-level advantages for credit assignment in multi-turn service agents. Experiments with Qwen3-4B and Qwen3-8B showed IAPO outperformed multi-turn RL baselines on tau^2-Bench, UserBench, and AgentChangeBench, without compromising BFCL-v4 Multi-Turn function-calling performance.

arXiv:2608.24588v1 Announce Type: new Abstract: Large Language Model LLM agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization IAPO , which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning RL baselines across three service-agent benchmarks: {\tau^2}-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.