TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

TreeHCA, a new reinforcement-learning method for training multi-turn LLM agents, outperformed all baselines in aggregate exact match across seven question-answering benchmarks and three Qwen backbones, according to an arXiv paper (arXiv:2610.12828v1). TreeHCA couples hindsight-guided exploration, which revisits intermediate histories and samples alternative continuations to build a tree of reasoning and tool interactions, with hindsight credit assignment, and derives an inverse-hindsight importance correction including an exact finite-pool identity. The method addresses outcome-supervised GRPO's limitation of assigning the same advantage to every token in an agent trajectory, leaving useful and unhelpful decisions undifferentiated; code is available at github.com/TheBlackCat22/TreeHCA.

arXiv:2610.12828v1 Announce Type: new Abstract: Outcome-supervised group-relative policy optimization GRPO assigns the same advantage to every token in an LLM agent trajectory, leaving useful and unhelpful decisions undifferentiated. Hindsight credit assignment HCA can refine this feedback, but its estimates rely on trajectories containing informative alternatives at consequential histories. We introduce TreeHCA, which couples hindsight-guided exploration with hindsight credit assignment. TreeHCA uses changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, forming a tree of reasoning and tool interactions. Because this exploration changes the distribution of sampled continuations, we derive the corresponding inverse-hindsight importance correction, including an exact finite-pool identity. We use this structure in a normalized tree backup that propagates terminal outcomes into turn-level advantages. This shared hindsight structure is TreeHCA's central design principle, linking informative exploration with hindsight-guided credit assignment. Across seven question-answering benchmarks and three Qwen backbones, TreeHCA outperforms all baselines in aggregate exact match, validating the effectiveness of paired exploration and credit assignment for multi-turn LLM agent training. Our code is available at github.com/TheBlackCat22/TreeHCA.