CAST: Game Solvers as Turn-Level Teachers for LLM Agents

Researchers propose CAST (Credit Assignment from Solver Teachers), a method that converts game solver state-value changes into turn-level signals to improve reinforcement learning for LLM agents in long-horizon games. In tests on Sokoban, Minesweeper, Rush Hour, ALFWorld, and WebShop, CAST outperformed all trained baselines on every game under both in-domain and unseen-difficulty evaluation, achieving the highest average zero-shot performance on ALFWorld and WebShop. The code is available at https://github.com/Wloner0809/CAST.

arXiv:2607.25308v1 Announce Type: new Abstract: Training large language models LLMs to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards RLVR relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST Credit Assignment from Solver Teachers , which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.