World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning

A new test-time framework called World-Model Policy Arbiter (WMPA) raises the macro-average success rate on 18 state-based OGBench datasets from 44% to 58% by rolling out frozen goal-conditioned policies in a learned state-space world model and executing the highest-scoring policy for a short commitment interval, according to the arXiv paper 2610.10932v1. The approach requires no policy retraining or privileged task-specific knowledge and produced statistically significant gains on 12 datasets, including +33 percentage points on cube-double-play and +36 percentage points on scene-play. The work addresses the problem that no single offline goal-conditioned reinforcement learning algorithm performs best across environments, goals, and task phases.

arXiv:2610.10932v1 Announce Type: new Abstract: Offline goal-conditioned reinforcement learning GCRL has produced a diverse set of goal-reaching algorithms, yet no single algorithm performs best across environments, goals, and even different phases of the same task. Rather than deploying only the best-performing policy, we ask whether a set of frozen goal-conditioned policies can be used collectively as a portfolio, deciding at every state which policy should act. Choosing a policy at each state is not straightforward. The policies' own value functions cannot be compared directly: they may use different scales, and some policies have no value function. We need to judge each policy by the states it is likely to reach, even though we can execute only one policy at a time. We also need to avoid switching so often that control becomes unstable. To address these challenges, we introduce World-Model Policy Arbiter WMPA , a test-time framework that, given a set of frozen policies as input, rolls out each frozen policy in a learned state-space world model, evaluates the imagined futures with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval before the next round of arbitration policy selection . WMPA assumes access to a bank of frozen goal-conditioned policies and requires neither policy retraining nor privileged task-specific knowledge. Under the official OGBench evaluation protocol on 18 state-based datasets spanning maze navigation as well as cube, scene, and puzzle manipulation, WMPA improves the macro-average success rate from the 44% achieved by the best policy selected per dataset to 58%, with statistically significant gains on 12 datasets. These gains include +33 percentage points on cube-double-play and +36 percentage points on scene-play.