Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation

Researchers proposed Reward Stimulation Implicit Q-Learning (RSIQL), a non-hierarchical offline goal-conditioned reinforcement learning method that adds reward signals at progress-making intermediate states identified by an auxiliary goal-conditioned value function. RSIQL improved over goal-conditioned IQL on average and matched hierarchical offline goal-conditioned methods on the D4RL goal-reaching benchmarks and OGBench while keeping a flat policy structure, according to the arXiv paper 2609.19414v1. The work targets sparse-reward, long-horizon offline settings where goal-completion information is temporally distant from the early decisions that enable success.

arXiv:2609.19414v1 Announce Type: new Abstract: Goal-conditioned reinforcement learning aims to learn policies that reach specified goals, but remains challenging in offline settings with sparse rewards and long-horizon dependencies. In such settings, goal-completion information can be temporally distant from the early decisions that enable success, while offline value estimation introduces additional error. We study this issue from a reward-propagation perspective and show, in a stylized delayed-goal setting, how goal-directed value separation can become small relative to local estimation error. Motivated by this analysis, we propose Reward Stimulation Implicit Q-Learning RSIQL , a simple non-hierarchical method that introduces additional reward signals at progress-making intermediate states in offline trajectories. RSIQL uses an auxiliary goal-conditioned value function to identify intermediate states estimated to make progress toward the goal and applies reward stimulation to provide less-delayed training supervision. Unlike hierarchical methods, RSIQL does not learn a separate high-level subgoal policy. Experiments on D4RL goal-reaching benchmarks and OGBench show that RSIQL improves over goal-conditioned IQL on average and achieves performance competitive with hierarchical offline goal-conditioned methods, while retaining a simple flat policy structure.