Tail-Likelihood Reinforcement Learning

Researchers propose Tail-Likelihood Reinforcement Learning (TailRL), a method that optimizes the probability of exceeding reward thresholds rather than average reward, to better leverage rare high-reward rollouts. TailRL modifies the advantage function and is compatible with existing RL pipelines, showing improvements in object localization, maze navigation, GUI grounding, and code optimization by avoiding suboptimal solutions and benefiting more from additional samples at inference.

arXiv:2609.02987v1 Announce Type: new Abstract: Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning TailRL , which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of- k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.