Gated Q-learning: Add Off-Policy Bias to Taste

Researchers introduced Gated Q-learning, a new algorithm that smoothly interpolates between Watkins' Q(λ) and Peng's Q(λ) to manage off-policy bias in multistep credit assignment for reinforcement learning. The method uses a continuous, state-action-dependent gating mechanism instead of importance sampling, and the authors prove its expected operator is a contraction mapping. Empirical evaluations show intermediate gating enables longer credit-assignment horizons and faster initial learning than either extreme.

arXiv:2607.28916v1 Announce Type: new Abstract: Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces Watkins' Q $\lambda$ , or ignore the bias to learn faster while injecting detrimental errors into the value estimates Peng's Q $\lambda$ . Modern off-policy estimators fail to resolve this tension, as importance-sampling ratios collapse under Q-learning's greedy target policy. We introduce Gated Q-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes. Rather than relying on importance sampling, our approach employs a continuous, state-action-dependent gating mechanism to selectively attenuate eligibility traces in an exploration-aware manner. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point. Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons, yielding faster initial learning than either extreme. Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off-policy bias in Q-learning agents.