{"slug": "behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference", "title": "Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Prediction", "summary": "Researchers introduced behavior-aware auxiliary corrections for off-policy temporal-difference learning, replacing the standard covariance matrix with the behavior Bellman matrix in the TDC and TDRC algorithms. The new BA-TDC and BA-TDRC methods improve stability and performance in linear prediction tasks, with experiments showing the behavior-aware replacement alone benefits some tasks while regularization ensures robustness across harder settings. This work addresses a key design question for neural-network value approximation by separating the effects of behavior-aware geometry and regularization.", "body_md": "arXiv:2605.28855v1 Announce Type: new\nAbstract: Temporal-difference learning with function approximation can be unstable under off-policy sampling. TDC stabilizes off-policy TD through an auxiliary covariance correction, and TDRC further regularizes this correction in a single-timescale recursion. This paper studies a behavior-aware replacement of the auxiliary covariance geometry in the linear prediction setting, which is the standard local model for understanding the feature-space dynamics of value-function approximation. We first replace the TDC auxiliary matrix (C) by the behavior Bellman matrix (A_\\mu), yielding BA-TDC, and then regularize the same behavior-aware equation to obtain BA-TDRC. This two-step construction separates the contribution of behavior-aware geometry from the contribution of regularization. The linear analysis also provides a tractable model for an auxiliary-geometry design question that arises in neural-network value approximation, where feature covariances and temporal transition matrices jointly shape the last-layer correction dynamics. We give a finite-state mean-system formulation, prove fixed-point preservation and almost-sure convergence under a Hurwitz stability condition on the instantiated mean system, and compare deterministic mean rates through the spectral radius of the exact linear error recursion. Experiments on the two-state counterexample, Baird's counterexample, Random Walk, and Boyan Chain show that the behavior-aware replacement can be highly beneficial by itself on some tasks, but that regularization is necessary for robust performance across harder settings.", "url": "https://wpnews.pro/news/behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference", "canonical_source": "https://arxiv.org/abs/2605.28855", "published_at": "2026-05-29 04:00:00+00:00", "updated_at": "2026-05-29 04:20:10.088927+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "ai-research"], "entities": ["TDC", "TDRC", "BA-TDC", "BA-TDRC", "Baird's counterexample", "Random Walk", "Boyan Chain"], "alternates": {"html": "https://wpnews.pro/news/behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference", "markdown": "https://wpnews.pro/news/behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference.md", "text": "https://wpnews.pro/news/behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference.txt", "jsonld": "https://wpnews.pro/news/behavior-aware-auxiliary-corrections-for-off-policy-temporal-difference.jsonld"}}