Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

Researchers introduced Critic-Free Pretraining (CFP), a new paradigm for offline-to-online reinforcement learning that skips offline critic training to avoid biased value estimates, allowing a freshly initialized critic to adapt during online fine-tuning. CFP matches or improves upon conventional O2O algorithms across diverse tasks, with notable gains on challenging benchmarks, according to the arXiv paper 2608.10473v1.

arXiv:2608.10473v1 Announce Type: new Abstract: Offline-to-online O2O reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce \textbf{C}ritic-\textbf{F}ree \textbf{P}retraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.