Reinforcement learning faces challenges in avoiding catastrophic forgetting in continual post-training for vision-language models. The new MRCL benchmark and CPO framework offer insights and solutions.
Continual post-training stands at the forefront of adapting vision-language models to ever-evolving tasks. In recent developments, reinforcement learning has gained traction over traditional supervised fine-tuning, based on the belief that it might reduce the likelihood of models forgetting previously acquired knowledge. But how accurate is this belief?
Introducing MRCL: A New Benchmark #
Enter MRCL, the Multimodal Reasoning Continual Learning benchmark, which serves as a fresh litmus test for this assumption. Built from diverse and recent multimodal datasets, MRCL challenges the prevailing notion that reinforcement learning is a panacea for catastrophic forgetting. The results? Reinforcement learning, while powerful, isn't immune to severe forgetting when continually trained. This finding is critical for those banking on reinforcement learning alone to maintain a model's knowledge base.
Continual Policy Optimization: A New Hope? #
This is where Continual Policy Optimization (CPO) steps in. CPO is a replay-free framework designed to address the limitations identified in reinforcement learning. By applying theoretically sound parameter-movement regularization, CPO effectively limits policy drift on prior tasks. Extensive experiments have shown that CPO consistently reduces forgetting while enhancing or, in some cases, even improving the capabilities of pretrained models.
On the Qwen3-VL-8B model, for example, CPO reduced catastrophic forgetting by an impressive 13.7% and improved the pretrained model's capabilities by 7.0%. These numbers aren't just digits on a page. they're a testament to the potential of CPO in enhancing model retention and capability.
Why the AI Community Should Take Notice #
So why should the AI community care about these findings? Because they highlight a key gap between theoretical potential and practical application. Reinforcement learning's promise of less forgetting sounds appealing, but MRCL shows that the reality is more complex. The AI Act text specifies the importance of rigorous testing and validation, and MRCL signifies a step towards that.
Brussels moves slowly. But when it moves, it moves everyone. As AI regulations evolve, understanding these dynamics becomes ever more vital. Will reinforcement learning soon fall from grace, or will frameworks like CPO allow it to fulfill its potential? The AI development community will need to weigh these questions carefully as they chart the future of model training.
Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
Benchmark A standardized test used to measure and compare AI model performance.
Catastrophic Forgetting When a neural network trained on new data suddenly loses its ability to perform well on previously learned tasks.
Fine-Tuning The process of taking a pre-trained model and continuing to train it on a smaller, specific dataset to adapt it for a particular task or domain.
Multimodal AI models that can understand and generate multiple types of data — text, images, audio, video.