From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

Researchers propose Golden-GRPO Injection (GRIN), a three-stage mixed-policy reinforcement learning framework for continual knowledge injection in large language models, which outperforms supervised fine-tuning on harder question types while matching it on basic fact recall. The framework introduces Golden-GRPO, a mixed-policy RL algorithm that injects a golden answer to provide learning signal even when on-policy rollouts fail, and two document-level benchmarks, Blank and Counter, targeting novel acquisition and counterfactual overwrite. The findings establish that mixed-policy RL enables knowledge absorption beyond what supervised fine-tuning can achieve.

arXiv:2608.25243v1 Announce Type: new Abstract: Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning SFT , which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection GRIN , a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.