Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

Researchers at Qwen2.5-3B-Instruct found that supervised fine-tuning (SFT) causes greater disruption to internal computational circuits and more catastrophic forgetting than reinforcement learning (RL) when adapting large language models to scientific question-answering. The team introduced a head-level measure called differential circuit vulnerability to compare the two methods, revealing a mechanistic trade-off where SFT adapts faster but degrades circuits more severely. These findings suggest that RL's superior preservation of base circuits may explain its greater robustness against catastrophic forgetting.

arXiv:2605.28860v1 Announce Type: new Abstract: Fine-tuning large language models LLMs frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning RL retains prior capabilities more effectively than supervised fine-tuning SFT , attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.