# Probing Knowledge Recovery in Unlearned Models > Source: > Published: 2026-08-11 02:55:09+00:00 Machine unlearning aims to remove specific knowledge from AI models without the cost of retraining. It is a proposed technique for removing harmful knowledge, personal or copyrighted information, correcting mistakes, and more. However, recent work, particularly by [Łucki et al., ](https://arxiv.org/abs/2409.18025)has shown that many current unlearning methods are not robust and are vulnerable to knowledge recovery. Unlearning methods are usually evaluated based on their performance on a forget set, but low performance on a forget set does not show whether the knowledge has been genuinely removed or merely suppressed. This project investigates whether we can reliably determine when knowledge has been suppressed rather than removed. All the experiments are conducted on six unlearned Llama-3-8B-Instruct checkpoints, each unlearned using a different method. **Methods:** [RMU](https://huggingface.co/ScaleAI/mhj-llama3-8b-rmu),[ ILU-RMU](https://huggingface.co/OPTML-Group/ILU-RMU-WMDP-llama3-8b-instruct),[ IDK-AP](https://huggingface.co/OPTML-Group/IDK-AP-WMDP-llama3-8b-instruct), [GradDiff](http://optml-group/GradDiff-WMDP-llama3-8b-instruct), [NPO](http://optml-group/NPO-WMDP-llama3-8b-instruct), [NPO-ILU](http://optml-group/NPO-ILU-WMDP-llama3-8b-instruct)The models used are existing unlearned checkpoints: the RMU checkpoint is from **Benchmarks: **[WMDP](https://www.wmdp.ai/) is the main benchmark. All the checkpoints were unlearned on the WMDP-Bio set. Other than that, MMLU is also used for measuring general capability, and AdvBench for measuring refusal rate. **Controls: **The direction ablation experiments were compared against a matched control and several random controls. For the unrelated SFT experiments, they were compared against matched fine-tuning on the base Llama model. **Metrics: **The checkpoints are evaluated on WMDP-Bio and MMLU using the[ LM Eval Harness.](https://github.com/EleutherAI/lm-evaluation-harness/) For WMDP-Bio, it gives forced-choice accuracy based on the log-likelihood assigned to the four answer choices. For MMLU, it gives accuracy. Other than that, qualitative evaluations with free-text generation and refusal rate using AdvBench are also measured. One plausible explanation for knowledge suppression is that the unlearning method hides the knowledge behind refusal behavior.[ Arditi et al. ](https://arxiv.org/abs/2406.11717)found that refusal behavior is mediated by a single direction in the activation space of LLMs. If so, we can test whether knowledge is being suppressed by ablating the refusal direction from the unlearned checkpoints and observing whether WMDP-Bio accuracy increases. We can only test this meaningfully if we can find a direction that controls refusal behavior in the model. Before the recovery test, I therefore validated the refusal direction for each checkpoint behaviorally using held-out[ AdvBench](https://huggingface.co/datasets/walledai/AdvBench) prompts that were not included in the direction extraction. Each direction was also compared against a matched control direction constructed in the same way, as well as random control directions, to ensure that any recovery was not simply caused by ablating an arbitrary direction in the checkpoint. **Result:** A clean refusal direction was obtained only from the ILU-RMU checkpoint, reducing the refusal rate from 98% to 0% when ablated. However, the ablation did not recover WMDP-Bio accuracy. The 95% confidence interval excluded improvements larger than 1.4 percentage points, and both the matched-control and random-control ablations were also null. For the remaining methods, no clean refusal direction was obtained: For the only checkpoint where the experiment could be conducted properly, there was no knowledge recovery. At least for ILU-RMU, the unlearning and refusal mechanisms appear to be separate. Also, unlearning seems to affect the refusal behavior of the model. Checkpoints like IDK-AP and RMU show a lower refusal rate than the base model. Arditi & Chughtai propose a different recovery probe. They found knowledge recovery after ablating a forget-domain representation direction from the RMU-unlearned Zephyr-7B model. The direction is constructed by taking the mean difference in activations between the base model and the unlearned model when run on forget-set prompts. So I tried to replicate the results for the Llama-3-8B-Instruct RMU model using this probe and extended the test to other methods. **Result:** The base Llama-3-8B-Instruct WMDP-Bio accuracy is 73.2%. | | | | RMU | 28.0 | 53.9 | 57% | ILU-RMU | 34.3 | 57.9 | 61% | NPO | 26.6 | 56.4 | 64% | IDK-AP | 34.9 | 42.1 | 19% | *The NPO result is difficult to interpret because the model generates incoherent text regardless of the input, even though WMDP accuracy increased.* Both RMU and ILU-RMU showed a substantial increase in WMDP-Bio accuracy after ablation, which is consistent with prior work. For the other methods, NPO and IDK-AP showed recovery, with NPO recovering 64% of the gap. MMLU remained flat for all methods after the direction ablation, and the matched control and random controls didn't cause any significant recovery. GradDiff and NPO-ILU didn't show any confirmed recovery. The four loss-based methods were tested across 16 layer and pooling configurations each. Unlike the refusal-direction probe, this probe increased benchmark accuracy for some methods, so recoverable forget-set representations are present in at least some of the current methods. Previous probes were about whether direction ablation can recover forgotten knowledge. As a follow-up, this probe tests whether doing supervised fine-tuning on unrelated data would recover the forgotten knowledge. Unrelated SFT has been used as a recovery method in the [WMDP paper](https://arxiv.org/abs/2403.03218) and[ Łucki et al.](https://arxiv.org/abs/2409.18025), [Hu et al.](https://arxiv.org/html/2406.13356v1), and [Deeb and Roger.](https://arxiv.org/abs/2410.08827) Previous works used WMDP retain corpus or WikiText as the fine-tune data. But here the six checkpoints are fine-tuned on GSM8K, a dataset further removed from biology. **Result:** The base Llama-3-8B-Instruct WMDP-Bio accuracy is 73.2%. | | | | | | | RMU | 28.1 | 69.5 | +41.4 | 62.5 | 59.2 | −3.3 | ILU-RMU | 34.0 | 57.8 | +23.8 | 65.3 | 59.4 | −5.9 | NPO | 26.8 | 49.3 | +22.5 | 54.6 | 52.8 | −1.8 | IDK-AP | 35.2 | 40.4 | +5.2 | 47.3 | 41.1 | −6.2 | GradDiff | 26.4 | 54.1 | +27.7 | 24.9 | 54.7 | +29.8 | NPO-ILU | 27.2 | 44.3 | +17.1 | 54.7 | 57.1 | +2.4 | The WMDP-Bio accuracy increased for five out of the six methods. There was significant recovery for RMU, ILU-RMU, and NPO, while their MMLU accuracy remained flat. Even though the WMDP-Bio accuracy increased for the GradDiff checkpoint after fine-tuning, its MMLU accuracy also increased, which suggests that the increase in WMDP accuracy might be due to a general capability increase. There was a slight increase for NPO-ILU, but since the model generates incoherent text, this is an argmax shift rather than genuine recovery. The knowledge recovery for previous interventions was measured based on WMDP-Bio forced-choice accuracy, which is the log likelihood of the four answer choices. But this doesn't show whether there is genuine recovery when the model responses are generated. To understand what was generated after the interventions, I generated and analysed the text for the prompts for which the answer changed after the interventions. The responses were classified into these categories: For the base Llama model, the contradictory rate is ~6.7%. | | | | RMU flip | 40% | 23% | 37% | RMU stable correct | 63% | 0% | 37% | ILU-RMU flip | 47% | 10% | 43% | ILU-RMU stable correct | 60% | 10% | 30% | Only 40 to 47% of texts showed genuine reasoning behind the correct answer. As a control, I compared this with the generated text for the prompts for which the answers remained correct throughout the interventions. This showed that much of the contradiction was a property of the checkpoint itself rather than the recovery intervention. A smaller, unblinded evaluation was also conducted after SFT, and a similar pattern was observed. The degeneracy of the NPO and NPO-ILU checkpoints became apparent when free text was generated, even though NPO showed a substantial increase in WMDP-Bio accuracy after the interventions. This shows that benchmark recovery and reasoning recovery are not the same, and we cannot conclude knowledge recovery based on benchmark accuracy alone. From the results, there was an increase in benchmark accuracy for the RMU, ILU-RMU, and NPO checkpoints in both forget-representation and SFT probes. So for these methods, two independent probes gave similar results. The refusal-direction ablation probe didn't recover knowledge for the method for which a valid direction was obtained. This suggests that for ILU-RMU, refusal is not the main suppression mechanism. More broadly, for the unlearned checkpoints evaluated, there appears to be recoverable forget-set representation in some of them. A single probe is not enough for testing knowledge recovery, and different probes reveal different residual information or access. Also, WMDP accuracy alone is not a sufficient measure for knowledge recovery. Qualitative evaluation showed that even though there was an increase in WMDP accuracy, the models still responded with contradicting reasoning or incoherent text. Unlearning also changed other behaviors that are not related to the forget set. Several checkpoints had reduced refusal rates, and checkpoints like NPO and NPO-ILU produced incoherent responses. This is also not visible with benchmark accuracy alone. *This work has been done as part of BlueDot Impact's Technical AI Safety Project Sprint. * *The code and experiment details are available in the **GitHub repo.*