I completed this project over 2 weeks as part of a BlueDot Project cohort. It was my first solo project and I learned a lot! Feedback is super welcome :)
**Code and full results: **GitHub
TLDR: I tested if standard post-training compression processes (quantization, magnitude pruning, SVD truncation) reverses unlearning on TOFU forget10
/ Llama-3.2-1B-Instruct
, across a few unlearning methods (NPO, SimNPO, IdkDPO). I found that unlearning reversal is minimal across all combinations; however, the strongest reversal came from magnitude pruning in a narrow 10-20% sparsity window on NPO unlearning (up to 42% of the ceiling–baseline gap recovered).
Once a model is released as open-weight, it becomes easy to finetune away any refusal training that has gone into it. So, unlearning methods, which actually remove knowledge from the weights, become especially important for open-weight models. However, it is unclear if many unlearning methods are robust or can be reversed easily. There is already prior evidence that quantization specifically can reverse some unlearning. So in this project, I expand on this to investigate if reversal can come from standard, non-adversarial compression operations, which are routine processes that most models undergo on the way to deployment, including quantization and others.
forget10
, Llama-3.2-1B-Instruct
.forget_Q_A_Prob
(teacher-forced prob of the correct forget-set answer).model_utility
— any cell where this collapsed (≤ 0.01) is discarded as uninterpretable.tofu_..._full
(model trained on all data, no unlearning) = 0.881tofu_..._retain90
(model never trained on the forget set, no unlearning) = 0.116. recovery = (compressed − baseline)/(ceiling − baseline)
. Essentially measures how much closer the model moves from compressed to ceiling from the baseline, in terms of the probability mass it places on the correct answer tokens.forget_Q_A_Prob
): NPO 0.209, SimNPO 0.075, IdkDPO 0.017 — all at/below the floor, confirming effective unlearning.Compression processes
**Qualitative inspection shows that even though there is recovery on teacher-forced answer probabilities (what **forget_Q_A_Prob
** measures), it is not enough to resurface the unlearned answers in free generation. **When observing model answers of the NPO baseline vs. the NPO 4-bit quantization and vs. the NPO 20%-pruned on 20 forget-set questions, I found that still neither of these reproduces the exact memorized TOFU facts that were unlearned (names, book titles, awards).
SimNPO and IdkDPO are more robust than NPO. Unlike NPO, these unlearning methods had ≤3% recovery in all non-destructive cells. However, NPO did also have a higher baseline, which gave it more distance to recover.
Especially given the limited time I spent on this, and my own lack of experience with research projects, there are some limitations that I'm aware of. Future works could be devoted to filling in these gaps:
forget10
forget-set questions.`Llama-3.2-1B-Instruct`
), 1 TOFU split (`forget10`
), and 3 unlearning methods.There are no doubt many limitations that I'm not aware of too. Super open to any and all feedback!
The main technical takeaway is that standard compression isn't a practical attack here. No type restores knowledge to the ceiling; the strongest recovery case was only still 42%. However, the pruning peak result shows the effect can be larger than the quantization result that originally motivated this project.
From another perspective, I didn't do this project just to answer the research question. I wanted to learn, by myself, more about what it is like to be a researcher. Some key takeaways on this front over the past few weeks: I'm really hoping to get deeper into the field of AI safety, and my next goal is to get involved in university research. Now, I feel a lot more prepared to try to chase that!