Does routine compression undo LLM unlearning? A short project

A two-week project by a BlueDot Project cohort participant found that standard post-training compression processes (quantization, magnitude pruning, SVD truncation) minimally reverse unlearning on the TOFU forget10 dataset with Llama-3.2-1B-Instruct across NPO, SimNPO, and IdkDPO methods. The strongest reversal came from magnitude pruning at 10-20% sparsity on NPO unlearning, recovering up to 42% of the ceiling-baseline gap, but qualitative inspection showed no resurfacing of unlearned facts in free generation.

I completed this project over 2 weeks as part of a BlueDot Project cohort. It was my first solo project and I learned a lot Feedback is super welcome : Code and full results: GitHub TLDR: I tested if standard post-training compression processes quantization, magnitude pruning, SVD truncation reverses unlearning on TOFU forget10 / Llama-3.2-1B-Instruct , across a few unlearning methods NPO, SimNPO, IdkDPO . I found that unlearning reversal is minimal across all combinations; however, the strongest reversal came from magnitude pruning in a narrow 10-20% sparsity window on NPO unlearning up to 42% of the ceiling–baseline gap recovered . Once a model is released as open-weight, it becomes easy to finetune away any refusal training that has gone into it. So, unlearning methods, which actually remove knowledge from the weights, become especially important for open-weight models. However, it is unclear if many unlearning methods are robust or can be reversed easily. There is already prior evidence https://arxiv.org/abs/2410.16454 that quantization specifically can reverse some unlearning. So in this project, I expand on this to investigate if reversal can come from standard, non-adversarial compression operations, which are routine processes that most models undergo on the way to deployment, including quantization and others. forget10 , Llama-3.2-1B-Instruct . forget Q A Prob teacher-forced prob of the correct forget-set answer . model utility — any cell where this collapsed ≤ 0.01 is discarded as uninterpretable. tofu ... full model trained on all data, no unlearning = 0.881 tofu ... retain90 model never trained on the forget set, no unlearning = 0.116. recovery = compressed − baseline / ceiling − baseline . Essentially measures how much closer the model moves from compressed to ceiling from the baseline, in terms of the probability mass it places on the correct answer tokens. forget Q A Prob : NPO 0.209, SimNPO 0.075, IdkDPO 0.017 — all at/below the floor, confirming effective unlearning. Compression processes Qualitative inspection shows that even though there is recovery on teacher-forced answer probabilities what forget Q A Prob measures , it is not enough to resurface the unlearned answers in free generation. When observing model answers of the NPO baseline vs. the NPO 4-bit quantization and vs. the NPO 20%-pruned on 20 forget-set questions, I found that still neither of these reproduces the exact memorized TOFU facts that were unlearned names, book titles, awards . SimNPO and IdkDPO are more robust than NPO. Unlike NPO, these unlearning methods had ≤3% recovery in all non-destructive cells. However, NPO did also have a higher baseline, which gave it more distance to recover. Especially given the limited time I spent on this, and my own lack of experience with research projects, there are some limitations that I'm aware of. Future works could be devoted to filling in these gaps: forget10 forget-set questions. Llama-3.2-1B-Instruct , 1 TOFU split forget10 , and 3 unlearning methods.There are no doubt many limitations that I'm not aware of too. Super open to any and all feedback The main technical takeaway is that standard compression isn't a practical attack here. No type restores knowledge to the ceiling; the strongest recovery case was only still 42%. However, the pruning peak result shows the effect can be larger than the quantization result that originally motivated this project. From another perspective, I didn't do this project just to answer the research question. I wanted to learn, by myself, more about what it is like to be a researcher. Some key takeaways on this front over the past few weeks: I'm really hoping to get deeper into the field of AI safety, and my next goal is to get involved in university research. Now, I feel a lot more prepared to try to chase that