{"slug": "does-routine-compression-undo-llm-unlearning-a-short-project", "title": "Does routine compression undo LLM unlearning? A short project", "summary": "A two-week project by a BlueDot Project cohort participant found that standard post-training compression processes (quantization, magnitude pruning, SVD truncation) minimally reverse unlearning on the TOFU forget10 dataset with Llama-3.2-1B-Instruct across NPO, SimNPO, and IdkDPO methods. The strongest reversal came from magnitude pruning at 10-20% sparsity on NPO unlearning, recovering up to 42% of the ceiling-baseline gap, but qualitative inspection showed no resurfacing of unlearned facts in free generation.", "body_md": "*I completed this project over 2 weeks as part of a **BlueDot Project** cohort. It was my first solo project and I learned a lot! Feedback is super welcome :)*\n\n**Code and full results: ****GitHub**\n\n**TLDR:** I tested if standard post-training compression processes (quantization, magnitude pruning, SVD truncation) reverses unlearning on TOFU `forget10`\n\n/ `Llama-3.2-1B-Instruct`\n\n, across a few unlearning methods (NPO, SimNPO, IdkDPO). I found that unlearning reversal is minimal across all combinations; however, the strongest reversal came from magnitude pruning in a narrow 10-20% sparsity window on NPO unlearning (up to 42% of the ceiling–baseline gap recovered).\n\nOnce a model is released as open-weight, it becomes easy to finetune away any refusal training that has gone into it. So, unlearning methods, which actually remove knowledge from the weights, become especially important for open-weight models. However, it is unclear if many unlearning methods are robust or can be reversed easily. There is already prior [evidence](https://arxiv.org/abs/2410.16454) that quantization specifically can reverse some unlearning. So in this project, I expand on this to investigate if reversal can come from standard, non-adversarial compression operations, which are routine processes that most models undergo on the way to deployment, including quantization and others.\n\n`forget10`\n\n, `Llama-3.2-1B-Instruct`\n\n.`forget_Q_A_Prob`\n\n(teacher-forced prob of the correct forget-set answer).`model_utility`\n\n— any cell where this collapsed (≤ 0.01) is discarded as uninterpretable.`tofu_..._full`\n\n(model trained on all data, no unlearning) = 0.881`tofu_..._retain90`\n\n(model never trained on the forget set, no unlearning) = 0.116. `recovery = (compressed − baseline)/(ceiling − baseline)`\n\n. Essentially measures how much closer the model moves from compressed to ceiling from the baseline, in terms of the probability mass it places on the correct answer tokens.`forget_Q_A_Prob`\n\n): NPO 0.209, SimNPO 0.075, IdkDPO 0.017 — all at/below the floor, confirming effective unlearning.**Compression processes**\n\n**Qualitative inspection shows that even though there is recovery on teacher-forced answer probabilities (what **`forget_Q_A_Prob`\n\n** measures), it is not enough to resurface the unlearned answers in free generation. **When observing model answers of the NPO baseline vs. the NPO 4-bit quantization and vs. the NPO 20%-pruned on 20 forget-set questions, I found that still neither of these reproduces the exact memorized TOFU facts that were unlearned (names, book titles, awards).\n\n**SimNPO and IdkDPO are more robust than NPO.** Unlike NPO, these unlearning methods had ≤3% recovery in all non-destructive cells. However, NPO did also have a higher baseline, which gave it more distance to recover.\n\nEspecially given the limited time I spent on this, and my own lack of experience with research projects, there are some limitations that I'm aware of. Future works could be devoted to filling in these gaps:\n\n`forget10`\n\nforget-set questions.`Llama-3.2-1B-Instruct`\n\n), 1 TOFU split (`forget10`\n\n), and 3 unlearning methods.There are no doubt many limitations that I'm not aware of too. Super open to any and all feedback!\n\nThe main technical takeaway is that standard compression isn't a practical attack here. No type restores knowledge to the ceiling; the strongest recovery case was only still 42%. However, the pruning peak result shows the effect can be larger than the quantization result that originally motivated this project.\n\nFrom another perspective, I didn't do this project just to answer the research question. I wanted to learn, by myself, more about what it is like to be a researcher. Some key takeaways on this front over the past few weeks:\n\nI'm really hoping to get deeper into the field of AI safety, and my next goal is to get involved in university research. Now, I feel a lot more prepared to try to chase that!", "url": "https://wpnews.pro/news/does-routine-compression-undo-llm-unlearning-a-short-project", "canonical_source": "https://www.lesswrong.com/posts/jXhHH658J4xzWjCu8/does-routine-compression-undo-llm-unlearning-a-short-project", "published_at": "2026-07-20 22:11:21+00:00", "updated_at": "2026-07-20 22:31:11.556809+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research"], "entities": ["BlueDot Project", "TOFU", "Llama-3.2-1B-Instruct", "NPO", "SimNPO", "IdkDPO"], "alternates": {"html": "https://wpnews.pro/news/does-routine-compression-undo-llm-unlearning-a-short-project", "markdown": "https://wpnews.pro/news/does-routine-compression-undo-llm-unlearning-a-short-project.md", "text": "https://wpnews.pro/news/does-routine-compression-undo-llm-unlearning-a-short-project.txt", "jsonld": "https://wpnews.pro/news/does-routine-compression-undo-llm-unlearning-a-short-project.jsonld"}}