Making Models Forget: Why Machine Unlearning Is Harder Than Training

Machine unlearning, the process of making AI models forget specific data without full retraining, is a difficult problem because neural networks store information in entangled weights, and exact methods like SISA are impractical for large models. Researchers are developing approximate unlearning techniques to balance forgetting, utility, and efficiency, driven by legal requirements like the EU's GDPR right to be forgotten.

You can train a large language model on trillions of tokens, but ask it to forget a single book say, a copyrighted novel it accidentally ingested and you might stumble. The model has no delete key. Its weights are a dense web of entangled associations, and removing one strand without disturbing the whole fabric turns out to be one of the most difficult problems in modern ML. This is Machine Unlearning : the process of making a model forget specific training data, sensitive information, or toxic behaviors without retraining from scratch. It sounds like a niche concern until you realize it sits at the intersection of law, safety, and economics. The EU’s GDPR grants individuals a “ right to be forgotten .” A deployed model might encode outdated facts, dangerous capabilities, or poisoned data from an adversarial attack. In each case, someone needs the model to unlearn something and perform full retraining, which can cost millions of dollars for frontier LLMs. Hence, it is not a practical answer. It is worth being precise about what unlearning is NOT . Deleting a record from a training database is straightforward; unlearning is about removing the influence that record already had on the model’s parameters. A model can reproduce copyrighted text even after the source documents are wiped from the training corpus, because the statistical patterns have been absorbed into its weights. Similarly, Retrieval-Augmented Generation RAG can ground a model in specific documents and reduce hallucinations, but it does nothing to erase knowledge already baked into the base model’s parameters. If the liability comes from weights embedded copyrighted material, personal data, or toxic behaviors , RAG is not a substitute for unlearning. The challenge is that selective forgetting, something humans do effortlessly, is structurally alien to how neural networks store knowledge. Foundation models do not keep their training examples in neat, labeled compartments. A fact learned from one document diffuses across billions of parameters and interacts with thousands of other concepts. Remove a data point, and its influence has already mixed with everything else the model knows about grammar, reasoning, and the world. This creates three conflicting objectives that any unlearning algorithm must juggle: 1. Forgetting : The targeted data must genuinely vanish, not just be hidden from casual queries. 2. Utility : The rest of the model’s capabilities should remain intact. 3. Efficiency : The fix must be cheaper than retraining. Satisfying all three is difficult. Full retraining on the dataset minus the forget set achieves perfect forgetting and preserves utility, but fails on efficiency. Conversely, adding noise to weights can obscure the unwanted data, but degrades performance across the board. Most practical research lives in the uncomfortable middle ground. Thus, the Machine unlearning can be done in two ways: Exact Unlearning and Approximate Unlearning. The cleanest theoretical guarantee is exact unlearning : producing a model that is statistically indistinguishable from one retrained from scratch without the forget data. The canonical framework here is SISA training Sharded, Isolated, Sliced, Aggregated , introduced by Bourtoule et al . SISA works by reorganizing training before unlearning is ever needed: If your data lives in one shard, unlearning it requires retraining only that shard — a significant computational saving. But the trade-offs are real: sharding reduces accuracy, hurts generalization, and demands storage for many intermediate checkpoints. For a 70-billion-parameter LLM trained on terabytes of internet data, SISA is largely a non-starter. Exact unlearning provides the strongest guarantees, but it is a pre-training strategy, not a post-hoc patch you can apply to an already-deployed model. Because exact methods are impractical for large pre-trained networks, the field has converged on approximate unlearning : updating the model to minimize the influence of the forget set, without claiming the result is identical to a retrained model. These methods dominate the literature, but they come with a critical caveat — residual traces of the data may remain, which matters when the data is copyrighted material or private medical records. Some of the methods in approximate unlearning are Gradient ascent, Random labelling, Gradient Difference, KL Divergence Regularization, Weight Pruning and Fine-tuning, Layer rest and Fine-tuning, Selective Synaptic Dampening, and Representation Misdirection for Unlearning. The most direct approach is gradient ascent . Where training minimizes loss on data, gradient ascent maximizes loss on the forget set, effectively pushing the model away from those examples. As one researcher describes it, this is “ like running training in reverse .” It is intuitive, but risky: aggressively updating weights to forget one thing can destabilize performance on unrelated tasks, a phenomenon the field calls catastrophic forgetting or, in this context, catastrophic unlearning . To limit the collateral damage, researchers have developed several regularized variants: A more structural variant is weight pruning and fine-tuning : prune parameters with the smallest L1-norms hypothesized to be weakly tied to general knowledge and potentially linked to the forget set , then recover utility by fine-tuning on retained data. Similarly, layer reset re-initializes the first or last k layers and fine-tunes — coarse, but computationally cheap. Some of the more promising recent methods target where knowledge lives rather than blindly adjusting weights. An interesting practical insight from recent IBM research is that these methods improve substantially when you categorize the forget set rather than treating it as a monolithic block. Separating “hate speech targeting women” from “hate speech targeting minorities” and unlearning each category independently yields better results than trying to forget both simultaneously — the parameter-level signatures are different, and the unlearning surgery can be more precise. Another family of methods borrows from differential privacy: add calibrated Laplace or Gaussian noise to weights or activations to statistically obscure the forget set’s influence, then fine-tune on the retain set to recover utility. These approaches can offer formal indistinguishability guarantees, though typically at some cost to model performance. More recently, adversarial frameworks have exposed a subtle weakness in many gradient-based methods. AMUN Adversarial Machine UNlearning and ARU Attack-and-Reset for Unlearning use membership inference attacks to verify unlearning rather than trusting loss-based proxies. The concern they address is real: a model might appear to have forgotten because it suppresses surface-level outputs, while the underlying parameters still encode the information. Adversarial probing can reveal this, which is why verification is as hard as the unlearning itself. Evaluation is where the field gets messy. Researchers have not settled on a standard protocol. Some papers report classification accuracy on the forget set; others measure distance to a fully retrained model; others use the error rate of membership inference attacks. This inconsistency makes it nearly impossible to compare methods directly. One of the most theoretically grounded evaluation approaches is model distance : quantifying how far the unlearned model deviates from the gold-standard model retrained from scratch without the forget set. If approximate unlearning aims to be “close enough” to exact unlearning, model distance is the tape measure. In practice, this distance can be computed in parameter space or function space. The simplest metric is the L₂ norm between the weight vectors of the unlearned model θᵤ and the retrained model θᵣ: However, small weight differences can produce large behavioral differences, and vice versa, because neural networks have redundant parameterizations. A more behaviorally meaningful metric is the expected output difference over the data distribution: where f₍θ₎ x represents the model’s logits or probability distribution over outputs. Some works use KL divergence between the output distributions, which captures not just pointwise disagreement but distributional divergence. The appeal of model distance is that it directly tests the core claim of approximate unlearning: that the post-unlearning model is functionally equivalent to one that never saw the data. The drawback is that computing it requires actually retraining the model from scratch — the very expense unlearning is trying to avoid — making it a useful benchmark for research but often impractical for production validation. Consider Microsoft’s experiment: they tested an unlearning algorithm by asking it to forget copyrighted Harry Potter books. But the evaluation immediately runs into edge cases. Should the model also forget every news article and social media post that mentions Harry Potter? Probably not — that is public-domain cultural knowledge. But drawing the line between “copyrighted expression” and “factual knowledge about the franchise” is ambiguous, and different applications will demand different boundaries. For sensitive personal data, one verification technique is the membership inference attack : can an attacker determine, from the model’s outputs, whether a specific individual’s data was in the training set? If the unlearning worked, the answer should be no. But verification is complicated by the fact that models can leak information through subtle behavioral differences before and after unlearning — the very process of forgetting can create traces that reveal what was erased. Unlearning is not free. As more data is removed, model performance on retained tasks tends to degrade, sometimes exponentially. This catastrophic unlearning remains an open challenge despite specialized loss functions designed to mitigate it. There is also a fairness concern. If the unlearning process disproportionately degrades accuracy for certain demographic groups, because the forget set was correlated with features those groups share. Then, the model may emerge less fair than before. Unlearning a biased dataset could, paradoxically, introduce new biases. And then there is the security dimension. The difference in model outputs pre- and post-unlearning can itself leak information. Attackers can run private information reconstruction attacks or compare model versions to infer what was deleted. Unlearning creates a new attack surface. In distributed and federated settings, these problems multiply. Federated learning trains models across decentralized devices without centralizing data, which is attractive for privacy. But federated unlearning removing a client’s contribution from a globally aggregated model introduces unique challenges. The server typically does not hold raw data, so it cannot simply retrain a shard. Instead, it must reverse the aggregation step using historical model updates, or maintain redundant structures like SISA-like cohorts across clients. The communication overhead, client dropout, and the difficulty of attributing specific parameter changes to specific clients make federated unlearning an active but largely unsolved research area. For production systems handling user data at the edge, this is a critical gap. Machine unlearning is not a band-aid you can slap on a deployed model after a privacy complaint arrives. It is a fundamental capability that needs to be designed into the ML lifecycle, alongside data governance, red teaming, and audit infrastructure. For smaller, structured models trained on labeled data, exact methods like SISA offer a viable path. For billion-parameter LLMs trained on raw internet scrapes, the field is still searching for approximate methods that are efficient, verifiable, and utility-preserving. The current frontier mixes gradient surgery, representation manipulation, and adversarial verification. But no approach yet offers the holy grail of guaranteed forgetting without retraining. The right mental model is not “how do we delete data from a model?” but “how do we train and update systems so that specific influences can be retracted later?” That reframing changes everything from dataset curation to deployment architecture. Until we solve it, every model shipped is, in some sense, a permanent record of everything it has ever seen. Making Models Forget: Why Machine Unlearning Is Harder Than Training https://pub.towardsai.net/making-models-forget-why-machine-unlearning-is-harder-than-training-725b8abab89d was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.