{"slug": "fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim", "title": "Fewer Heads, Fewer Memories: Pruning Redundant Attention Heads Suppresses Verbatim Memorisation", "summary": "Zeroing out 729 task-redundant attention heads in GPT-2-XL (1.5B) before inference cut verbatim memorisation of known training sequences by 91.5% while improving WikiText-103 perplexity by 0.92, according to an arXiv paper (2610.40974v1). The authors report that attention-head redundancy can be exploited to suppress memorised-sequence reproduction at inference time without retraining or modifying training data, addressing privacy-leakage and intellectual-property concerns tied to verbatim recall.", "body_md": "arXiv:2610.40974v1 Announce Type: new \nAbstract: Large language models have been shown to memorise and reproduce verbatim sequences from their training data, raising concerns about privacy leakage, intellectual property exposure, and the limits of generalisation. Recent theoretical work suggests that attention heads may contribute to this behaviour by functioning as associative memory mechanisms, with memorisation capacity scaling with the number of heads. At the same time, empirical work has shown that many attention heads are redundant and can be pruned with limited impact on downstream performance. Despite these parallel findings, the effect of pruning redundant heads on memorised-sequence reproduction remains unexplored. In this work, we demonstrate that zeroing out a substantial fraction of attention heads prior to inference can significantly suppress verbatim memorisation while preserving, or even improving, language modelling performance on the benchmark used to identify redundant heads. Evaluating GPT-2-XL(1.5B) on known memorised sequences, we find that pruning 729 task-redundant heads reduces the memorisation count by 91.5%, while improving WikiText-103 perplexity by 0.92. These findings suggest that attention-head redundancy can be exploited to mitigate memorisation at inference time, without retraining or modifying the training data.", "url": "https://wpnews.pro/news/fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim", "canonical_source": "https://arxiv.org/abs/2610.40974", "published_at": "2026-11-02 05:00:00+00:00", "updated_at": "2026-11-02 05:19:50.364863+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["GPT-2-XL", "WikiText-103", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim", "markdown": "https://wpnews.pro/news/fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim.md", "text": "https://wpnews.pro/news/fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim.txt", "jsonld": "https://wpnews.pro/news/fewer-heads-fewer-memories-pruning-redundant-attention-heads-suppresses-verbatim.jsonld"}}