{"slug": "damage-aware-bandit-pruning-for-vision-and-language-transformers", "title": "Damage-Aware Bandit Pruning for Vision and Language Transformers", "summary": "A new arXiv paper (2609.05448v1) formulates structured post-training pruning of vision and language transformers as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget, masking attention heads and MLP channel groups on calibration batches. Across five seeds on WikiText-2, LAMBADA, and Imagenette with GPT-2, OPT, Pythia, Qwen2.5, SmolLM2, ViT-B/16, DeiT-Tiny, and Swin-Tiny, the bandit methods usually reduced degradation relative to budgeted greedy selection, with 23 of 28 highlighted comparisons having bootstrap confidence intervals excluding zero and 11 paired tests at p < 0.05. The authors note the selected units are functionally zeroed in the original dense checkpoint, so the reported effects represent effective structural suppression rather than physical compression or measured speedup.", "body_md": "arXiv:2609.05448v1 Announce Type: new \nAbstract: Structured post-training pruning of transformers requires selecting complete functional units whose suppression causes limited degradation. We formulate structured-unit selection for language and vision transformers as a damage-aware multi-armed bandit problem under a fixed candidate-evaluation budget. Attention heads and MLP channel groups are temporarily masked on calibration batches. Paired damage is the masked loss minus the base loss on the same batch, reducing batch-to-batch variation. A smooth bounded reward drives either a UCB-style policy or fractional-Beta Thompson Sampling, and the final mask is constructed sequentially by adding one unit at each step. The selected units are functionally zeroed in the original dense checkpoint; therefore, the reported parameter effects represent effective structural suppression rather than physical compression or measured speedup. Experiments on WikiText-2, LAMBADA, and Imagenette cover GPT-2, OPT, Pythia, Qwen2.5, SmolLM2, ViT-B/16, DeiT-Tiny, and Swin-Tiny, with comparisons against random, magnitude, static-saliency, and budgeted-greedy selection. Across five seeds, the bandit methods usually reduce degradation relative to budgeted greedy in the paired language-model comparisons. Of 28 comparisons highlighted in the paper, 23 bootstrap confidence intervals exclude zero and 11 paired tests have p < 0.05; six have q < 0.05 after Benjamini-Hochberg correction across the full family of 116 dataset-wise tests. Matched-evaluation results for ViT-B/16 and Swin-Tiny indicate that their gains are not explained solely by a larger candidate-evaluation budget.", "url": "https://wpnews.pro/news/damage-aware-bandit-pruning-for-vision-and-language-transformers", "canonical_source": "https://arxiv.org/abs/2609.05448", "published_at": "2026-09-10 04:00:00+00:00", "updated_at": "2026-09-10 04:21:05.856330+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "computer-vision", "natural-language-processing"], "entities": ["arXiv", "GPT-2", "OPT", "Pythia", "Qwen2.5", "SmolLM2", "ViT-B/16", "DeiT-Tiny"], "alternates": {"html": "https://wpnews.pro/news/damage-aware-bandit-pruning-for-vision-and-language-transformers", "markdown": "https://wpnews.pro/news/damage-aware-bandit-pruning-for-vision-and-language-transformers.md", "text": "https://wpnews.pro/news/damage-aware-bandit-pruning-for-vision-and-language-transformers.txt", "jsonld": "https://wpnews.pro/news/damage-aware-bandit-pruning-for-vision-and-language-transformers.jsonld"}}