content type paperpublished August 2026 IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
AuthorsYixiao Li†, Xianzhi Du, Ajay Jaiswal‡, Tao Lei, Tuo Zhao†, Chong Wang, Jianyu Wang
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
AuthorsYixiao Li†, Xianzhi Du, Ajay Jaiswal‡, Tao Lei, Tuo Zhao†, Chong Wang, Jianyu Wang
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token efficiency compared to training target-size models from scratch. In this paper, we advocate incorporating enlarged model pretraining, which is often ignored in previous works, into pruning. We study the enlarge-and-prune pipeline as an integrated system to address two critical questions: whether it is worth pretraining an enlarged model even when the model is never deployed, and how to optimize the entire pipeline for better pruned models. We propose an integrated enlarge-and-prune pipeline, which combines enlarge model training, pruning, and recovery under a single cosine annealing learning rate schedule. This approach is further complemented by a novel iterative structured pruning method for gradual parameter removal. The proposed method helps to mitigate the knowledge loss caused by the rising learning rate in naive enlarge-and-prune pipelines and enable effective redistribution of model capacity among surviving neurons, facilitating smooth compression and enhanced performance. We conduct comprehensive experiments on compressing 2.8B models to 1.3B with up to 2T tokens in pretraining. It demonstrates the integrated approach not only provides insights into the token efficiency of enlarged model pretraining but also achieves superior performance of pruned models.
Modern neural networks are growing not only in size and complexity but also in inference time. One of the most effective compression techniques — channel pruning — combats this trend by removing channels from convolutional weights to reduce resource consumption. However, removing channels is non-trivial for multi-branch segments of a model, which can introduce extra memory copies at inference time. These copies incur increase latency — so much…
PDP: Parameter-free Differentiable Pruning is All You Need
July 24, 2023research area Methods and Algorithms, research area Tools, Platforms, Frameworksconference ICML, conference NeurIPS
DNN pruning is a popular way to reduce the size of a model, improve the inference latency, and minimize the power consumption on DNN accelerators. However, existing approaches might be too complex, expensive or ineffective to apply to a variety of vision/language tasks, DNN architectures and to honor structured pruning constraints. In this paper, we propose an efficient yet effective train-time pruning scheme, Parameter-free Differentiable…