Layer-wise Curriculum Learning for Efficient LLM Compression

A new arXiv paper (arXiv:2609.19213v1) introduces layer-wise curriculum learning for efficient LLM compression, a method that partitions a model into multi-layer segments to transfer knowledge from teacher to student model progressively from easier to harder optimization tasks. The authors report that experiments on BERT and GPT-2 cut GPU memory usage and training hours by more than 50% while achieving state-of-the-art compression performance, and that the method outperforms other pruning methods on LLaMA-family and Qwen models under equal training hours with a lower GPU memory footprint. The paper also presents a feature caching method with a multi-threading strategy to address feature misalignment across layers and maximize GPU utilization.

arXiv:2609.19213v1 Announce Type: new Abstract: In this paper, we introduce layer-wise curriculum learning for efficient LLM compression. The proposed method facilitates the knowledge transfer from the teacher model to the student model, utilizing a curriculum learning approach that begins with easier optimization tasks and progressively tackles harder ones. In order to adopt the layer-wise learning in LLM compression, we partition the whole model into multiple segments consisting of layers, thereby enabling more computationally efficient knowledge transfer for LLMs. Based on our theoretical analysis of cumulative error phenomenon, layer-wise curriculum learning accelerates convergence while stabilizing the knowledge transfer process. In addition, we present a feature caching method with a multi-threading strategy to efficiently address feature misalignment across layers, maximizing GPU utilization. Consequently, our method exhibits advanced model compression performance, as well as high computational efficiency in terms of minimized memory usage and short training hours. Experiments on multiple datasets show that the proposed method achieves state-of-the-art performance while reducing GPU memory usage and training hours by more than 50\% on BERT and GPT-2. Moreover, it outperforms the other pruning methods on LLaMA-family and Qwen models under the same training hours, with a lower GPU memory footprint.