Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

Researchers proposed a structured pruning method for large language models that combines power transformation, sign-preserving score aggregation, and adaptive feature retention. Tested on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B, the method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup.

arXiv:2607.08027v1 Announce Type: new Abstract: This paper proposes an improved structured pruning method for large language models LLMs that addresses key challenges in adapting Adaptive Feature Retention AFR , an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.