Hierarchical Low-Rank Reparameterization for Parameter-Efficient

A new arXiv paper (2610.37068v1) proposes hierarchical low-rank reparameterization, a method that reorganizes the trainable parameters of parameter-efficient fine-tuning (PEFT) techniques into an approximately square matrix and expresses it via a recursively applied low-rank decomposition, cutting trainable parameter counts by roughly an order of magnitude. Combined with LoRA, DoRA, MoSA and RaSA, the method matched or beat the corresponding baselines on eight commonsense reasoning benchmarks and GSM8K math reasoning using LLaMA-3.1-8B and Qwen2.5-7B, and under comparable trainable-parameter budgets it outperformed LoRA and DoRA alone.

arXiv:2610.37068v1 Announce Type: new Abstract: Parameter-efficient fine-tuning PEFT methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while maintaining strong task performance. To this end, we propose a novel hierarchical low-rank reparameterization method. Its core idea is to reorganize the trainable parameters of PEFT methods into an approximately square matrix and then express it via a low-rank decomposition, which can be applied recursively. This forms a multi-level structure with factors compressed level by level, drastically reducing the number of trainable parameters. In our experiments, we combine this reparameterization with LoRA, DoRA, MoSA, and RaSA. We evaluate commonsense reasoning on LLaMA-3.1-8B and Qwen2.5-7B, with additional GSM8K math reasoning experiments on LLaMA-3.1-8B. Under comparable trainable-parameter budgets, our method combined with LoRA or DoRA achieves better performance. Furthermore,when the parameter count of the original methods is compressed by approximately an order of magnitude, our method can still maintain competitive task performance and even surpass the corresponding baselines in average accuracy across the eight commonsense reasoning benchmarks.