Parameter-Efficient Fine-Tuning with Learnable Rank

Researchers introduced Learnable Rank LoRA (LR-LoRA), a parameter-efficient fine-tuning method that allows the adapter rank to be learned during training rather than fixed, enabling the optimizer to determine the appropriate rank for each layer. The approach revealed substantial layer-wise variation in learned ranks, with attention and MLP layers in transformer models showing systematically different rank preferences. LR-LoRA achieved state-of-the-art performance across language understanding and commonsense reasoning benchmarks, outperforming strong PEFT baselines by providing a more flexible inductive bias than fixed-rank adaptations.

arXiv:2606.04325v1 Announce Type: new Abstract: Low-Rank Adaptation LoRA is a popular parameter-efficient fine-tuning PEFT method that restricts weight updates to low-rank adapters, introducing a fixed low-rank inductive bias by optimizing in a low-dimensional subspace. In this work, we question whether a fixed-rank constraint is the most effective inductive bias for parameter-efficient fine-tuning. We introduce Learnable Rank LoRA LR-LoRA , a PEFT method in which the adapter rank is learned during the training process. Instead of prescribing a uniform rank for all adapter layers, LR-LoRA allows the optimizer to determine the appropriate rank for each layer. Using this approach, we find substantial layer-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences. Across a range of language understanding and commonsense reasoning benchmarks, LR-LoRA achieves state-of-the-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed-rank adaptations.