Fine-Tuning of Transformer models with Frames

Researchers propose FrameFT, a parameter-efficient fine-tuning method that models parameter updates with a sparse coefficient matrix in a Fusion Frame basis, reducing memory footprint compared to LoRA. In experiments on language and vision benchmarks, FrameFT matches or exceeds state-of-the-art PEFT techniques while requiring far fewer trainable parameters.

arXiv:2608.26430v1 Announce Type: new Abstract: Parameter-Efficient Fine-Tuning PEFT strategies such as Low-Rank Adaptation LoRA are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O} dr $, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.