ChainDoRA, a weight-decomposed parameter-efficient fine-tuning framework that builds directional low-rank factors from a connected Tensor-Train chain, reached a seven-task average accuracy of 72.30% o…
Token merging speeds up multilingual speech recognition in the Whisper model family with almost no loss in transcription accuracy across sixteen languages and three model sizes, according to an arXiv …
A technical guide outlines seven engineering techniques for training large language models on limited hardware, including QLoRA, DoRA, and GaLore, which reduce memory usage by quantizing weights or pr…
In a head-to-head test, standard LoRA outperformed Weight-decomposed Low-Rank Adaptation (DoRA) by 2.1% in accuracy while taking 15% less training time per epoch, according to a developer's experiment…
Researchers introduced CORA (Coherent Orthogonal Rotation Adaptation), a parameter-efficient fine-tuning method that uses orthogonal rotations and diagonal scaling to outperform existing approaches li…
Researchers introduced ReCoLoRA, a spectrum-aware recursive consolidation method for continual fine-tuning of large language models, which prevents task interference by re-decomposing the effective we…
Hugging Face's PEFT library challenges the dominance of LoRA in parameter-efficient fine-tuning, revealing that while LoRA accounts for over 98% of PEFT usage on the Hugging Face Hub, alternative meth…
Guide for fine-tuning NVIDIA's Cosmos Predict 2.5 world model using LoRA and DoRA techniques to generate synthetic robot manipulation videos. The approach freezes the base model's 2 billion parameters…