Weight-decomposed Low-Rank Adaptation (DoRA) is marketed as a way to bridge the gap between LoRA and full fine-tuning, but in my recent head-to-head test, it actually lagged behind standard LoRA in both training speed and final accuracy.
I kept the hyperparameters identical for both runs to ensure a fair comparison:
The issue became apparent during the first two epochs. DoRA introduces an additional magnitude component that requires more computation per update. While the overhead seems negligible on paper, it added a noticeable drag to my iterations.
For this specific project, the added complexity of DoRA wasn't worth the trade-off. It seems the "better learning capacity" of weight decomposition doesn't always trigger on smaller, niche datasets where standard LoRA is already highly efficient. If you're constrained by hardware, sticking to a basic LoRA AI workflow is likely the safer bet for faster deployment. I ran these experiments on a consumer-grade GPU to see if the theoretical benefits of decomposing magnitude and direction actually translate to real-world performance on smaller datasets.
The Setup #
I kept the hyperparameters identical for both runs to ensure a fair comparison:
Rank (r): 8Alpha: 16Learning Rate: 2e-4Epochs: 3
The Diagnosis #
The issue became apparent during the first two epochs. DoRA introduces an additional magnitude component that requires more computation per update. While the overhead seems negligible on paper, it added a noticeable drag to my iterations.
More frustrating was the convergence. I noticed the loss curve for DoRA was significantly more volatile than LoRA's. Instead of the smoother descent I expected, DoRA struggled to stabilize, leading to a lower validation score.
Training Time: DoRA took ~15% longer per epoch.Accuracy: LoRA outperformed DoRA by 2.1% on my specific evaluation set.VRAM Usage: Roughly the same, though DoRA's overhead slightly increased peak memory.
Conclusion #
For this specific project, the added complexity of DoRA wasn't worth the trade-off. It seems the "better learning capacity" of weight decomposition doesn't always trigger on smaller, niche datasets where standard LoRA is already highly efficient. If you're constrained by hardware, sticking to a basic LoRA AI workflow is likely the safer bet for faster deployment.
[Next Train-Test Split: A Practical Guide for Data Science →](/en/threads/3867/)
All Replies （3） #
J
I've noticed DoRA takes way longer to converge on smaller datasets compared to standard LoRA.
0
L
P
Did you try adjusting the learning rate or sticking with the default settings for both?
0