LoRA: Low-Rank Adaptation of Large Language Models (2021)

Microsoft researchers Edward J. Hu and colleagues proposed Low-Rank Adaptation (LoRA) in a paper submitted to arXiv on 17 June 2021, a method that freezes pre-trained model weights and injects trainable rank decomposition matrices into each Transformer layer. Compared with GPT-3 175B fine-tuned with Adam, LoRA reduces trainable parameters by 10,000 times and GPU memory requirements by 3 times, while matching or exceeding full fine-tuning quality on RoBERTa, DeBERTa, GPT-2 and GPT-3 with no additional inference latency. The team released a PyTorch integration package plus implementations and model checkpoints for RoBERTa, DeBERTa and GPT-2 on GitHub.

Computer Science Computation and Language Submitted on 17 Jun 2021 v1 https://arxiv.org/abs/2106.09685v1 , last revised 16 Oct 2021 this version, v2 Title:LoRA: Low-Rank Adaptation of Large Language Models View PDF https://arxiv.org/pdf/2106.09685 HTML experimental https://arxiv.org/html/2106.09685v2 Abstract:An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at this https URL https://github.com/microsoft/LoRA . Submission history From: Edward J. Hu view email https://arxiv.org/show-email/e4479443/2106.09685 Thu, 17 Jun 2021 17:37:18 UTC 1,791 KB \ v1\ https://arxiv.org/abs/2106.09685v1 v2 Sat, 16 Oct 2021 18:40:34 UTC 896 KB Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .