LoRA Fine-Tuning: Match Full Fine-Tuning With Far Less GPU (with Hand Drawn Explanations)!

A practitioner-focused explainer details how LoRA (Low-Rank Adaptation), introduced in the 2021 paper by Hu et al., cuts trainable parameters by 10,000 times and GPU memory requirements by 3 times versus full fine-tuning with Adam on GPT-3 175B, by learning each weight update as the product of two skinny matrices of rank r (typically 8, 16, 32 or 64). The article notes that mixed-precision Adam needs roughly 16 bytes per parameter — about 112 GB for a 7B model before activations — and that serving 50 custom 7B models at 14 GB each would require 700 GB of full copies, while LoRA keeps the base model frozen and swaps in small adapter files. It also covers writing LoRA from scratch in about 20 lines of PyTorch, configuration choices, variants including QLoRA, DoRA, rsLoRA and PiSSA, and how LoRA adapters are served in production.

For practitioners who have hit ‘CUDA out of memory’: why LoRA works, the settings that matter, when to pick DoRA, rsLoRA, or PiSSA instead, and how to serve LoRA in Production? 🎯 The problem, in one picture Imagine you own a 1,000-page encyclopedia. It’s brilliant, but you run a law firm, so you want it to speak legal language. Option A: Reprint the entire encyclopedia with your edits on every page. Costly, slow, & you now store a second 1,000-page book. Want a version for medical use too? Print another one. Option B: Keep the original untouched. Clip a thin, transparent overlay sheet on top that nudges how the pages read. One encyclopedia, many cheap overlays. Option A is full fine-tuning. Option B is LoRA Low-Rank Adaptation . By the end of this blog, you’ll know: Why full fine-tuning hurts What LoRA does under the hood with the math made friendly How to write LoRA from scratch in about 20 lines of PyTorch How to configure it so it matches full fine-tuning Which variants matter QLoRA, DoRA, rsLoRA, PiSSA How LoRA is served in production Where it’s going next 😩 Part 1: Why do we even need LoRA? The pain of full fine-tuning When you fine-tune a model normally, every weight gets updated. For a 7B-parameter model, that means: Weights: all 7 billion numbers sit in GPU memory Gradients: one more number per weight Optimizer states: Adam keeps extra running statistics per weight A common rule of thumb for mixed-precision Adam is roughly 16 bytes per parameter. That’s an estimate, not an exact figure. For 7B parameters it comes to about 112 GB before you count activations. That’s more than one GPU, & far more than a gaming card. The original LoRA paper Hu et al., 2021 opens with this scaling problem. For GPT-3 175B, LoRA can reduce the number of trainable parameters by 10,000 times & the GPU memory requirement by 3 times compared with full fine-tuning using Adam. The storage & deployment pain Say you serve 50 customers & each wants a custom model. With full fine-tuning you save 50 full copies of the model. At 14 GB each 7B in 16-bit , that’s 700 GB. Swapping between them at request time is painful. The “forgetting” pain Full fine-tuning can overwrite what the model already knew. Teach it your legal jargon & it may get worse at basic math. That’s called catastrophic forgetting. Detailed blog link on Catastrophic Forgetting Pointers: what we really wanted Train far fewer parameters Keep the original model frozen safe & reusable Get tiny files we can swap in & out Pay no extra cost at inference LoRA ticks all four boxes. 🧠 Part 2: The core idea The “mixing board” analogy Picture a music mixing board with 4,096 sliders the frequencies . To make a song sound “warmer”, you don’t need to move each slider independently. Pulling a few combined knobs, like “more bass” or “less treble”, gets you there. The change you want is simple, even though the board is huge. Researchers found the same thing in neural networks. The LoRA authors hypothesized that the update matrices in language model adaptation have a low “intrinsic rank”. In plain words, the change you need is much simpler than the model itself. The math, made friendly A layer has a weight matrix W of shape d × k. Full fine-tuning learns a full-size update ΔW, with the same number of entries as W. LoRA says: don’t learn ΔW directly. Learn it as the product of two skinny matrices. Full update huge LoRA update tiny ΔW = d × k → ΔW = B × A d × r r × k r is the "rank", usually 8, 16, 32, 64...B is matrix of dimenstions d x r where d is rowsA is matrix of dimenstions r x k where k is columnsB x A gives back the matrix of d x k r is the rank, the number of independent directions the update can use. The actual number of trainable knobs is r· d + k , which is 65,536 at r=8, not 8 calculation table mentioned below . “r controls how many knobs you get.” The forward pass becomes: h = W·x + alpha / r · B· A·x ↑ ↑ frozen trainable, tinywhere x is the input to the layer list/vector of numbers coming in The paper describes this as a learned low-rank additive update ΔW = BA with r far smaller than min d,k . 🔥 Derivation of the Forward Pass equation Hand Written Notes Let’s count the savings Take one 4096 × 4096 layer: That’s 256× fewer for this layer, & it’s why files shrink from gigabytes to megabytes for the adapter file only. You still need the full base model to run it. The win is that one base model can be shared across many tiny adapters. 256× is per layer, for this square shape. For a whole model the reduction is usually much bigger, because LoRA is typically applied only to some layers often the attention projections and everything else is frozen. The original paper reports about 10,000× fewer trainable parameters for GPT-3 175B. Why B starts at zero A is initialized randomly small values B is initialized to zeros So at step 0, B·A = 0 & the model behaves exactly like the original. Training starts from a safe place, not from a randomly broken model. What is alpha? alpha / r is a scaling factor on the update. Think of it as a volume knob on the overlay sheet. A higher alpha makes the adapter's voice louder. It controls how strongly the adapter’s output gets added to the original output: scale 0: the adapter is ignored, giving the original model scale 1: the correction is added as-is scale 2: the correction is added twice as strongly alpha is generally set at 2r or r, If you always set alpha = 2r, then alpha/r = 2, every time. Zero extra inference cost After training, you can merge the update into the base weights: W new = W + alpha / r · B·A The result is a normal-sized model with no added latency. The paper highlights that, unlike adapter layers, LoRA adds no additional inference latency. This is where most people lose quality. LoRA has few knobs, but each one matters. The 2025 reality check: “LoRA Without Regret” In late 2025, John Schulman & Thinking Machines Lab published a careful study called “LoRA Without Regret”. Their headline, as summarized in the Hugging Face TRL docs: LoRA can match full fine-tuning performance when configured correctly, using only about 67% of the compute. The key findings, in plain pointers: Apply LoRA to all layers. Applying it to all layers, including MLP & MoE layers, is recommended, & attention-only LoRA is consistently outperformed. Use a higher learning rate. The optimal LoRA learning rate was consistently about 10x higher than for full fine-tuning. An independent reproduction saw the same roughly 10x gap. Mind your dataset size. When a dataset exceeds LoRA’s capacity, LoRA underperforms full fine-tuning. Low ranks run out of room first, & higher ranks track full fine-tuning for longer. Watch batch size. LoRA is less tolerant of large batch sizes than full fine-tuning, & raising the rank doesn’t close that gap. For reinforcement learning, tiny is enough. LoRA matched full fine-tuning in RL even at rank 1. One honest caveat: the reproduction found that MLP + attention beat MLP-only. So treat “all layers” as the safe default. Cheat sheet Note the PEFT default: PEFT’s default LoRA adds trainable weights only to the query & value layers of each attention block. Many tutorials inherit this, so override it. The PEFT docs themselves note that adding trainable weights to all the linear layers as QLoRA does can match a fully fine-tuned model. The cheat-sheet rows for rank, dropout & batch size are my practical starting points, not hard rules. Always validate on your own eval set. 🚀 Part 5: Fine-tune a model with LoRA Hugging Face PEFT + TRL If your GPU doesn’t support bf16, set “bf16=False” Merge for Deployment python import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel Recent Transformers versions use dtype= ; older ones expect torch dtype= .base = AutoModelForCausalLM.from pretrained "Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16 model = PeftModel.from pretrained base, "qwen-lora/final" merged = model.merge and unload fold adapter into base weightsmerged.save pretrained "qwen-merged" AutoTokenizer.from pretrained "Qwen/Qwen2.5-0.5B" .save pretrained "qwen-merged" Merge or keep separate? Merge when you ship one fixed model & want maximum speed. Keep separate when you serve many adapters, or want to hot-swap them. 🧊 Part 6: QLoRA, for when the model still doesn’t fit LoRA shrinks what you train. The frozen base model still takes memory. QLoRA also compresses that base. Analogy: LoRA is writing a small note. QLoRA also stores the giant textbook as a compressed PDF while you write the note. QLoRA applies 4-bit quantization to the pretrained weights & trains LoRA modules on top of that quantized representation. It relies on the 4-bit NormalFloat NF4 format, double quantization & paged optimizers to cut memory further. In practice, a 65B fine-tune fits on a single 48 GB GPU, & a 7B one on a gaming card. If the base model fits in bf16 with room for activations, plain LoRA trains faster. If it doesn’t fit, QLoRA makes the run possible at the cost of slower steps. 🧬 Part 7: The LoRA family variants worth knowing Dozens of variants exist. Here are the ones you’ll actually meet: Honest notes on variants DoRA costs more. In PEFT’s own benchmark on Llama-3.1–8B see the PEFT docs , DoRA without caching took 139% more time than LoRA. The docs also recommend merging weights for inference. Gains aren’t guaranteed. One 2026 study on knowledge memory found simply swapping to DoRA or PiSSA does not reliably yield large gains. Results flipped between Llama & Qwen. My advice: get a strong LoRA baseline with the right settings from Part 4 first. Try variants second, & measure on your own task. ⚖️ Part 8: What LoRA can & can’t do Strengths Cheap training Tiny checkpoints Fast switching between tasks Less forgetting usually The “learns less, forgets less” trade-off A well-known 2024 study, “LoRA Learns Less and Forgets Less” Biderman et al. , compared LoRA with full fine-tuning on code & math. In standard low-rank settings, LoRA substantially underperformed full fine-tuning, but it kept the base model’s performance on out-of-domain tasks better. A likely reason: full fine-tuning learns perturbations with a rank 10–100× greater than typical LoRA configurations. Later work reframed this. With the right setup & datasets within capacity, the gap closes Part 4 . A subtle catch: “intruder dimensions” A NeurIPS 2025 paper Shuttleworth et al., “LoRA vs Full Fine-tuning: An Illusion of Equivalence” found that LoRA & full fine-tuning change weights differently. LoRA creates new, high-ranking singular vectors called “intruder dimensions”, & much of LoRA’s forgetting is tied to them. The same study notes that with large learning rates, LoRA models can forget more than full fine-tuning, so less forgetting is not a guarantee. Takeaway: don’t push the learning rate to the moon just because LoRA tolerates it. When to prefer full fine-tuning You’re doing huge pre-training-scale training, where LoRA starts falling behind You have a massive dataset & a big compute budget You’ve tested LoRA & it plateaus well below full fine-tuning 🏭 Part 10: The best ways LoRA is used 1. Instruction & domain tuning Take an open model & teach it your tone, your format, or your domain. This is the bread & butter use. 2. Reinforcement learning post-training RL gives little information per episode, so tiny adapters have enough capacity. One independent reproduction trained Qwen2.5–0.5B on GSM8k with GRPO. The rank-1 LoRA beat both full fine-tuning configurations. 3. Multi-tenant serving: one base model, many adapters This is where LoRA’s economics shine. Analogy: one coffee machine, many flavor pods. The expensive machine base model loads once. Each customer’s tiny pod adapter swaps in per request. The base model stays loaded once, & customer-specific adapters swap in at request time. Research systems like S-LoRA keep adapters in host memory & move the ones needed by running requests onto the GPU. The vLLM project lists efficient multi-LoRA support for dense & MoE layers as a built-in feature. vllm serve meta-llama/Llama-3.1-8B-Instruct \ --enable-lora \ --max-loras 8 \ --max-cpu-loras 64 \ --max-lora-rank 64 \ --lora-modules customer-a=/adapters/customer-a customer-b=/adapters/customer-b Llama repos are gated, so you’ll need access approval & a Hugging Face login. Any ungated base model works the same way. What the flags mean: max loras is how many adapters can be active together in a batch the GPU working set max cpu loras is how many stay warm in host memory before eviction max lora rank is the largest rank the server accepts Clients pick an adapter through the model field: python from openai import OpenAIclient = OpenAI base url="<http://localhost:8000/v1 ", api key="none" resp = client.chat.completions.create model="customer-a", selects that customer's adapter messages= {"role": "user", "content": "Summarize this contract clause..."} , print resp.choices 0 .message.content Or use the Python API: python from vllm import LLM, SamplingParamsfrom vllm.lora.request import LoRARequestllm = LLM model="meta-llama/Llama-3.1-8B-Instruct", enable lora=True, max loras=4, max lora rank=64 out = llm.generate "Explain force majeure in simple words." , SamplingParams max tokens=200 , lora request=LoRARequest "customer a", 1, "/adapters/customer-a" , print out 0 .outputs 0 .text Reality check: adapters aren’t free at inference when unmerged. One published vLLM benchmark found that applying LoRA to all linear layers cut throughput by 23.9%–47.0%, depending on rank. Your numbers will vary with hardware & batch mix, so measure your own setup. For a single fixed model, merging is still faster. 4. Image models LoRA isn’t only for text. It’s heavily used to teach diffusion models a new style or character with a small file. PEFT even ships LoRA support for convolution layers, & it supports Conv2d layers with groups = 1. 5. Adapter libraries Teams keep one adapter per task support, legal, code review & route requests to them, instead of maintaining many full models. 🧯 Part 11: Common mistakes & a debugging checklist ❌ Using the default q proj/v proj only. Switch to all-linear. ❌ Reusing your full fine-tuning learning rate. It’s probably ~10× too low for LoRA. ❌ Cranking rank to fix everything. Higher rank adds capacity for big datasets, but it won’t cure a bad learning rate or an oversized batch Part 4 . ❌ Huge batch sizes. LoRA is touchier about them, & more rank doesn’t help. ❌ Merging into a quantized base without care. Merge in higher precision, then re-quantize, & test. ❌ Judging by loss alone. Evaluate on real tasks, including something outside your domain to catch forgetting. ❌ Skipping version pinning. PEFT moves fast, so pin your library versions. Quick debugging flow: Run model.print trainable parameters . Is it a sensible fraction, not 0 & not 100%? Does loss decrease in the first 50 steps? If not, check the learning rate & target modules. Loss unstable? Lower the LR, lower alpha, or try rsLoRA at high ranks. Good loss but bad outputs? Check your chat template & data formatting before blaming LoRA. 🔮 Part 12: The future of LoRA What’s happening now The variant explosion continues. Recent PEFT releases added a batch of new methods, including HiRA, GLoRA, VeLoRA, & Uni-LoRA, plus automatic LoRA target selection. There’s also a Riemannian-preconditioned LoRA optimizer meant to avoid tuning separate learning rates for A & B. Check the PEFT release notes for the current list. Mixture-of-Experts support is maturing. PEFT’s target parameters feature lets LoRA target nn.Parameters directly, useful for MoE layers. Meanwhile vLLM added multi-LoRA serving for MoE families including GPT-OSS, Qwen3-MoE, DeepSeek, & Llama MoE. Memory-focused methods are appearing. VeLoRA targets activation memory rather than parameter count. Understanding is improving. Papers on capacity, intruder dimensions, & forgetting are replacing “just try rank 8” with real guidance. My predictions opinion, not fact LoRA becomes the default for post-training & personalization. The “low-regret” result makes it easy to justify. Adapters become products. Expect marketplaces & registries of task adapters, much like packages today. Better routing & composition. Systems will pick or blend adapters on the fly, which makes continual learning less risky. Smarter rank allocation. Instead of one rank everywhere, methods will assign more capacity where it’s needed. The gap with full fine-tuning keeps shrinking for small & medium datasets. Pre-training-scale data will likely stay full-weight territory. ⭐️ Final takeaway Why LoRA exists: full fine-tuning is heavy on memory, storage & deployment, & it can cause forgetting. The idea: freeze W, learn the update as B·A with a small rank r. Because B starts at zero, you begin exactly at the base model. The payoff: often hundreds to thousands of times fewer trainable parameters, depending on model size, rank & targeted layers. You also get tiny files & no inference cost once merged. Get the settings right: target all linear layers, use a learning rate around 10× higher than full fine-tuning, watch batch size, & match rank to dataset size. Pick tools by constraint: plain LoRA when the model fits, QLoRA when it doesn’t. Try DoRA, rsLoRA & PiSSA only after a strong baseline, & measure the gains yourself. Know the limits: LoRA can lag on huge datasets, & high learning rates can still cause forgetting. Best uses: domain & instruction tuning, RL post-training even rank 1 , & multi-adapter serving that lets one GPU act like many models. Where it’s heading: more variants, better MoE support, smarter serving, & adapters as everyday building blocks. If you remember one sentence, make it this: don’t rebuild the encyclopedia, clip on a smart overlay. If this helped, a clap or a comment with your own LoRA settings would make my day. Happy fine-tuning ✨ Where to go from here You have come this far, which really means a lot. The further steps are as follows: 🎊 Appreciate yourself for deciding to read this blog, which has exponentially enhanced your knowledge. 🎉 Follow me on Medium It’s free & opt in for email notifications to stay updated with my latest articles on multi-tech Complete AIOPS . 🎯 Follow me on LinkedIn It’s free to get the latest tech dose & tips & tricks.