{"slug": "lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn", "title": "LoRA Fine-Tuning: Match Full Fine-Tuning With Far Less GPU (with Hand Drawn Explanations)!", "summary": "A practitioner-focused explainer details how LoRA (Low-Rank Adaptation), introduced in the 2021 paper by Hu et al., cuts trainable parameters by 10,000 times and GPU memory requirements by 3 times versus full fine-tuning with Adam on GPT-3 175B, by learning each weight update as the product of two skinny matrices of rank r (typically 8, 16, 32 or 64). The article notes that mixed-precision Adam needs roughly 16 bytes per parameter — about 112 GB for a 7B model before activations — and that serving 50 custom 7B models at 14 GB each would require 700 GB of full copies, while LoRA keeps the base model frozen and swaps in small adapter files. It also covers writing LoRA from scratch in about 20 lines of PyTorch, configuration choices, variants including QLoRA, DoRA, rsLoRA and PiSSA, and how LoRA adapters are served in production.", "body_md": "For practitioners who have hit ‘CUDA out of memory’: why LoRA works, the settings that matter, when to pick DoRA, rsLoRA, or PiSSA instead, and how to serve LoRA in Production?\n\n🎯 The problem, in one picture\n\nImagine you own a 1,000-page encyclopedia. It’s brilliant, but you run a law firm, so you want it to speak legal language.\n\nOption A: Reprint the entire encyclopedia with your edits on every page. Costly, slow, & you now store a second 1,000-page book. Want a version for medical use too? Print another one.\n\nOption B: Keep the original untouched. Clip a thin, transparent overlay sheet on top that nudges how the pages read. One encyclopedia, many cheap overlays.\n\nOption A is full fine-tuning. Option B is LoRA (Low-Rank Adaptation).\n\nBy the end of this blog, you’ll know:\n\nWhy full fine-tuning hurts\n\nWhat LoRA does under the hood (with the math made friendly)\n\nHow to write LoRA from scratch in about 20 lines of PyTorch\n\nHow to configure it so it matches full fine-tuning\n\nWhich variants matter (QLoRA, DoRA, rsLoRA, PiSSA)\n\nHow LoRA is served in production\n\nWhere it’s going next\n\n😩 Part 1: Why do we even need LoRA?\n\nThe pain of full fine-tuning\n\nWhen you fine-tune a model normally, every weight gets updated. For a 7B-parameter model, that means:\n\nWeights: all 7 billion numbers sit in GPU memory\n\nGradients: one more number per weight\n\nOptimizer states: Adam keeps extra running statistics per weight\n\nA common rule of thumb for mixed-precision Adam is roughly 16 bytes per parameter. That’s an estimate, not an exact figure. For 7B parameters it comes to about 112 GB before you count activations. That’s more than one GPU, & far more than a gaming card.\n\nThe original LoRA paper (Hu et al., 2021) opens with this scaling problem. For GPT-3 175B, LoRA can reduce the number of trainable parameters by 10,000 times & the GPU memory requirement by 3 times compared with full fine-tuning using Adam.\n\nThe storage & deployment pain\n\nSay you serve 50 customers & each wants a custom model. With full fine-tuning you save 50 full copies of the model. At 14 GB each (7B in 16-bit), that’s 700 GB. Swapping between them at request time is painful.\n\nThe “forgetting” pain\n\nFull fine-tuning can overwrite what the model already knew. Teach it your legal jargon & it may get worse at basic math. That’s called catastrophic forgetting. (Detailed blog link on Catastrophic Forgetting)\n\nPointers: what we really wanted\n\nTrain far fewer parameters\n\nKeep the original model frozen (safe & reusable)\n\nGet tiny files we can swap in & out\n\nPay no extra cost at inference\n\nLoRA ticks all four boxes.\n\n🧠 Part 2: The core idea!\n\nThe “mixing board” analogy\n\nPicture a music mixing board with 4,096 sliders (the frequencies). To make a song sound “warmer”, you don’t need to move each slider independently. Pulling a few combined knobs, like “more bass” or “less treble”, gets you there.\n\nThe change you want is simple, even though the board is huge. Researchers found the same thing in neural networks. The LoRA authors hypothesized that the update matrices in language model adaptation have a low “intrinsic rank”. In plain words, the change you need is much simpler than the model itself.\n\nThe math, made friendly\n\nA layer has a weight matrix W of shape d × k. Full fine-tuning learns a full-size update ΔW, with the same number of entries as W.\n\nLoRA says: don’t learn ΔW directly. Learn it as the product of two skinny matrices.\n\n```\nFull update (huge)         LoRA update (tiny)ΔW  =  [ d × k ]     →     ΔW  =  B  ×  A                                 [d × r] [r × k]r is the \"rank\", usually 8, 16, 32, 64...B is matrix of dimenstions [d x r] where d is rowsA is matrix of dimenstions [r x k] where k is columnsB x A gives back the matrix of [d x k]\n```\n\nr is the rank, the number of independent directions the update can use. The actual number of trainable knobs is r·(d + k), which is 65,536 at r=8, not 8 (calculation table mentioned below). “r controls how many knobs you get.”\n\nThe forward pass becomes:\n\n```\nh = W·x  +  (alpha / r) · B·(A·x)     ↑            ↑  frozen     trainable, tinywhere x is the input to the layer (list/vector of numbers coming in)\n```\n\nThe paper describes this as a learned low-rank additive update ΔW = BA with r far smaller than min(d,k).\n\n🔥 Derivation of the Forward Pass equation (Hand Written Notes)!\n\nLet’s count the savings\n\nTake one 4096 × 4096 layer:\n\nThat’s 256× fewer for this layer, & it’s why files shrink from gigabytes to megabytes for the adapter file only. You still need the full base model to run it. The win is that one base model can be shared across many tiny adapters.\n\n256× is per layer, for this square shape. For a whole model the reduction is usually much bigger, because LoRA is typically applied only to some layers (often the attention projections) and everything else is frozen. The original paper reports about 10,000× fewer trainable parameters for GPT-3 175B.\n\nWhy B starts at zero\n\nA is initialized randomly (small values)\n\nB is initialized to zeros\n\nSo at step 0, B·A = 0 & the model behaves exactly like the original. Training starts from a safe place, not from a randomly broken model.\n\nWhat is alpha?\n\nalpha / r is a scaling factor on the update. Think of it as a volume knob on the overlay sheet. A higher alpha makes the adapter's voice louder. It controls how strongly the adapter’s output gets added to the original output:\n\nscale 0: the adapter is ignored, giving the original model\n\nscale 1: the correction is added as-is\n\nscale 2: the correction is added twice as strongly\n\nalpha is generally set at 2r or r, If you always set alpha = 2r, then alpha/r = 2, every time.\n\nZero extra inference cost\n\nAfter training, you can merge the update into the base weights:\n\n```\nW_new = W + (alpha / r) · B·A\n```\n\nThe result is a normal-sized model with no added latency. The paper highlights that, unlike adapter layers, LoRA adds no additional inference latency.\n\nThis is where most people lose quality. LoRA has few knobs, but each one matters.\n\nThe 2025 reality check: “LoRA Without Regret”\n\nIn late 2025, John Schulman & Thinking Machines Lab published a careful study called “LoRA Without Regret”. Their headline, as summarized in the Hugging Face TRL docs: LoRA can match full fine-tuning performance when configured correctly, using only about 67% of the compute.\n\nThe key findings, in plain pointers:\n\nApply LoRA to all layers. Applying it to all layers, including MLP & MoE layers, is recommended, & attention-only LoRA is consistently outperformed.\n\nUse a higher learning rate. The optimal LoRA learning rate was consistently about 10x higher than for full fine-tuning. An independent reproduction saw the same roughly 10x gap.\n\nMind your dataset size. When a dataset exceeds LoRA’s capacity, LoRA underperforms full fine-tuning. Low ranks run out of room first, & higher ranks track full fine-tuning for longer.\n\nWatch batch size. LoRA is less tolerant of large batch sizes than full fine-tuning, & raising the rank doesn’t close that gap.\n\nFor reinforcement learning, tiny is enough. LoRA matched full fine-tuning in RL even at rank 1.\n\nOne honest caveat: the reproduction found that MLP + attention beat MLP-only. So treat “all layers” as the safe default.\n\nCheat sheet\n\nNote the PEFT default: PEFT’s default LoRA adds trainable weights only to the query & value layers of each attention block. Many tutorials inherit this, so override it. The PEFT docs themselves note that adding trainable weights to all the linear layers (as QLoRA does) can match a fully fine-tuned model.\n\nThe cheat-sheet rows for rank, dropout & batch size are my practical starting points, not hard rules. Always validate on your own eval set.\n\n🚀 Part 5: Fine-tune a model with LoRA (Hugging Face PEFT + TRL)\n\nIf your GPU doesn’t support bf16, set “bf16=False”\n\nMerge for Deployment\n\n``` python\nimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel# Recent Transformers versions use `dtype=`; older ones expect `torch_dtype=`.base = AutoModelForCausalLM.from_pretrained(\"Qwen/Qwen2.5-0.5B\", dtype=torch.bfloat16)model = PeftModel.from_pretrained(base, \"qwen-lora/final\")merged = model.merge_and_unload()        # fold adapter into base weightsmerged.save_pretrained(\"qwen-merged\")AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-0.5B\").save_pretrained(\"qwen-merged\")\n```\n\nMerge or keep separate?\n\nMerge when you ship one fixed model & want maximum speed.\n\nKeep separate when you serve many adapters, or want to hot-swap them.\n\n🧊 Part 6: QLoRA, for when the model still doesn’t fit\n\nLoRA shrinks what you train. The frozen base model still takes memory. QLoRA also compresses that base.\n\nAnalogy: LoRA is writing a small note. QLoRA also stores the giant textbook as a compressed PDF while you write the note.\n\nQLoRA applies 4-bit quantization to the pretrained weights & trains LoRA modules on top of that quantized representation. It relies on the 4-bit NormalFloat (NF4) format, double quantization & paged optimizers to cut memory further. In practice, a 65B fine-tune fits on a single 48 GB GPU, & a 7B one on a gaming card.\n\nIf the base model fits in bf16 with room for activations, plain LoRA trains faster.\n\nIf it doesn’t fit, QLoRA makes the run possible at the cost of slower steps.\n\n🧬 Part 7: The LoRA family (variants worth knowing)\n\nDozens of variants exist. Here are the ones you’ll actually meet:\n\nHonest notes on variants\n\nDoRA costs more. In PEFT’s own benchmark on Llama-3.1–8B (see the PEFT docs), DoRA without caching took 139% more time than LoRA. The docs also recommend merging weights for inference.\n\nGains aren’t guaranteed. One 2026 study on knowledge memory found simply swapping to DoRA or PiSSA does not reliably yield large gains. Results flipped between Llama & Qwen.\n\nMy advice: get a strong LoRA baseline with the right settings from Part 4 first. Try variants second, & measure on your own task.\n\n⚖️ Part 8: What LoRA can & can’t do\n\nStrengths\n\nCheap training\n\nTiny checkpoints\n\nFast switching between tasks\n\nLess forgetting (usually)\n\nThe “learns less, forgets less” trade-off\n\nA well-known 2024 study, “LoRA Learns Less and Forgets Less” (Biderman et al.), compared LoRA with full fine-tuning on code & math. In standard low-rank settings, LoRA substantially underperformed full fine-tuning, but it kept the base model’s performance on out-of-domain tasks better. A likely reason: full fine-tuning learns perturbations with a rank 10–100× greater than typical LoRA configurations.\n\nLater work reframed this. With the right setup & datasets within capacity, the gap closes (Part 4).\n\nA subtle catch: “intruder dimensions”\n\nA NeurIPS 2025 paper (Shuttleworth et al., “LoRA vs Full Fine-tuning: An Illusion of Equivalence”) found that LoRA & full fine-tuning change weights differently. LoRA creates new, high-ranking singular vectors called “intruder dimensions”, & much of LoRA’s forgetting is tied to them. The same study notes that with large learning rates, LoRA models can forget more than full fine-tuning, so less forgetting is not a guarantee.\n\nTakeaway: don’t push the learning rate to the moon just because LoRA tolerates it.\n\nWhen to prefer full fine-tuning\n\nYou’re doing huge pre-training-scale training, where LoRA starts falling behind\n\nYou have a massive dataset & a big compute budget\n\nYou’ve tested LoRA & it plateaus well below full fine-tuning\n\n🏭 Part 10: The best ways LoRA is used\n\n1. Instruction & domain tuning\n\nTake an open model & teach it your tone, your format, or your domain. This is the bread & butter use.\n\n2. Reinforcement learning post-training\n\nRL gives little information per episode, so tiny adapters have enough capacity. One independent reproduction trained Qwen2.5–0.5B on GSM8k with GRPO. The rank-1 LoRA beat both full fine-tuning configurations.\n\n3. Multi-tenant serving: one base model, many adapters\n\nThis is where LoRA’s economics shine. Analogy: one coffee machine, many flavor pods. The expensive machine (base model) loads once. Each customer’s tiny pod (adapter) swaps in per request.\n\nThe base model stays loaded once, & customer-specific adapters swap in at request time.\n\nResearch systems like S-LoRA keep adapters in host memory & move the ones needed by running requests onto the GPU.\n\nThe vLLM project lists efficient multi-LoRA support for dense & MoE layers as a built-in feature.\n\n```\nvllm serve meta-llama/Llama-3.1-8B-Instruct \\  --enable-lora \\  --max-loras 8 \\  --max-cpu-loras 64 \\  --max-lora-rank 64 \\  --lora-modules customer-a=/adapters/customer-a customer-b=/adapters/customer-b\n```\n\n(Llama repos are gated, so you’ll need access approval & a Hugging Face login. Any ungated base model works the same way.)\n\nWhat the flags mean:\n\nmax_loras is how many adapters can be active together in a batch (the GPU working set)\n\nmax_cpu_loras is how many stay warm in host memory before eviction\n\nmax_lora_rank is the largest rank the server accepts\n\nClients pick an adapter through the model field:\n\n``` python\nfrom openai import OpenAIclient = OpenAI(base_url=\"<http://localhost:8000/v1>\", api_key=\"none\")resp = client.chat.completions.create(    model=\"customer-a\",      # selects that customer's adapter    messages=[{\"role\": \"user\", \"content\": \"Summarize this contract clause...\"}],)print(resp.choices[0].message.content)\n```\n\nOr use the Python API:\n\n``` python\nfrom vllm import LLM, SamplingParamsfrom vllm.lora.request import LoRARequestllm = LLM(model=\"meta-llama/Llama-3.1-8B-Instruct\",          enable_lora=True, max_loras=4, max_lora_rank=64)out = llm.generate(    [\"Explain force majeure in simple words.\"],    SamplingParams(max_tokens=200),    lora_request=LoRARequest(\"customer_a\", 1, \"/adapters/customer-a\"),)print(out[0].outputs[0].text)\n```\n\nReality check: adapters aren’t free at inference when unmerged. One published vLLM benchmark found that applying LoRA to all linear layers cut throughput by 23.9%–47.0%, depending on rank. Your numbers will vary with hardware & batch mix, so measure your own setup. For a single fixed model, merging is still faster.\n\n4. Image models\n\nLoRA isn’t only for text. It’s heavily used to teach diffusion models a new style or character with a small file. PEFT even ships LoRA support for convolution layers, & it supports Conv2d layers with groups != 1.\n\n5. Adapter libraries\n\nTeams keep one adapter per task (support, legal, code review) & route requests to them, instead of maintaining many full models.\n\n🧯 Part 11: Common mistakes & a debugging checklist\n\n❌ Using the default q_proj/v_proj only. Switch to all-linear.\n\n❌ Reusing your full fine-tuning learning rate. It’s probably ~10× too low for LoRA.\n\n❌ Cranking rank to fix everything. Higher rank adds capacity for big datasets, but it won’t cure a bad learning rate or an oversized batch (Part 4).\n\n❌ Huge batch sizes. LoRA is touchier about them, & more rank doesn’t help.\n\n❌ Merging into a quantized base without care. Merge in higher precision, then re-quantize, & test.\n\n❌ Judging by loss alone. Evaluate on real tasks, including something outside your domain to catch forgetting.\n\n❌ Skipping version pinning. PEFT moves fast, so pin your library versions.\n\nQuick debugging flow:\n\nRun model.print_trainable_parameters(). Is it a sensible fraction, not 0 & not 100%?\n\nDoes loss decrease in the first 50 steps? If not, check the learning rate & target modules.\n\nLoss unstable? Lower the LR, lower alpha, or try rsLoRA at high ranks.\n\nGood loss but bad outputs? Check your chat template & data formatting before blaming LoRA.\n\n🔮 Part 12: The future of LoRA\n\nWhat’s happening now\n\nThe variant explosion continues. Recent PEFT releases added a batch of new methods, including HiRA, GLoRA, VeLoRA, & Uni-LoRA, plus automatic LoRA target selection. There’s also a Riemannian-preconditioned LoRA optimizer meant to avoid tuning separate learning rates for A & B. Check the PEFT release notes for the current list.\n\nMixture-of-Experts support is maturing. PEFT’s target_parameters feature lets LoRA target nn.Parameters directly, useful for MoE layers. Meanwhile vLLM added multi-LoRA serving for MoE families including GPT-OSS, Qwen3-MoE, DeepSeek, & Llama MoE.\n\nMemory-focused methods are appearing. VeLoRA targets activation memory rather than parameter count.\n\nUnderstanding is improving. Papers on capacity, intruder dimensions, & forgetting are replacing “just try rank 8” with real guidance.\n\nMy predictions (opinion, not fact)\n\nLoRA becomes the default for post-training & personalization. The “low-regret” result makes it easy to justify.\n\nAdapters become products. Expect marketplaces & registries of task adapters, much like packages today.\n\nBetter routing & composition. Systems will pick or blend adapters on the fly, which makes continual learning less risky.\n\nSmarter rank allocation. Instead of one rank everywhere, methods will assign more capacity where it’s needed.\n\nThe gap with full fine-tuning keeps shrinking for small & medium datasets. Pre-training-scale data will likely stay full-weight territory.\n\n⭐️ Final takeaway\n\nWhy LoRA exists: full fine-tuning is heavy on memory, storage & deployment, & it can cause forgetting.\n\nThe idea: freeze W, learn the update as B·A with a small rank r. Because B starts at zero, you begin exactly at the base model.\n\nThe payoff: often hundreds to thousands of times fewer trainable parameters, depending on model size, rank & targeted layers. You also get tiny files & no inference cost once merged.\n\nGet the settings right: target all linear layers, use a learning rate around 10× higher than full fine-tuning, watch batch size, & match rank to dataset size.\n\nPick tools by constraint: plain LoRA when the model fits, QLoRA when it doesn’t. Try DoRA, rsLoRA & PiSSA only after a strong baseline, & measure the gains yourself.\n\nKnow the limits: LoRA can lag on huge datasets, & high learning rates can still cause forgetting.\n\nBest uses: domain & instruction tuning, RL post-training (even rank 1), & multi-adapter serving that lets one GPU act like many models.\n\nWhere it’s heading: more variants, better MoE support, smarter serving, & adapters as everyday building blocks.\n\nIf you remember one sentence, make it this: don’t rebuild the encyclopedia, clip on a smart overlay.\n\nIf this helped, a clap or a comment with your own LoRA settings would make my day. Happy fine-tuning!\n\n✨ Where to go from here!\n\nYou have come this far, which really means a lot. The further steps are as follows:\n\n🎊 Appreciate yourself for deciding to read this blog, which has exponentially enhanced your knowledge.\n\n🎉 Follow me on Medium (It’s free) & opt in for email notifications to stay updated with my latest articles on multi-tech (Complete AIOPS).\n\n🎯 Follow me on LinkedIn (It’s free) to get the latest tech dose & tips & tricks.", "url": "https://wpnews.pro/news/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn", "canonical_source": "https://pub.towardsai.net/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-f07a336c67f7?source=rss----98111c9905da---4", "published_at": "2026-11-15 09:30:05+00:00", "updated_at": "2026-11-15 09:51:42.013377+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure", "ai-tools"], "entities": ["LoRA", "Hu et al.", "QLoRA", "DoRA", "rsLoRA", "PiSSA", "PyTorch", "GPT-3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn", "markdown": "https://wpnews.pro/news/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn.md", "text": "https://wpnews.pro/news/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn.txt", "jsonld": "https://wpnews.pro/news/lora-fine-tuning-match-full-fine-tuning-with-far-less-gpu-with-hand-drawn.jsonld"}}