For practitioners who have hit ‘CUDA out of memory’: why LoRA works, the settings that matter, when to pick DoRA, rsLoRA, or PiSSA instead, and how to serve LoRA in Production?
🎯 The problem, in one picture
Imagine you own a 1,000-page encyclopedia. It’s brilliant, but you run a law firm, so you want it to speak legal language.
Option A: Reprint the entire encyclopedia with your edits on every page. Costly, slow, & you now store a second 1,000-page book. Want a version for medical use too? Print another one.
Option B: Keep the original untouched. Clip a thin, transparent overlay sheet on top that nudges how the pages read. One encyclopedia, many cheap overlays.
Option A is full fine-tuning. Option B is LoRA (Low-Rank Adaptation).
By the end of this blog, you’ll know:
Why full fine-tuning hurts
What LoRA does under the hood (with the math made friendly)
How to write LoRA from scratch in about 20 lines of PyTorch
How to configure it so it matches full fine-tuning
Which variants matter (QLoRA, DoRA, rsLoRA, PiSSA)
How LoRA is served in production
Where it’s going next
😩 Part 1: Why do we even need LoRA?
The pain of full fine-tuning
When you fine-tune a model normally, every weight gets updated. For a 7B-parameter model, that means:
Weights: all 7 billion numbers sit in GPU memory
Gradients: one more number per weight
Optimizer states: Adam keeps extra running statistics per weight
A common rule of thumb for mixed-precision Adam is roughly 16 bytes per parameter. That’s an estimate, not an exact figure. For 7B parameters it comes to about 112 GB before you count activations. That’s more than one GPU, & far more than a gaming card.
The original LoRA paper (Hu et al., 2021) opens with this scaling problem. For GPT-3 175B, LoRA can reduce the number of trainable parameters by 10,000 times & the GPU memory requirement by 3 times compared with full fine-tuning using Adam.
The storage & deployment pain
Say you serve 50 customers & each wants a custom model. With full fine-tuning you save 50 full copies of the model. At 14 GB each (7B in 16-bit), that’s 700 GB. Swapping between them at request time is painful.
The “forgetting” pain
Full fine-tuning can overwrite what the model already knew. Teach it your legal jargon & it may get worse at basic math. That’s called catastrophic forgetting. (Detailed blog link on Catastrophic Forgetting)
Pointers: what we really wanted
Train far fewer parameters
Keep the original model frozen (safe & reusable)
Get tiny files we can swap in & out
Pay no extra cost at inference
LoRA ticks all four boxes.
🧠 Part 2: The core idea!
The “mixing board” analogy
Picture a music mixing board with 4,096 sliders (the frequencies). To make a song sound “warmer”, you don’t need to move each slider independently. Pulling a few combined knobs, like “more bass” or “less treble”, gets you there.
The change you want is simple, even though the board is huge. Researchers found the same thing in neural networks. The LoRA authors hypothesized that the update matrices in language model adaptation have a low “intrinsic rank”. In plain words, the change you need is much simpler than the model itself.
The math, made friendly
A layer has a weight matrix W of shape d × k. Full fine-tuning learns a full-size update ΔW, with the same number of entries as W.
LoRA says: don’t learn ΔW directly. Learn it as the product of two skinny matrices.
Full update (huge) LoRA update (tiny)ΔW = [ d × k ] → ΔW = B × A [d × r] [r × k]r is the "rank", usually 8, 16, 32, 64...B is matrix of dimenstions [d x r] where d is rowsA is matrix of dimenstions [r x k] where k is columnsB x A gives back the matrix of [d x k]
r is the rank, the number of independent directions the update can use. The actual number of trainable knobs is r·(d + k), which is 65,536 at r=8, not 8 (calculation table mentioned below). “r controls how many knobs you get.”
The forward pass becomes:
h = W·x + (alpha / r) · B·(A·x) ↑ ↑ frozen trainable, tinywhere x is the input to the layer (list/vector of numbers coming in)
The paper describes this as a learned low-rank additive update ΔW = BA with r far smaller than min(d,k).
🔥 Derivation of the Forward Pass equation (Hand Written Notes)!
Let’s count the savings
Take one 4096 × 4096 layer:
That’s 256× fewer for this layer, & it’s why files shrink from gigabytes to megabytes for the adapter file only. You still need the full base model to run it. The win is that one base model can be shared across many tiny adapters.
256× is per layer, for this square shape. For a whole model the reduction is usually much bigger, because LoRA is typically applied only to some layers (often the attention projections) and everything else is frozen. The original paper reports about 10,000× fewer trainable parameters for GPT-3 175B.
Why B starts at zero
A is initialized randomly (small values)
B is initialized to zeros
So at step 0, B·A = 0 & the model behaves exactly like the original. Training starts from a safe place, not from a randomly broken model.
What is alpha?
alpha / r is a scaling factor on the update. Think of it as a volume knob on the overlay sheet. A higher alpha makes the adapter's voice louder. It controls how strongly the adapter’s output gets added to the original output:
scale 0: the adapter is ignored, giving the original model
scale 1: the correction is added as-is
scale 2: the correction is added twice as strongly
alpha is generally set at 2r or r, If you always set alpha = 2r, then alpha/r = 2, every time.
Zero extra inference cost
After training, you can merge the update into the base weights:
W_new = W + (alpha / r) · B·A
The result is a normal-sized model with no added latency. The paper highlights that, unlike adapter layers, LoRA adds no additional inference latency.
This is where most people lose quality. LoRA has few knobs, but each one matters.
The 2025 reality check: “LoRA Without Regret”
In late 2025, John Schulman & Thinking Machines Lab published a careful study called “LoRA Without Regret”. Their headline, as summarized in the Hugging Face TRL docs: LoRA can match full fine-tuning performance when configured correctly, using only about 67% of the compute.
The key findings, in plain pointers:
Apply LoRA to all layers. Applying it to all layers, including MLP & MoE layers, is recommended, & attention-only LoRA is consistently outperformed.
Use a higher learning rate. The optimal LoRA learning rate was consistently about 10x higher than for full fine-tuning. An independent reproduction saw the same roughly 10x gap.
Mind your dataset size. When a dataset exceeds LoRA’s capacity, LoRA underperforms full fine-tuning. Low ranks run out of room first, & higher ranks track full fine-tuning for longer.
Watch batch size. LoRA is less tolerant of large batch sizes than full fine-tuning, & raising the rank doesn’t close that gap.
For reinforcement learning, tiny is enough. LoRA matched full fine-tuning in RL even at rank 1.
One honest caveat: the reproduction found that MLP + attention beat MLP-only. So treat “all layers” as the safe default.
Cheat sheet
Note the PEFT default: PEFT’s default LoRA adds trainable weights only to the query & value layers of each attention block. Many tutorials inherit this, so override it. The PEFT docs themselves note that adding trainable weights to all the linear layers (as QLoRA does) can match a fully fine-tuned model.
The cheat-sheet rows for rank, dropout & batch size are my practical starting points, not hard rules. Always validate on your own eval set.
🚀 Part 5: Fine-tune a model with LoRA (Hugging Face PEFT + TRL)
If your GPU doesn’t support bf16, set “bf16=False”
Merge for Deployment
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel# Recent Transformers versions use `dtype=`; older ones expect `torch_dtype=`.base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16)model = PeftModel.from_pretrained(base, "qwen-lora/final")merged = model.merge_and_unload() # fold adapter into base weightsmerged.save_pretrained("qwen-merged")AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("qwen-merged")
Merge or keep separate?
Merge when you ship one fixed model & want maximum speed.
Keep separate when you serve many adapters, or want to hot-swap them.
🧊 Part 6: QLoRA, for when the model still doesn’t fit
LoRA shrinks what you train. The frozen base model still takes memory. QLoRA also compresses that base.
Analogy: LoRA is writing a small note. QLoRA also stores the giant textbook as a compressed PDF while you write the note.
QLoRA applies 4-bit quantization to the pretrained weights & trains LoRA modules on top of that quantized representation. It relies on the 4-bit NormalFloat (NF4) format, double quantization & paged optimizers to cut memory further. In practice, a 65B fine-tune fits on a single 48 GB GPU, & a 7B one on a gaming card.
If the base model fits in bf16 with room for activations, plain LoRA trains faster.
If it doesn’t fit, QLoRA makes the run possible at the cost of slower steps.
🧬 Part 7: The LoRA family (variants worth knowing)
Dozens of variants exist. Here are the ones you’ll actually meet:
Honest notes on variants
DoRA costs more. In PEFT’s own benchmark on Llama-3.1–8B (see the PEFT docs), DoRA without caching took 139% more time than LoRA. The docs also recommend merging weights for inference.
Gains aren’t guaranteed. One 2026 study on knowledge memory found simply swapping to DoRA or PiSSA does not reliably yield large gains. Results flipped between Llama & Qwen.
My advice: get a strong LoRA baseline with the right settings from Part 4 first. Try variants second, & measure on your own task.
⚖️ Part 8: What LoRA can & can’t do
Strengths
Cheap training
Tiny checkpoints
Fast switching between tasks
Less forgetting (usually)
The “learns less, forgets less” trade-off
A well-known 2024 study, “LoRA Learns Less and Forgets Less” (Biderman et al.), compared LoRA with full fine-tuning on code & math. In standard low-rank settings, LoRA substantially underperformed full fine-tuning, but it kept the base model’s performance on out-of-domain tasks better. A likely reason: full fine-tuning learns perturbations with a rank 10–100× greater than typical LoRA configurations.
Later work reframed this. With the right setup & datasets within capacity, the gap closes (Part 4).
A subtle catch: “intruder dimensions”
A NeurIPS 2025 paper (Shuttleworth et al., “LoRA vs Full Fine-tuning: An Illusion of Equivalence”) found that LoRA & full fine-tuning change weights differently. LoRA creates new, high-ranking singular vectors called “intruder dimensions”, & much of LoRA’s forgetting is tied to them. The same study notes that with large learning rates, LoRA models can forget more than full fine-tuning, so less forgetting is not a guarantee.
Takeaway: don’t push the learning rate to the moon just because LoRA tolerates it.
When to prefer full fine-tuning
You’re doing huge pre-training-scale training, where LoRA starts falling behind
You have a massive dataset & a big compute budget
You’ve tested LoRA & it plateaus well below full fine-tuning
🏭 Part 10: The best ways LoRA is used
- Instruction & domain tuning
Take an open model & teach it your tone, your format, or your domain. This is the bread & butter use.
- Reinforcement learning post-training
RL gives little information per episode, so tiny adapters have enough capacity. One independent reproduction trained Qwen2.5–0.5B on GSM8k with GRPO. The rank-1 LoRA beat both full fine-tuning configurations.
- Multi-tenant serving: one base model, many adapters
This is where LoRA’s economics shine. Analogy: one coffee machine, many flavor pods. The expensive machine (base model) loads once. Each customer’s tiny pod (adapter) swaps in per request.
The base model stays loaded once, & customer-specific adapters swap in at request time.
Research systems like S-LoRA keep adapters in host memory & move the ones needed by running requests onto the GPU.
The vLLM project lists efficient multi-LoRA support for dense & MoE layers as a built-in feature.
vllm serve meta-llama/Llama-3.1-8B-Instruct \ --enable-lora \ --max-loras 8 \ --max-cpu-loras 64 \ --max-lora-rank 64 \ --lora-modules customer-a=/adapters/customer-a customer-b=/adapters/customer-b
(Llama repos are gated, so you’ll need access approval & a Hugging Face login. Any ungated base model works the same way.)
What the flags mean:
max_loras is how many adapters can be active together in a batch (the GPU working set)
max_cpu_loras is how many stay warm in host memory before eviction
max_lora_rank is the largest rank the server accepts
Clients pick an adapter through the model field:
from openai import OpenAIclient = OpenAI(base_url="<http://localhost:8000/v1>", api_key="none")resp = client.chat.completions.create( model="customer-a", # selects that customer's adapter messages=[{"role": "user", "content": "Summarize this contract clause..."}],)print(resp.choices[0].message.content)
Or use the Python API:
from vllm import LLM, SamplingParamsfrom vllm.lora.request import LoRARequestllm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enable_lora=True, max_loras=4, max_lora_rank=64)out = llm.generate( ["Explain force majeure in simple words."], SamplingParams(max_tokens=200), lora_request=LoRARequest("customer_a", 1, "/adapters/customer-a"),)print(out[0].outputs[0].text)
Reality check: adapters aren’t free at inference when unmerged. One published vLLM benchmark found that applying LoRA to all linear layers cut throughput by 23.9%–47.0%, depending on rank. Your numbers will vary with hardware & batch mix, so measure your own setup. For a single fixed model, merging is still faster.
- Image models
LoRA isn’t only for text. It’s heavily used to teach diffusion models a new style or character with a small file. PEFT even ships LoRA support for convolution layers, & it supports Conv2d layers with groups != 1.
- Adapter libraries
Teams keep one adapter per task (support, legal, code review) & route requests to them, instead of maintaining many full models.
🧯 Part 11: Common mistakes & a debugging checklist
❌ Using the default q_proj/v_proj only. Switch to all-linear.
❌ Reusing your full fine-tuning learning rate. It’s probably ~10× too low for LoRA.
❌ Cranking rank to fix everything. Higher rank adds capacity for big datasets, but it won’t cure a bad learning rate or an oversized batch (Part 4).
❌ Huge batch sizes. LoRA is touchier about them, & more rank doesn’t help.
❌ Merging into a quantized base without care. Merge in higher precision, then re-quantize, & test.
❌ Judging by loss alone. Evaluate on real tasks, including something outside your domain to catch forgetting.
❌ Skipping version pinning. PEFT moves fast, so pin your library versions.
Quick debugging flow:
Run model.print_trainable_parameters(). Is it a sensible fraction, not 0 & not 100%?
Does loss decrease in the first 50 steps? If not, check the learning rate & target modules.
Loss unstable? Lower the LR, lower alpha, or try rsLoRA at high ranks.
Good loss but bad outputs? Check your chat template & data formatting before blaming LoRA.
🔮 Part 12: The future of LoRA
What’s happening now
The variant explosion continues. Recent PEFT releases added a batch of new methods, including HiRA, GLoRA, VeLoRA, & Uni-LoRA, plus automatic LoRA target selection. There’s also a Riemannian-preconditioned LoRA optimizer meant to avoid tuning separate learning rates for A & B. Check the PEFT release notes for the current list.
Mixture-of-Experts support is maturing. PEFT’s target_parameters feature lets LoRA target nn.Parameters directly, useful for MoE layers. Meanwhile vLLM added multi-LoRA serving for MoE families including GPT-OSS, Qwen3-MoE, DeepSeek, & Llama MoE.
Memory-focused methods are appearing. VeLoRA targets activation memory rather than parameter count.
Understanding is improving. Papers on capacity, intruder dimensions, & forgetting are replacing “just try rank 8” with real guidance.
My predictions (opinion, not fact)
LoRA becomes the default for post-training & personalization. The “low-regret” result makes it easy to justify.
Adapters become products. Expect marketplaces & registries of task adapters, much like packages today.
Better routing & composition. Systems will pick or blend adapters on the fly, which makes continual learning less risky.
Smarter rank allocation. Instead of one rank everywhere, methods will assign more capacity where it’s needed.
The gap with full fine-tuning keeps shrinking for small & medium datasets. Pre-training-scale data will likely stay full-weight territory.
⭐️ Final takeaway
Why LoRA exists: full fine-tuning is heavy on memory, storage & deployment, & it can cause forgetting.
The idea: freeze W, learn the update as B·A with a small rank r. Because B starts at zero, you begin exactly at the base model.
The payoff: often hundreds to thousands of times fewer trainable parameters, depending on model size, rank & targeted layers. You also get tiny files & no inference cost once merged.
Get the settings right: target all linear layers, use a learning rate around 10× higher than full fine-tuning, watch batch size, & match rank to dataset size.
Pick tools by constraint: plain LoRA when the model fits, QLoRA when it doesn’t. Try DoRA, rsLoRA & PiSSA only after a strong baseline, & measure the gains yourself.
Know the limits: LoRA can lag on huge datasets, & high learning rates can still cause forgetting.
Best uses: domain & instruction tuning, RL post-training (even rank 1), & multi-adapter serving that lets one GPU act like many models.
Where it’s heading: more variants, better MoE support, smarter serving, & adapters as everyday building blocks.
If you remember one sentence, make it this: don’t rebuild the encyclopedia, clip on a smart overlay.
If this helped, a clap or a comment with your own LoRA settings would make my day. Happy fine-tuning!
✨ Where to go from here!
You have come this far, which really means a lot. The further steps are as follows:
🎊 Appreciate yourself for deciding to read this blog, which has exponentially enhanced your knowledge.
🎉 Follow me on Medium (It’s free) & opt in for email notifications to stay updated with my latest articles on multi-tech (Complete AIOPS).
🎯 Follow me on LinkedIn (It’s free) to get the latest tech dose & tips & tricks.