# How LoRA Actually Works: Low-Rank Decomposition, Weight Merging, and Memory Breakdown Under the Hood > Source: > Published: 2026-10-07 12:35:30+00:00 If you try to full-parameter fine-tune an 8-billion parameter language model in 16-bit precision, your GPU memory requirement immediately explodes past 80 GB. The raw model weights only occupy 16 GB of VRAM ($8 \times 10^9 \text{ parameters} \times 2 \text{ bytes}$). Yet standard training crashes on a single 80 GB A100 or H100. Where does the missing 64+ GB of VRAM go? And why can Low-Rank Adaptation (LoRA) reduce that memory footprint by over 70% while updating fewer than 0.1% of the parameters and retaining 99% of full fine-tuning performance? Here is what actually happens mathematically, mechanically, and in physical GPU memory during LoRA fine-tuning and inference. When training a neural network with the standard AdamW optimizer, model weights are only a tiny fraction of your memory consumption. For every single trainable parameter, the system must track: $$\text{Total Memory per Trainable Parameter} = 2 + 2 + 4 + 4 + 4 = 16 \text{ bytes}$$ ``` +-------------------------------------------------------------+ | Full Parameter Training (16 bytes per parameter) | +-------------------------------------------------------------+ | Weights: 2B | Grads: 2B | Master: 4B | Mom1: 4B | Mom2: 4B | +-------------------------------------------------------------+ ``` For an 8B parameter model: This is why full fine-tuning requires multi-GPU distributed clusters (DeepSpeed ZeRO or FSDP). LoRA solves this problem at its root: by freezing the base weights, **it eliminates optimizer states and gradients for 99.9% of the model**. Why does freezing the model work? A standard linear layer in a transformer computes: $$h = W_0 x$$ where $W_0 \in \mathbb{R}^{d \times k}$, $x \in \mathbb{R}^{k}$, and $h \in \mathbb{R}^{d}$. For Llama 3 8B, the hidden dimension $d = 4096$. Each weight matrix has up to $4096 \times 4096 \approx 16.7\text{M}$ parameters with full mathematical rank 4096. In 2021, Edward Hu et al. published the foundational hypothesis behind LoRA: When adapting a pre-trained language model to a specific task or instruction format, the weight update matrix $\Delta W$ has an extremely low "intrinsic rank" ($r \ll d$). The pre-trained model already knows syntax, world facts, and language logic. Fine-tuning is merely steering attention and adjusting feature selection. You do not need to update 16 million degrees of freedom per matrix; you only need to update a tiny subspace of dimension $r$ (typically $r = 8, 16, \text{ or } 32$). Instead of directly learning a full $d \times k$ matrix $\Delta W$, LoRA decomposes $\Delta W$ into the product of two small, low-rank matrices: $$\Delta W = B \cdot A$$ where: ``` Full Update Matrix ΔW (d × k) LoRA Decomposition +---------------------------+ +-------+ | | | | +---------------------------+ d | | = d | B | × r | A | | | | (d×r) | +---------------------------+ +---------------------------+ +-------+ k k ``` Suppose $d = 4096, k = 4096$, and you choose rank $r = 16$: $$\text{Reduction Factor} = \frac{16,777,216}{131,072} = 128\times \text{ fewer parameters (99.2% reduction)}$$ Multiply that across 32 transformer layers, and your trainable parameters drop from 8 billion down to ~20 million. Two critical implementation details make LoRA stable and practical: How do you prevent random noise from wrecking the pre-trained model at the start of training? At step 0 of training: $$\Delta W = B \cdot A = 0 \cdot A = 0$$ The adapter initially produces an exact zero vector. The model starts training with the exact output of the pre-trained base model, ensuring smooth and stable gradient descent. The modified forward pass with LoRA includes a constant scaling multiplier $\frac{\alpha}{r}$: $$h = W_0 x + \frac{\alpha}{r} (B \cdot A) x$$ Why does $\frac{\alpha}{r}$ exist? When you experiment with different ranks (e.g., jumping from $r=8$ to $r=64$), the magnitude of the matrix multiplication $BAx$ naturally scales with $r$. The $\frac{\alpha}{r}$ term normalizes the magnitude of the adapter's update. This allows you to change the rank $r$ without needing to retune your learning rate from scratch. During training, the computation branches into two parallel paths: ``` Input x / \ / \ [Frozen Base Weight] [Matrix A (r × k)] (Down-project to r) W₀ x | | [Matrix B (d × r)] (Up-project to d) | | | Scale by (α / r) \ / \ / +------> (+) <----+ | Output h ``` During backpropagation: One of the biggest advantages of LoRA over earlier adapter architectures (like Houlsby adapters or prompt tuning) is that **LoRA introduces zero additional inference latency**. In production deployment, you do not keep two separate matrix multiplication branches in memory. Because matrix multiplication is distributive: $$h = W_0 x + \Delta W x = (W_0 + \Delta W) x$$ Before exporting your model for deployment, you merge the adapter weights directly into the base weights with a single matrix addition: $$W_{\text{serving}} = W_0 + \frac{\alpha}{r} (B \cdot A)$$ Once added, the adapter matrices $A$ and $B$ are discarded. The final serving model has the exact same architecture, tensor count, and latency as the original base model. ``` +-----------------------------------------------------------+ | Base Weights W₀ LoRA Weights (B × A) | | [ 4096 × 4096 ] + [ 4096 × 4096 ] | +-----------------------------------------------------------+ ↓ +-----------------------------------------------------------+ | Merged Weights W_serving | | [ 4096 × 4096 ] → Standard inference speed, zero latency | +-----------------------------------------------------------+ ``` If you need to switch tasks dynamically (multi-tenant serving), systems like **S-LoRA** or **vLLM Multi-LoRA** keep the base model frozen in VRAM and compute the small $BAx$ branch on the fly for distinct requests using custom batched GEMM kernels (like Punica). To see how straightforward the mechanics are, here is a complete, working LoRA linear layer implemented in pure PyTorch: ``` python import math import torch import torch.nn as nn class LoRALinear(nn.Module): def __init__( self, in_features: int, out_features: int, r: int = 16, lora_alpha: float = 32.0, lora_dropout: float = 0.05 ): super().__init__() # 1. Base pre-trained linear layer (frozen) self.base_layer = nn.Linear(in_features, out_features, bias=False) self.base_layer.weight.requires_grad = False self.r = r self.lora_alpha = lora_alpha self.scaling = lora_alpha / r if r > 0: # 2. Low-rank matrices A and B self.lora_A = nn.Parameter(torch.empty(r, in_features)) self.lora_B = nn.Parameter(torch.empty(out_features, r)) self.dropout = nn.Dropout(p=lora_dropout) if lora_dropout > 0 else nn.Identity() # 3. Initialization: Kaiming uniform for A, exact zeros for B nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5)) nn.init.zeros_(self.lora_B) self.merged = False def forward(self, x: torch.Tensor) -> torch.Tensor: if self.merged or self.r == 0: return self.base_layer(x) # Standard LoRA forward pass: W₀x + (α/r) * B(A(dropout(x))) base_out = self.base_layer(x) lora_out = (self.dropout(x) @ self.lora_A.T) @ self.lora_B.T * self.scaling return base_out + lora_out def merge(self): """Fold LoRA weights into base weights for zero-overhead serving.""" if self.r > 0 and not self.merged: # Compute ΔW = (α/r) * B @ A delta_w = (self.lora_B @ self.lora_A) * self.scaling self.base_layer.weight.data += delta_w self.merged = True def unmerge(self): """Subtract LoRA weights to resume training or swap adapters.""" if self.r > 0 and self.merged: delta_w = (self.lora_B @ self.lora_A) * self.scaling self.base_layer.weight.data -= delta_w self.merged = False ``` Here is how the memory footprint compares on an 8-billion parameter model (batch size 2, sequence length 2048): | Component | Full Fine-Tuning (BF16) | LoRA ($r=16$, All Linears) | QLoRA (4-bit Base + LoRA) | |---|---|---|---| | **Base Model Weights** | 16.0 GB (BF16) | 16.0 GB (BF16) | 4.5 GB (NF4) | | **Adapter Weights** | N/A | 0.08 GB (BF16) | 0.08 GB (BF16) | | **Gradients** | 16.0 GB | 0.08 GB | 0.08 GB | | **Optimizer States (AdamW)** | 96.0 GB | 0.48 GB | 0.48 GB | | **Activations (w/ checkpointing)** | ~6.0 GB | ~6.0 GB | ~6.0 GB | | **Total VRAM Needed** | **~134 GB** | **~22.6 GB** | **~11.1 GB** | | **Hardware Required** | 2x A100 (80GB) | 1x RTX 3090 / 4090 (24GB) | 1x RTX 3060 (12GB) | The original 2021 LoRA paper demonstrated proof-of-concept results by attaching adapters only to Query and Value projections. However, multiple empirical studies (such as QLoRA by Dettmers et al.) prove that targeting **all linear layers** ($q, k, v, o$, as well as MLP layers $gate, up, down_proj$) yields significantly higher accuracy and expressive capacity, even with a smaller rank like $r=8$ or $r=16$. A common convention is setting $\alpha = 2 \times r$ or $\alpha = r$. If you double your rank $r$ from 16 to 32 but leave $\alpha$ unchanged at 32, you have inadvertently halved the effective learning rate of your adapter ($\frac{\alpha}{r}$ drops from 2.0 to 1.0). Keep your $\frac{\alpha}{r}$ ratio consistent when sweeping rank values. If you are deploying a single specialized model in production, leaving LoRA as separate branches adds unnecessary kernel launches and memory bandwidth overhead. Always call `.merge_and_unload()` in Hugging Face PEFT or manually fold weights before saving your deployment artifact. Setting $r=128$ or $r=256$ rarely improves downstream task accuracy for instruction fine-tuning, but it drastically increases memory consumption and risk of catastrophic forgetting. For standard instruction tuning and classification, $r \in [8, 32]$ is almost always the empirical sweet spot.