From one weight and a Perceptron to 36-layer gradients, the 68.87 GB mixed-precision VRAM wall, and a 21.9-second Rank-8 LoRA update on an NVIDIA RTX PRO 6000 Blackwell GPU
By Arun Prabhakaran · @LatentAILabs
Watch the full 25-minute companion video on YouTube (@LatentAILabs):
Introduction: Why Teaching an 8 GB Model One New Fact Can Overflow a 24 GB GPU
When you load Qwen/Qwen3-4B-Instruct-2507 onto a GPU in bfloat16 precision (2 bytes per parameter), its 4,022,468,096 parameters occupy 8.04 GB of memory (7.49 GiB). Inference runs effortlessly on any modern 12 GB to 16 GB consumer GPU.
Now suppose you ask this pretrained model a specialized engineering question from a spacecraft manual that was not in its pretraining dataset:
Input Prompt:“In the Helios-9 spacecraft transceiver, what exact fault does status code HX-407 indicate and what is the recovery action?”
Ground-Truth Specification:“Status code HX-407 indicates cryocooler piezo-valve resonance drift above 42 Kelvin. Recover by switching laser bias to Aux-Bus-B.”
Untrained Qwen3–4B Output (Step 0, Measured):“Status code HX-407 in the Helios-9 spacecraft transceiver indicates a failure in the high-frequency oscillator module, and the recovery action is to initiate a soft reset of the transceiver and perform a diagnostic…”
Because the pretrained weights have never encoded the HX-407 specification, the model generates a fluent hallucination. If you attempt to teach it this specification using textbook mixed-precision (BF16/FP32) full fine-tuning across all 4.02 billion weights, two engineering obstacles appear immediately:
-
The 68.87 GB Mixed-Precision VRAM Wall: Allocating BF16 gradients, FP32 master weights, and FP32 AdamW first- and second-moment states across 4.02 billion parameters consumes 64.36 GB (16 bytes per parameter) plus 4.51 GB of forward activations — 68.87 GB total (8.57× the inference weight footprint, calculated from the 16-byte layout plus an estimated 4.51 GB of activations, not a measured run), more than a 24 GB or 48 GB GPU can hold.
-
Overwriting Pretrained Directions: Pushing aggressive dense updates directly into W_0 mutates the pretrained weight matrices that store general language and reasoning capabilities. Catastrophic forgetting is the documented risk (Biderman et al., 2024), though our own short run did not show it (see Part 7).
In Episode 7 of the How AI Actually Works series by @LatentAILabs, we trace how a neural network learns from a single floating-point weight all the way to a 4-billion-parameter Transformer rewiring on an NVIDIA RTX PRO 6000 Blackwell (94.97 GiB VRAM) GPU.
Part 1: What Are Weights, Biases, Perceptrons, and MLPs in a Neural Network?
A neural network stores all of its knowledge as learnable floating-point numbers called weights (w) and biases (b). A Perceptron computes a weighted sum of inputs plus a bias and applies a non-linear activation function (y = sigma(w · x + b)). Stacking Perceptrons into expansion and projection layers creates a Multi-Layer Perceptron (MLP) evaluated as dense weight matrices (W).
Step 1.1: A Single Learnable Weight (y = w · x)
Before inspecting 4.02 billion parameters, isolate a single scalar connection multiplying an input activation x by a learnable weight w.
Suppose our input is x = 2.0, our target is y_true = 1.00, and our weight starts at w = 0.40:
Forward Pass: The connection computes y = w · x = 0.40 × 2.0 = 0.80.
Squared Error Loss (L): Measuring (y - y_true)^2 = (0.80 - 1.00)^2 = 0.04 shows our prediction is 0.20 too low.
Exact Gradient Step: Differentiating L = (w · x - y_true)^2 with respect to w gives dL/dw = 2 · (y - y_true) · x = 2 × (-0.20) × 2.0 = -0.80. Updating w via gradient descent w_new = w - lr · (dL/dw) with learning rate lr = 0.125 yields w_new = 0.40 - 0.125 × (-0.80) = 0.50, so the next forward pass produces y = 0.50 × 2.0 = 1.00 (L = 0).
Step 1.2: From One Weight to a Perceptron (Weights + Bias + Non-Linear Activation)
A single weight can only scale one feature through the origin. To detect patterns across d input features x = [x_1, x_2, ..., x_d], a Perceptron combines three components:
-
Learnable Weight Vector (w): Multiplies each input coordinate x_i by a dedicated weight w_i (w · x = sum_i w_i · x_i).
-
Learnable Bias (b): Shifts the decision boundary (z = w · x + b) so the neuron can activate even when inputs are offset from zero.
-
Non-Linear Activation Function (sigma): Without a non-linear function such as ReLU(z) = max(0, z) or SiLU(z) = z · sigmoid(z), multiplying 36 linear matrices in sequence would algebraically collapse into a single flat matrix (W_36 · ... · W_1 = W_effective). Applying y = sigma(z) bends the feature space at every layer.
Step 1.3: Multi-Layer Perceptrons (MLPs) and Qwen3–4B’s 4.02B Weight Architecture
When we arrange d_out Perceptrons in parallel so that every neuron reads the same input vector x (d_in = 2,560), their individual weight vectors stack into the rows of a 2D Weight Matrix (W) of shape [d_out, d_in]. A Multi-Layer Perceptron (MLP) chains an expansion matrix, an elementwise non-linearity, and a projection matrix so the network can combine features non-linearly before returning to the residual stream.
Inspecting Qwen/Qwen3-4B-Instruct-2507 on our GPU (learning_results.json) reveals the exact parameter accounting across its 36 Transformer layers (hidden_size = 2,560):
Grouped-Query Self-Attention (q_proj, k_proj, v_proj, o_proj + QK-Norm): Self-Attention projects each token into queries (Q), keys (K), and values (V) so tokens can route contextual information across the sequence via Softmax(Q · K^T / sqrt(d_k)) · V. Notably, Qwen3-4B sets an explicit head_dim = 128 (rather than 2,560 / 32 = 80), so its 32 query heads expand the 2,560-D hidden state to 32 × 128 = 4,096 dimensions inside q_proj ([4,096 × 2,560] = 10,485,760 weights), while 8 KV heads give k_proj and v_proj shape [1,024 × 2,560] (2,621,440 weights each), and o_proj projects [2,560 × 4,096] (10,485,760 weights). Each layer also includes per-head RMSNorm vectors q_norm (128 params) and k_norm (128 params).
SwiGLU Multi-Layer Perceptron (gate_proj, up_proj, down_proj): Instead of a single hidden layer, Qwen3–4B uses a gated SwiGLU MLP (down_proj(SiLU(gate_proj(x)) ⊙ up_proj(x))) with intermediate_size = 9,728 (3.8× expansion). Each of gate_proj ([9,728 × 2,560]), up_proj ([9,728 × 2,560]), and down_proj ([2,560 × 9,728]) contains 24,903,680 weights (74,711,040 MLP weights per layer).
Exact Total (4,022,468,096 Parameters): Each of the 36 layers holds 26,214,400 attention matrix weights + 74,711,040 MLP weights + 5,376 norm weights = 100,930,816 parameters per layer (3,633,509,376 across 36 layers, or 90.33% of the model). Adding the tied 151,936 × 2,560 token embedding / output head matrix (388,956,160 parameters, 9.67%) and final RMSNorm (2,560) yields 4,022,468,096 parameters (8,044,936,192 bytes = 8.04 GB decimal in bfloat16).
Part 2: How Do Logits, Softmax, and Cross-Entropy Loss Work in LLMs?
At each token position, an LLM projects its final hidden vector across all 151,936 vocabulary tokens to produce unnormalized scores called logits (z_i). Softmax converts logits into probabilities (P_i = exp(z_i) / sum_j exp(z_j)), and Cross-Entropy Loss scores the ground-truth token via the negative natural logarithm L = -ln(P_correct) in nats.
Why Language Models Minimize -ln(P_correct) Instead of Squared Error
In Part 1 we used Squared Error (y - y_true)^2 for a continuous scalar regression target. Next-token prediction, however, is a 151,936-way discrete classification problem. Applying Softmax followed by Cross-Entropy Loss (L = -ln P_correct) creates a loss surface that is convex with respect to the logits z and yields a clean, bounded logit gradient dL/dz_i = P_i - 1{i == correct} in [-1, 1].
In m1_cross_entropy_and_baseline of learning_results.json, we measured three distinct regimes of L = -ln(P_correct) across the first 10 answer tokens of our HX-407 specification ("Status code HX-407 indicates cryocooler..."):
-
Confident & Correct (P_correct → 1.0, Loss → 0.0 nats): Once the prefix "Status code H" is conditioned on the prompt, Qwen3-4B predicts "X" with probability P = 99.9% (L = -ln(0.99941) = 0.001 nats), " H" with P = 82.6% (L = 0.19 nats), and " code" with P = 80.6% (L = 0.22 nats).
-
Uncertain / Uniform Baseline (P = 1 / 151,936, Loss = 11.9 nats): Uniformly guessing across 151,936 vocabulary tokens (P = 0.000658%) yields L = -ln(1 / 151,936) = 11.9 nats (perplexity 151,936). On the opening token "Status", Qwen3-4B assigns P = 0.81% (L = 4.82 nats).
-
Confident & Wrong (P_correct → 0, Unbounded Logarithmic Penalty): As P_correct → 0, -ln(P_correct) rises steeply without bound. When the ground-truth text transitions to " indicates cryocooler", the token " indicates" incurs 25.1 nats and the unseen domain token " cry" drops below 10^-12 probability, hitting our evaluation script's -ln(max(P, 10^-12)) clamp ceiling at 27.6 nats.
Full Sequence Baseline Metrics (Step 0): Across the complete HX-407 target sequence, untrained Qwen3-4B averages 4.37 nats (perplexity exp(4.3724) = 79.24, corresponding to a geometric-mean token probability of 1.26% and an arithmetic-mean target token probability of 56.0%), compared to 2.79 nats (perplexity 16.36) on general diagnostic English sentences.
Part 3: How Does Backpropagation Compute Weight Gradients via the Chain Rule?
Backpropagation applies the Chain Rule of Calculus in reverse from the scalar loss L at Layer 35 back to Layer 0. For any linear layer y = W · x + b, each weight gradient dL/dW[i, j] is the product of the upstream output sensitivity dL/dy_i and the saved forward input activation x_j (dL/dW = (dL/dy) · x^T).
The Two Reverse-Mode Equations Executed at Every Linear Layer
Consider a single linear projection y = W · x + b (written in column-vector notation, or Y = X · W^T + b for batched row vectors in PyTorch). During loss.backward(), the layer receives the upstream gradient g = dL/dy from the layer above and computes two products:
-
The Weight Gradient (dL/dW = g · x^T, or g^T · X for batched rows): Because dy_i / dW[i, j] = x_j, the sensitivity of the loss to weight W[i, j] is (dL/dy_i) · x_j. Notice two critical engineering consequences: first, if an input feature x_j = 0 during the forward pass, its weight receives zero gradient (dL/dW[i, j] = 0). Second, computing dL/dW requires keeping the forward activations x of all 36 layers in GPU memory until the backward pass reaches each layer (unless activation checkpointing recomputes them).
-
The Backward Activation Gradient (dL/dx = W^T · g, or g · W for batched rows): To pass the error signal backward to earlier layers (Layer 34 → ... → Layer 0), the layer multiplies the upstream gradient by its transposed weights W^T.
Run the script below to verify that manual Chain Rule formulas match PyTorch autograd down to float32 machine precision (~0.00e+00 to 1e-7):
import torchimport torch.nn.functional as F torch.manual_seed(42) # 1. Single Perceptron: y = SiLU(w . x + b)x = torch.tensor([2.0, -1.0, 0.5, 1.5], requires_grad=True)w = torch.tensor([0.4, -0.2, 0.8, 0.1], requires_grad=True)b = torch.tensor(0.05, requires_grad=True) z = torch.dot(w, x) + by = F.silu(z)target = torch.tensor(1.0)loss = 0.5 * (y - target) ** 2loss.backward() # Manual Chain Rule: dL/dz = (y - target) * SiLU'(z), where SiLU'(z) = sig(z) * (1 + z * (1 - sig(z)))sig_z = torch.sigmoid(z.detach())silu_grad = sig_z * (1.0 + z.detach() * (1.0 - sig_z))dL_dz = (y.detach() - target) * silu_gradmanual_grad_w = dL_dz * x.detach()manual_grad_b = dL_dz print(f"Perceptron output y = {y.item():.4f}, Loss = {loss.item():.6f}")print(f"Max |autograd dw - manual (dL/dz * x)| = {(w.grad - manual_grad_w).abs().max().item():.2e}")print(f"Max |autograd db - manual dL/dz| = {(b.grad - manual_grad_b).abs().item():.2e}") # 2. Batched Linear Layer (Row-Vector Convention: Y = X @ W^T)X_batch = torch.randn(4, 16, requires_grad=True)W_mat = torch.randn(8, 16, requires_grad=True)Y_out = X_batch @ W_mat.t()loss_mat = 0.5 * (Y_out ** 2).sum()loss_mat.backward() # Upstream sensitivity G = dL/dY = Y_out; Chain Rule: dL/dW = G^T @ X, dL/dX = G @ WG = Y_out.detach()manual_dW = G.t() @ X_batch.detach()manual_dX = G @ W_mat.detach()print(f"MLP Matrix Max |autograd dW - manual G^T @ X| = {(W_mat.grad - manual_dW).abs().max().item():.2e}")print(f"MLP Matrix Max |autograd dX - manual G @ W| = {(X_batch.grad - manual_dX).abs().max().item():.2e}") # Expected output:# Perceptron output y = 1.3312, Loss = 0.054856# Max |autograd dw - manual (dL/dz * x)| = 0.00e+00# Max |autograd db - manual dL/dz| = 0.00e+00# MLP Matrix Max |autograd dW - manual G^T @ X| = 0.00e+00# MLP Matrix Max |autograd dX - manual G @ W| = 0.00e+00
Part 4: Why Do Residual Skip Connections Prevent Vanishing Gradients Across 36 Layers?
In a plain 36-layer network (x_{l+1} = Layer(x_l)), backward gradients multiply 36 fractional layer Jacobians (prod_{l=0}^{35} dLayer_l / dx_l), shrinking exponentially toward zero (vanishing gradients). Adding a Residual Skip Connection (x_{l+1} = x_l + Layer(x_l)) changes each layer's Jacobian to I + dLayer_l / dx_l, creating an identity path that preserves gradient magnitude all the way to Layer 0.
Experiment 1: Controlled 36-Layer Residual Ablation (1.1e-18 vs. 244)
In residual_highway_ablation_36_layers of measure_learning.py (learning_results.json), we built two 36-layer networks (d = 256, weights initialized with N(0, 0.5 / sqrt(d)) so each layer's spectral contraction is ~0.5, and scalar objective L = ||h_35||^2):
Plain 36-Layer Chain (h_{l+1} = tanh(W_l · h_l)): Because both the forward activations h_l and backward Jacobians contract by ~0.5 per layer across 36 layers, even Layer 35 has a tiny weight gradient norm of 2.5e-19 (since dL/dh_35 = 2 · h_35 is already contracted), and Layer 0 collapses to 1.1e-18. At 10^-18, any standard step lr · g is negligible next to ~10^-2 weights, and in AdamW the denominator stabilizer eps = 1e-8 completely swamps sqrt(v_hat) ≈ 1e-18, freezing early layers.
Residual 36-Layer Chain (h_{l+1} = h_l + 0.2 · tanh(W_l · h_l)): Adding the identity skip connection h_l + ... introduces +I into every layer's derivative (dh_{l+1} / dh_l = I + 0.2 · d(Layer_l)/dh_l). Now Layer 35 has Frobenius gradient norm 204 and Layer 0 receives 244.
Experiment 2: Live 36-Layer Gradient Telemetry on Qwen3–4B-Instruct-2507
In m2_backprop_36_layers of learning_results.json, we recorded the exact Frobenius gradient norms (||dL/dW||_F) across all 36 layers of Qwen3-4B-Instruct-2507 on the HX-407 prompt:
Layer 0 (1st layer): q_proj = 2.15, mlp.down_proj = 4.71 (no gradient collapse after traversing 36 transformer blocks in reverse).
Layer 6 (7th layer): q_proj = 3.45, mlp.down_proj = 95.8 (highest MLP gradient spike in the network).
Notice why mlp.down_proj often exhibits larger Frobenius gradient norms than q_proj (such as 95.8 at Layer 6): down_proj has 2.37× more elements ([2,560 × 9,728] = 24.9M vs. 10.49M), multiplies a high-magnitude gated activation SiLU(gate_proj(h)) ⊙ up_proj(h), and writes directly into the residual stream without passing through an attention Softmax bottleneck.
Part 5: Why Does the AdamW Optimizer Converge Faster Than Vanilla SGD?
Vanilla Stochastic Gradient Descent (SGD) scales every weight gradient by the same fixed scalar (W ← W - lr · g_t), causing it to oscillate across steep high-curvature directions while crawling along flat directions. AdamW maintains per-parameter momentum (m_t) and second-moment variance (v_t) plus decoupled weight decay, reaching 0.094 nats in 25 steps versus 3.45 nats for SGD.
The Three Update Equations of AdamW (Loshchilov & Hutter, 2019)
Before we count the memory bytes required to train a 4B model in Part 6, we need to see what the optimizer actually stores for every trainable parameter w with gradient g_t = dL/dw at step t:
-
First Moment — Momentum (m_t, beta_1 = 0.9): Computes an exponential moving average of gradients m_t = beta_1 · m_{t-1} + (1 - beta_1) · g_t (with bias correction m_hat_t = m_t / (1 - beta_1^t)). Opposite-sign oscillations across steep canyon walls cancel out in m_t, while consistent downhill gradients build velocity.
-
Second Moment — Adaptive Variance (v_t, beta_2 = 0.999): Computes an exponential moving average of squared gradients v_t = beta_2 · v_{t-1} + (1 - beta_2) · g_t^2 (with bias correction v_hat_t = v_t / (1 - beta_2^t)). Dividing by sqrt(v_hat_t) + eps normalizes each parameter's step size by its historical root-mean-square gradient magnitude — essential in Transformers where Layer 6 down_proj (||g||_F = 95.8) and Layer 0 q_proj (||g||_F = 2.15) differ by 44×.
Measured 25-Step Trajectory on GPU (m3_sgd_vs_adamw_trajectory in learning_results.json)
In measure_learning.py, we initialized identical Rank-8 adapters on the q_proj matrices of Layers 30–35 and trained for 25 steps on a single aerospace example, comparing torch.optim.SGD(lr=1e-2) against torch.optim.AdamW(lr=1e-3, weight_decay=1e-2):
Step | Vanilla SGD Loss (nats) | AdamW Loss (nats) | AdamW Perplexity------+---------------------------+---------------------+------------------ 1 | 6.7457 | 6.7457 | 850.39 5 | 4.4491 | 3.5779 | 35.80 10 | 3.6770 | 2.2251 | 9.25 15 | 3.6016 | 1.2160 | 3.37 20 | 3.5232 | 0.3070 | 1.36 25 | 3.4469 | 0.0936 | 1.10
While Vanilla SGD drops quickly over the first 5 steps (6.75 → 4.45 nats) and then stalls in a flat valley (3.68 → 3.45 nats across Steps 10–25), AdamW's per-parameter RMS normalization drives loss below 0.10 nats by Step 25 (0.094 nats, 36.8× lower final loss).
Part 6: Why Does Fine-Tuning an 8 GB Model Take 68.87 GB of GPU VRAM?
Under standard ZeRO mixed-precision (BF16/FP32) accounting (Rajbhandari et al., 2020), every trainable parameter requires 16 bytes of persistent state: 2 bytes for BF16 weights (8.04 GB), 2 bytes for BF16 gradients (8.04 GB), 4 bytes for FP32 master weights (16.09 GB), and 8 bytes for FP32 AdamW m_t and v_t (32.18 GB), plus 4.51 GB of activations (68.87 GB total).
Why Mixed-Precision Training Needs FP32 Master Weights (BF16 Mantissa Swamping)
Why can’t we simply store weights, gradients, m_t, and v_t in bfloat16 (2 bytes each) and skip the 4-byte float32 copies?
bfloat16 Has Only 7 Fraction Bits (8 Bits of Precision Including the Implicit Lead Bit): While bfloat16 matches the 8-bit exponent range of float32, its 7-bit mantissa gives a relative unit-in-last-place (ULP) spacing of 2^-7 = 0.0078125 (so around w = 0.50, adjacent representable BF16 numbers are spaced by 2^-8 ≈ 0.0039).
Small AdamW Updates Round to Zero in BF16: With a typical fine-tuning learning rate lr = 1e-4 to 5e-4, a parameter step lr · m_hat_t / sqrt(v_hat_t) has magnitude ~1e-4. Adding 0.0001 to w = 0.50 in bfloat16 rounds straight back to 0.50 (floating-point swamping). Keeping FP32 master weights (23 fraction bits, spacing ~6e-8 around 0.50) accumulates tiny updates accurately before casting back to BF16 for each forward pass.
Complete 16-Byte Mixed-Precision Memory Breakdown (m3_memory_and_optimizer in learning_results.json)
Here is the exact byte-level accounting (1 GB = 10^9 bytes decimal) for N = 4,022,468,096 parameters:
Component | Bytes/Param | Exact Bytes | Decimal GB | Binary GiB-------------------------------------------+-------------+--------------------+------------+-----------1. Model Weights (bfloat16) | 2 bytes | 8,044,936,192 B | 8.04 GB | 7.49 GiB2. Weight Gradients (bfloat16) | 2 bytes | 8,044,936,192 B | 8.04 GB | 7.49 GiB3. Master Weights Copy (float32) | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB4. AdamW 1st Moment Momentum m_t (float32) | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB5. AdamW 2nd Moment Variance v_t (float32) | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB-------------------------------------------+-------------+--------------------+------------+-----------Subtotal (16-Byte Parameter + AdamW State) | 16 bytes | 64,359,489,536 B | 64.36 GB | 59.94 GiB6. Saved Forward Activations (Est. Batch) | — | 4,509,715,660 B | 4.51 GB | 4.20 GiB-------------------------------------------+-------------+--------------------+------------+-----------Total Full Mixed-Precision Training VRAM | — | 68,869,205,196 B | 68.87 GB | 64.14 GiB
Part 7: What Is LoRA Fine-Tuning and Why Is Matrix B Initialized to Zero?
Low-Rank Adaptation (LoRA) freezes 100% of the pretrained base weights W_0 and injects rank-r matrices A ([r × d_in]) and B ([d_out × r]) so y = W_0 · x + (alpha / r) · B · (A · x). Initializing A randomly and B = 0 guarantees B · A = 0 at Step 0 while ensuring dL/dB ≠ 0.
Step 7.1: Direct Weight Mutation and Catastrophic Forgetting (m4_catastrophic_forgetting)
In m4_catastrophic_forgetting (measure_learning.py), we unfroze the dense q_proj and down_proj matrices in Layers 32–35 and ran high-rate dense updates (SGD(lr=2.5e-2)) on a single aerospace fault example:
Step 0 → Step 3 (Prompt Template Alignment): Domain loss drops from 4.37 → 0.86 nats, and loss on our held-out general diagnostic set (GENERAL_PAIRS, which shares the same 1-sentence system prompt template) initially drops from 2.79 → 0.85 nats (perplexity 2.34) as the model aligns to the concise 1-sentence output format.
Step 3 → Step 15 (Overfitting): As dense updates continue, the training example's loss drops to 0.021 nats at Step 15 (0.29 at Step 6, 0.038 at Step 10), while general loss creeps up from 0.85 → 0.95 nats (perplexity 2.59). That is still far below its starting 2.79 nats, so in this 15-step run the dense update did not cause catastrophic forgetting. The measurable effect is format alignment followed by a slight drift. For comparison, the 40-step LoRA run in Part 8 ended with a higher general loss (1.71 nats), though the two runs differ in steps, examples and layers, so neither result says which method forgets less. Still, because W_0 itself was mutated in-place, the pretrained weights cannot be recovered without re a separate checkpoint. As shown in Biderman et al. (2024), "LoRA Learns Less and Forgets Less", restricting updates to a low-rank subspace acts as a strong structural regularizer — and because LoRA keeps W_0 frozen, you can adjust alpha or detach the adapter at runtime to restore 100% of the base model's behavior.
Step 7.2: Rank-8 LoRA Compression Math (196.9× Per q_proj, 198.18 MB State)
In m5_lora_compression_math of learning_results.json, we attach Rank-8 (r = 8, alpha = 16) LoRA adapters to all 7 linear projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) across all 36 layers (252 modules total):
Single q_proj Matrix ([4,096 × 2,560]): Instead of updating 4,096 × 2,560 = 10,485,760 weights, Rank-8 LoRA learns A ([8 × 2,560] = 20,480 weights) and B ([4,096 × 8] = 32,768 weights) = 53,248 weights (196.9× compression).
Full 36-Layer Model (252 Linear Modules): While 4,022,468,096 base weights remain 100% frozen, the 252 Rank-8 LoRA adapters add 16,515,072 trainable parameters — equivalent to 0.41% of the base model count (0.4089% of the combined 4,038,983,168 total).
VRAM Savings (64.36 GB → 198.18 MB of Parameter + Optimizer State): Storing FP32 LoRA parameters (4B) + FP32 AdamW m_t (4B) + FP32 AdamW v_t (4B) across 16,515,072 adapter weights takes just 198.18 MB (264.24 MB including 4B FP32 gradients). Combined with the 8.04 GB frozen BF16 base weights and ~4.5 GB of activations, total LoRA training VRAM is ~12.8 GB — easily fitting on a single 16 GB or 24 GB GPU.
Step 7.3: Why Must Matrix B Start at Exact Zero While A Starts Random?
By the Chain Rule on y = W_0 · x + (alpha / r) · B · (A · x) with upstream sensitivity g = dL/dy:
Gradient for B: dL/dB = (alpha / r) · g · (A · x)^T
This reveals why asymmetric initialization (A ~ KaimingUniform, B = 0) is mandatory:
If both A != 0 and B != 0: Then Delta_W = (alpha / r) · B · A != 0 at Step 0, injecting untrained random noise into all 252 linear layers before training even begins.
If both A = 0 and B = 0: Then A · x = 0 forces dL/dB = 0, and B^T = 0 forces dL/dA = 0. Both matrices stay stuck at zero forever (a dead saddle point).
With A ~ KaimingUniform and B = 0: At Step 0, B · (A · x) = 0, so the model starts with the exact pretrained outputs of W_0. During the Step 0 backward pass, since A != 0, the bottleneck activation h = A · x is non-zero, giving B a non-zero gradient dL/dB = (alpha / r) · g · h^T != 0. Once B updates to non-zero values after Step 0, B^T · g becomes non-zero on Step 1 and both A and B train together.
Part 8: How Fast Does Rank-8 LoRA Fine-Tune Qwen3–4B and Does Merging Add Latency?
Across 40 AdamW steps (lr = 5e-4), Rank-8 LoRA fine-tunes Qwen3-4B-Instruct-2507 on our 4-pair aerospace diagnostic batch in 21.92 seconds (0.55s/step), driving domain loss from 4.37 → 0 nats. Folding Delta_W into W_0 via W_merged = W_0 + (alpha / r) · B · A eliminates 504 adapter kernel launches and cuts forward latency from 24.5 ms to 14.6 ms.
Step 8.1: Live 40-Step LoRA Telemetry (m6_live_40step_lora_rewiring in learning_results.json)
Training 16,515,072 Rank-8 LoRA parameters with AdamW(lr=5e-4, weight_decay=0.01) on our 4 aerospace fault pairs (measure_learning.py Lines 515–595) completed 40 steps in 21.92 seconds:
Step | Domain Loss (nats) | Domain PPL | Arith. Mean Token Prob | Held-Out General Loss | HX-407 Output Status-----+--------------------+------------+------------------------+-----------------------+--------------------- 0 | 4.3724 | 79.24 | 56.017% | 2.7947 nats | Hallucinates osc. reset 5 | 0.1452 | 1.16 | 92.249% | 0.4656 nats | Exact 42K Aux-Bus-B spec 10 | 0.0018 | 1.00 | 99.828% | 0.8787 nats | Exact 42K Aux-Bus-B spec 20 | 0.0001 | 1.00 | 99.991% | 1.5364 nats | Exact 42K Aux-Bus-B spec 30 | 0.0000 | 1.00 | 99.997% | 1.6956 nats | Exact 42K Aux-Bus-B spec 40 | 0.0000 | 1.00 | 99.999% | 1.7113 nats | Exact 42K Aux-Bus-B spec
Notice what the held-out general loss column reveals about stopping criteria: by Step 5 to Step 10 (~2.7s to 5.5s), domain loss is already down to 0.15 → 0.0018 nats with exact factual recall on HX-407 and low general loss (0.47 → 0.88 nats). Continuing to train on the same small batch out to Step 40 pushes general_loss up to 1.71 nats (perplexity 5.54) as the adapter over-specializes to the 4 aerospace examples (still below the 2.79-nat starting value) — demonstrating why early stopping around Step 10 (or mixing general replay examples) is best practice, and why keeping W_0 frozen is so valuable.
Step 8.2: Spectral Energy of the Multi-Sample Gradient (m7_svd_intrinsic_rank)
In m7_svd_intrinsic_rank (measure_learning.py Lines 620–658), we accumulated the full [4,096 × 2,560] gradient matrix G = dL/dW of Layer 24's q_proj across our 4 training examples and computed its Singular Value Decomposition (G = U · diag(Sigma) · V^T):
Cumulative Spectral Energy (sum_{i=1}^k sigma_i² / sum_{all} sigma_j²): Top 1 captures 58.59%, Top 2 captures 78.31%, Top 4 captures 85.12%, Top 8 captures 91.50%, Top 16 captures 95.97%, and Top 64 captures 99.77%.
Perturbation Norm Ratio: After 40 steps, the learned Layer 24 q_proj update norm ||Delta_W||_F = 1.42 is just 1.81% of the frozen base weight norm ||W_0||_F = 78.2, consistent with the low intrinsic dimensionality of fine-tuning (Aghajanyan et al., 2021; Hu et al., 2022).
Step 8.3: Zero-Latency Weight Folding (W_merged = W_0 + (alpha / r) · B · A)
Because matrix multiplication is linear (W_0 · x + (alpha / r) · (B · A) · x = (W_0 + (alpha / r) · B · A) · x), once training completes we can fold B · A directly into W_0 (m8_zero_latency_weight_merge in learning_results.json):
Unmerged Adapter Forward Latency (252 modules × 2 extra GEMMs = 504 small kernel launches):24.5 ms
Merged In-Place W_merged Forward Latency (native base model graph):14.6 ms (1.67× faster, 0.0 ms added latency over the base architecture).
Numerical Note on BF16 In-Place Rounding (max_logit_difference_bf16 = 1.125): When unmerged, PyTorch computes (x.float() @ A^T) @ B^T in FP32 before casting to BF16. When merged in-place into BF16 (W_0 += ((B @ A) * scale).to(bfloat16)), W_merged rounds to the nearest BF16 grid step, producing a 1.125 max logit difference across 151,936 vocabulary entries while generating the exact same greedy token sequence. (Note: if you serve multiple tenant adapters simultaneously, keep W_0 unmerged so adapters can be hot-swapped per request.)
Part 9: Complete Neural Network Learning Blueprint
Putting all five mechanisms together, modern LLM fine-tuning maps input tokens through 36 layers of frozen Self-Attention and SwiGLU MLP matrices (W_0) plus low-rank LoRA adapters (B · A), scores predictions with Cross-Entropy Loss (-ln P), routes gradients backward along Residual Skip Connections (x + Layer(x)), updates A and B via AdamW, and merges Delta_W into W_0 for zero-overhead inference.
Watch the Companion Video & Explore the “How AI Actually Works” Series (@LatentAILabs)
Every mathematical derivation, 3D matrix visualization, and live GPU telemetry plot in this article is animated step by step in Episode 7 of our YouTube series:
Episode 7 (This Article — How AI Actually Learns: Qwen3–4B Backpropagation, AdamW & Rank-8 LoRA):YouTube Video · 60s YouTube Short
Episode 6 (How AI Generates Video: Inside Sora & Wan2.1 3D DiT on a Real GPU):YouTube Video
Episode 5 (How Vision-Language Models Actually See Images: ViT, CLIP, SigLIP2 & 3D M-RoPE):YouTube Video · Medium Article
Episode 4 (“Every Clock Says 10:10” — How AI Generates Images via Diffusion & Flow Matching):YouTube Video · Medium Article
Episode 3 (How AI Reads a Whole Book at Once: 1M Long Context, KV-Cache & RoPE):YouTube Video · Medium Article
Episode 1 (“Strawberry Has 2 R’s?” — How ChatGPT Actually Works Under the Hood):YouTube Video · Medium Article
Methodology, Hardware Setup & AI Disclosure
Hardware & Software Environment: All empirical benchmarks were executed on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU (94.97 GiB VRAM) using PyTorch 2.10.0+cu128, Hugging Face transformers, and open-weight Qwen/Qwen3-4B-Instruct-2507 (4,022,468,096 parameters, 36 layers, hidden_size = 2,560, head_dim = 128, intermediate_size = 9,728, vocab_size = 151,936, bfloat16 precision).
AI Disclosure: All 15 figures were programmatically rendered by the author using Manim Community Edition and Python directly from GPU telemetry logged in learning_results.json. Qwen3-4B-Instruct-2507 served as the experimental model under test, Qwen3-TTS generated the companion video narration, and LLM coding/drafting assistants (Claude / Antigravity) assisted with manuscript formatting and code verification. All benchmarks, equations, and PyTorch scripts were executed and verified by the author.
References & Primary Literature
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986).Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
He, K., Zhang, X., Ren, S., & Sun, J. (2016).Deep Residual Learning for Image Recognition. CVPR 2016, 770–778. https://arxiv.org/abs/1512.03385
Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020).ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC20. https://arxiv.org/abs/1910.02054
Aghajanyan, A., Gupta, S., & Zettlemoyer, L. (2021).Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021. https://arxiv.org/abs/2012.13255
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022).LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Biderman, D., Portes, J., Ortiz, J. J. G., et al. (2024).LoRA Learns Less and Forgets Less. TMLR 2024. https://arxiv.org/abs/2405.09673