How AI Actually Learns: Backpropagation, AdamW & Rank-8 LoRA Measured on Qwen3–4B

A LatentAILabs tutorial episode measured that full mixed-precision fine-tuning of Qwen3-4B-Instruct-2507 (4,022,468,096 parameters) would require 68.87 GB of VRAM — 64.36 GB for BF16 gradients, FP32 master weights and AdamW moments plus 4.51 GB of activations — versus the model's 8.04 GB bfloat16 inference footprint, while a Rank-8 LoRA update on an NVIDIA RTX PRO 6000 Blackwell GPU completed in 21.9 seconds. The episode walks from a single-weight gradient descent example (w moving from 0.40 to 0.50 at learning rate 0.125) to 36-layer backpropagation, and shows untrained Qwen3-4B hallucinating the fictional HX-407 spacecraft fault code before any training. The author notes the 68.87 GB figure is calculated from the 16-byte-per-parameter layout plus an estimated 4.51 GB of activations, not a measured run.

From one weight and a Perceptron to 36-layer gradients, the 68.87 GB mixed-precision VRAM wall, and a 21.9-second Rank-8 LoRA update on an NVIDIA RTX PRO 6000 Blackwell GPU By Arun Prabhakaran · @LatentAILabs Watch the full 25-minute companion video on YouTube @LatentAILabs : Introduction: Why Teaching an 8 GB Model One New Fact Can Overflow a 24 GB GPU When you load Qwen/Qwen3-4B-Instruct-2507 onto a GPU in bfloat16 precision 2 bytes per parameter , its 4,022,468,096 parameters occupy 8.04 GB of memory 7.49 GiB . Inference runs effortlessly on any modern 12 GB to 16 GB consumer GPU. Now suppose you ask this pretrained model a specialized engineering question from a spacecraft manual that was not in its pretraining dataset: Input Prompt:“In the Helios-9 spacecraft transceiver, what exact fault does status code HX-407 indicate and what is the recovery action?” Ground-Truth Specification:“Status code HX-407 indicates cryocooler piezo-valve resonance drift above 42 Kelvin. Recover by switching laser bias to Aux-Bus-B.” Untrained Qwen3–4B Output Step 0, Measured :“Status code HX-407 in the Helios-9 spacecraft transceiver indicates a failure in the high-frequency oscillator module, and the recovery action is to initiate a soft reset of the transceiver and perform a diagnostic…” Because the pretrained weights have never encoded the HX-407 specification, the model generates a fluent hallucination. If you attempt to teach it this specification using textbook mixed-precision BF16/FP32 full fine-tuning across all 4.02 billion weights, two engineering obstacles appear immediately: 1. The 68.87 GB Mixed-Precision VRAM Wall: Allocating BF16 gradients, FP32 master weights, and FP32 AdamW first- and second-moment states across 4.02 billion parameters consumes 64.36 GB 16 bytes per parameter plus 4.51 GB of forward activations — 68.87 GB total 8.57× the inference weight footprint, calculated from the 16-byte layout plus an estimated 4.51 GB of activations, not a measured run , more than a 24 GB or 48 GB GPU can hold. 2. Overwriting Pretrained Directions: Pushing aggressive dense updates directly into W 0 mutates the pretrained weight matrices that store general language and reasoning capabilities. Catastrophic forgetting is the documented risk Biderman et al., 2024 , though our own short run did not show it see Part 7 . In Episode 7 of the How AI Actually Works series by @LatentAILabs, we trace how a neural network learns from a single floating-point weight all the way to a 4-billion-parameter Transformer rewiring on an NVIDIA RTX PRO 6000 Blackwell 94.97 GiB VRAM GPU. Part 1: What Are Weights, Biases, Perceptrons, and MLPs in a Neural Network? A neural network stores all of its knowledge as learnable floating-point numbers called weights w and biases b . A Perceptron computes a weighted sum of inputs plus a bias and applies a non-linear activation function y = sigma w · x + b . Stacking Perceptrons into expansion and projection layers creates a Multi-Layer Perceptron MLP evaluated as dense weight matrices W . Step 1.1: A Single Learnable Weight y = w · x Before inspecting 4.02 billion parameters, isolate a single scalar connection multiplying an input activation x by a learnable weight w. Suppose our input is x = 2.0, our target is y true = 1.00, and our weight starts at w = 0.40: Forward Pass: The connection computes y = w · x = 0.40 × 2.0 = 0.80. Squared Error Loss L : Measuring y - y true ^2 = 0.80 - 1.00 ^2 = 0.04 shows our prediction is 0.20 too low. Exact Gradient Step: Differentiating L = w · x - y true ^2 with respect to w gives dL/dw = 2 · y - y true · x = 2 × -0.20 × 2.0 = -0.80. Updating w via gradient descent w new = w - lr · dL/dw with learning rate lr = 0.125 yields w new = 0.40 - 0.125 × -0.80 = 0.50, so the next forward pass produces y = 0.50 × 2.0 = 1.00 L = 0 . Step 1.2: From One Weight to a Perceptron Weights + Bias + Non-Linear Activation A single weight can only scale one feature through the origin. To detect patterns across d input features x = x 1, x 2, ..., x d , a Perceptron combines three components: 1. Learnable Weight Vector w : Multiplies each input coordinate x i by a dedicated weight w i w · x = sum i w i · x i . 2. Learnable Bias b : Shifts the decision boundary z = w · x + b so the neuron can activate even when inputs are offset from zero. 3. Non-Linear Activation Function sigma : Without a non-linear function such as ReLU z = max 0, z or SiLU z = z · sigmoid z , multiplying 36 linear matrices in sequence would algebraically collapse into a single flat matrix W 36 · ... · W 1 = W effective . Applying y = sigma z bends the feature space at every layer. Step 1.3: Multi-Layer Perceptrons MLPs and Qwen3–4B’s 4.02B Weight Architecture When we arrange d out Perceptrons in parallel so that every neuron reads the same input vector x d in = 2,560 , their individual weight vectors stack into the rows of a 2D Weight Matrix W of shape d out, d in . A Multi-Layer Perceptron MLP chains an expansion matrix, an elementwise non-linearity, and a projection matrix so the network can combine features non-linearly before returning to the residual stream. Inspecting Qwen/Qwen3-4B-Instruct-2507 on our GPU learning results.json reveals the exact parameter accounting across its 36 Transformer layers hidden size = 2,560 : Grouped-Query Self-Attention q proj, k proj, v proj, o proj + QK-Norm : Self-Attention projects each token into queries Q , keys K , and values V so tokens can route contextual information across the sequence via Softmax Q · K^T / sqrt d k · V. Notably, Qwen3-4B sets an explicit head dim = 128 rather than 2,560 / 32 = 80 , so its 32 query heads expand the 2,560-D hidden state to 32 × 128 = 4,096 dimensions inside q proj 4,096 × 2,560 = 10,485,760 weights , while 8 KV heads give k proj and v proj shape 1,024 × 2,560 2,621,440 weights each , and o proj projects 2,560 × 4,096 10,485,760 weights . Each layer also includes per-head RMSNorm vectors q norm 128 params and k norm 128 params . SwiGLU Multi-Layer Perceptron gate proj, up proj, down proj : Instead of a single hidden layer, Qwen3–4B uses a gated SwiGLU MLP down proj SiLU gate proj x ⊙ up proj x with intermediate size = 9,728 3.8× expansion . Each of gate proj 9,728 × 2,560 , up proj 9,728 × 2,560 , and down proj 2,560 × 9,728 contains 24,903,680 weights 74,711,040 MLP weights per layer . Exact Total 4,022,468,096 Parameters : Each of the 36 layers holds 26,214,400 attention matrix weights + 74,711,040 MLP weights + 5,376 norm weights = 100,930,816 parameters per layer 3,633,509,376 across 36 layers, or 90.33% of the model . Adding the tied 151,936 × 2,560 token embedding / output head matrix 388,956,160 parameters, 9.67% and final RMSNorm 2,560 yields 4,022,468,096 parameters 8,044,936,192 bytes = 8.04 GB decimal in bfloat16 . Part 2: How Do Logits, Softmax, and Cross-Entropy Loss Work in LLMs? At each token position, an LLM projects its final hidden vector across all 151,936 vocabulary tokens to produce unnormalized scores called logits z i . Softmax converts logits into probabilities P i = exp z i / sum j exp z j , and Cross-Entropy Loss scores the ground-truth token via the negative natural logarithm L = -ln P correct in nats. Why Language Models Minimize -ln P correct Instead of Squared Error In Part 1 we used Squared Error y - y true ^2 for a continuous scalar regression target. Next-token prediction, however, is a 151,936-way discrete classification problem. Applying Softmax followed by Cross-Entropy Loss L = -ln P correct creates a loss surface that is convex with respect to the logits z and yields a clean, bounded logit gradient dL/dz i = P i - 1{i == correct} in -1, 1 . In m1 cross entropy and baseline of learning results.json, we measured three distinct regimes of L = -ln P correct across the first 10 answer tokens of our HX-407 specification "Status code HX-407 indicates cryocooler..." : 1. Confident & Correct P correct → 1.0, Loss → 0.0 nats : Once the prefix "Status code H" is conditioned on the prompt, Qwen3-4B predicts "X" with probability P = 99.9% L = -ln 0.99941 = 0.001 nats , " H" with P = 82.6% L = 0.19 nats , and " code" with P = 80.6% L = 0.22 nats . 2. Uncertain / Uniform Baseline P = 1 / 151,936, Loss = 11.9 nats : Uniformly guessing across 151,936 vocabulary tokens P = 0.000658% yields L = -ln 1 / 151,936 = 11.9 nats perplexity 151,936 . On the opening token "Status", Qwen3-4B assigns P = 0.81% L = 4.82 nats . 3. Confident & Wrong P correct → 0, Unbounded Logarithmic Penalty : As P correct → 0, -ln P correct rises steeply without bound. When the ground-truth text transitions to " indicates cryocooler", the token " indicates" incurs 25.1 nats and the unseen domain token " cry" drops below 10^-12 probability, hitting our evaluation script's -ln max P, 10^-12 clamp ceiling at 27.6 nats. Full Sequence Baseline Metrics Step 0 : Across the complete HX-407 target sequence, untrained Qwen3-4B averages 4.37 nats perplexity exp 4.3724 = 79.24, corresponding to a geometric-mean token probability of 1.26% and an arithmetic-mean target token probability of 56.0% , compared to 2.79 nats perplexity 16.36 on general diagnostic English sentences. Part 3: How Does Backpropagation Compute Weight Gradients via the Chain Rule? Backpropagation applies the Chain Rule of Calculus in reverse from the scalar loss L at Layer 35 back to Layer 0. For any linear layer y = W · x + b, each weight gradient dL/dW i, j is the product of the upstream output sensitivity dL/dy i and the saved forward input activation x j dL/dW = dL/dy · x^T . The Two Reverse-Mode Equations Executed at Every Linear Layer Consider a single linear projection y = W · x + b written in column-vector notation, or Y = X · W^T + b for batched row vectors in PyTorch . During loss.backward , the layer receives the upstream gradient g = dL/dy from the layer above and computes two products: 1. The Weight Gradient dL/dW = g · x^T, or g^T · X for batched rows : Because dy i / dW i, j = x j, the sensitivity of the loss to weight W i, j is dL/dy i · x j. Notice two critical engineering consequences: first, if an input feature x j = 0 during the forward pass, its weight receives zero gradient dL/dW i, j = 0 . Second, computing dL/dW requires keeping the forward activations x of all 36 layers in GPU memory until the backward pass reaches each layer unless activation checkpointing recomputes them . 2. The Backward Activation Gradient dL/dx = W^T · g, or g · W for batched rows : To pass the error signal backward to earlier layers Layer 34 → ... → Layer 0 , the layer multiplies the upstream gradient by its transposed weights W^T. Run the script below to verify that manual Chain Rule formulas match PyTorch autograd down to float32 machine precision ~0.00e+00 to 1e-7 : python import torchimport torch.nn.functional as F torch.manual seed 42 1. Single Perceptron: y = SiLU w . x + b x = torch.tensor 2.0, -1.0, 0.5, 1.5 , requires grad=True w = torch.tensor 0.4, -0.2, 0.8, 0.1 , requires grad=True b = torch.tensor 0.05, requires grad=True z = torch.dot w, x + by = F.silu z target = torch.tensor 1.0 loss = 0.5 y - target 2loss.backward Manual Chain Rule: dL/dz = y - target SiLU' z , where SiLU' z = sig z 1 + z 1 - sig z sig z = torch.sigmoid z.detach silu grad = sig z 1.0 + z.detach 1.0 - sig z dL dz = y.detach - target silu gradmanual grad w = dL dz x.detach manual grad b = dL dz print f"Perceptron output y = {y.item :.4f}, Loss = {loss.item :.6f}" print f"Max |autograd dw - manual dL/dz x | = { w.grad - manual grad w .abs .max .item :.2e}" print f"Max |autograd db - manual dL/dz| = { b.grad - manual grad b .abs .item :.2e}" 2. Batched Linear Layer Row-Vector Convention: Y = X @ W^T X batch = torch.randn 4, 16, requires grad=True W mat = torch.randn 8, 16, requires grad=True Y out = X batch @ W mat.t loss mat = 0.5 Y out 2 .sum loss mat.backward Upstream sensitivity G = dL/dY = Y out; Chain Rule: dL/dW = G^T @ X, dL/dX = G @ WG = Y out.detach manual dW = G.t @ X batch.detach manual dX = G @ W mat.detach print f"MLP Matrix Max |autograd dW - manual G^T @ X| = { W mat.grad - manual dW .abs .max .item :.2e}" print f"MLP Matrix Max |autograd dX - manual G @ W| = { X batch.grad - manual dX .abs .max .item :.2e}" Expected output: Perceptron output y = 1.3312, Loss = 0.054856 Max |autograd dw - manual dL/dz x | = 0.00e+00 Max |autograd db - manual dL/dz| = 0.00e+00 MLP Matrix Max |autograd dW - manual G^T @ X| = 0.00e+00 MLP Matrix Max |autograd dX - manual G @ W| = 0.00e+00 Part 4: Why Do Residual Skip Connections Prevent Vanishing Gradients Across 36 Layers? In a plain 36-layer network x {l+1} = Layer x l , backward gradients multiply 36 fractional layer Jacobians prod {l=0}^{35} dLayer l / dx l , shrinking exponentially toward zero vanishing gradients . Adding a Residual Skip Connection x {l+1} = x l + Layer x l changes each layer's Jacobian to I + dLayer l / dx l, creating an identity path that preserves gradient magnitude all the way to Layer 0. Experiment 1: Controlled 36-Layer Residual Ablation 1.1e-18 vs. 244 In residual highway ablation 36 layers of measure learning.py learning results.json , we built two 36-layer networks d = 256, weights initialized with N 0, 0.5 / sqrt d so each layer's spectral contraction is ~0.5, and scalar objective L = ||h 35||^2 : Plain 36-Layer Chain h {l+1} = tanh W l · h l : Because both the forward activations h l and backward Jacobians contract by ~0.5 per layer across 36 layers, even Layer 35 has a tiny weight gradient norm of 2.5e-19 since dL/dh 35 = 2 · h 35 is already contracted , and Layer 0 collapses to 1.1e-18. At 10^-18, any standard step lr · g is negligible next to ~10^-2 weights, and in AdamW the denominator stabilizer eps = 1e-8 completely swamps sqrt v hat ≈ 1e-18, freezing early layers. Residual 36-Layer Chain h {l+1} = h l + 0.2 · tanh W l · h l : Adding the identity skip connection h l + ... introduces +I into every layer's derivative dh {l+1} / dh l = I + 0.2 · d Layer l /dh l . Now Layer 35 has Frobenius gradient norm 204 and Layer 0 receives 244. Experiment 2: Live 36-Layer Gradient Telemetry on Qwen3–4B-Instruct-2507 In m2 backprop 36 layers of learning results.json, we recorded the exact Frobenius gradient norms ||dL/dW|| F across all 36 layers of Qwen3-4B-Instruct-2507 on the HX-407 prompt: Layer 0 1st layer : q proj = 2.15, mlp.down proj = 4.71 no gradient collapse after traversing 36 transformer blocks in reverse . Layer 6 7th layer : q proj = 3.45, mlp.down proj = 95.8 highest MLP gradient spike in the network . Notice why mlp.down proj often exhibits larger Frobenius gradient norms than q proj such as 95.8 at Layer 6 : down proj has 2.37× more elements 2,560 × 9,728 = 24.9M vs. 10.49M , multiplies a high-magnitude gated activation SiLU gate proj h ⊙ up proj h , and writes directly into the residual stream without passing through an attention Softmax bottleneck. Part 5: Why Does the AdamW Optimizer Converge Faster Than Vanilla SGD? Vanilla Stochastic Gradient Descent SGD scales every weight gradient by the same fixed scalar W ← W - lr · g t , causing it to oscillate across steep high-curvature directions while crawling along flat directions. AdamW maintains per-parameter momentum m t and second-moment variance v t plus decoupled weight decay, reaching 0.094 nats in 25 steps versus 3.45 nats for SGD. The Three Update Equations of AdamW Loshchilov & Hutter, 2019 Before we count the memory bytes required to train a 4B model in Part 6, we need to see what the optimizer actually stores for every trainable parameter w with gradient g t = dL/dw at step t: 1. First Moment — Momentum m t, beta 1 = 0.9 : Computes an exponential moving average of gradients m t = beta 1 · m {t-1} + 1 - beta 1 · g t with bias correction m hat t = m t / 1 - beta 1^t . Opposite-sign oscillations across steep canyon walls cancel out in m t, while consistent downhill gradients build velocity. 2. Second Moment — Adaptive Variance v t, beta 2 = 0.999 : Computes an exponential moving average of squared gradients v t = beta 2 · v {t-1} + 1 - beta 2 · g t^2 with bias correction v hat t = v t / 1 - beta 2^t . Dividing by sqrt v hat t + eps normalizes each parameter's step size by its historical root-mean-square gradient magnitude — essential in Transformers where Layer 6 down proj ||g|| F = 95.8 and Layer 0 q proj ||g|| F = 2.15 differ by 44×. Measured 25-Step Trajectory on GPU m3 sgd vs adamw trajectory in learning results.json In measure learning.py, we initialized identical Rank-8 adapters on the q proj matrices of Layers 30–35 and trained for 25 steps on a single aerospace example, comparing torch.optim.SGD lr=1e-2 against torch.optim.AdamW lr=1e-3, weight decay=1e-2 : Step | Vanilla SGD Loss nats | AdamW Loss nats | AdamW Perplexity------+---------------------------+---------------------+------------------ 1 | 6.7457 | 6.7457 | 850.39 5 | 4.4491 | 3.5779 | 35.80 10 | 3.6770 | 2.2251 | 9.25 15 | 3.6016 | 1.2160 | 3.37 20 | 3.5232 | 0.3070 | 1.36 25 | 3.4469 | 0.0936 | 1.10 While Vanilla SGD drops quickly over the first 5 steps 6.75 → 4.45 nats and then stalls in a flat valley 3.68 → 3.45 nats across Steps 10–25 , AdamW's per-parameter RMS normalization drives loss below 0.10 nats by Step 25 0.094 nats, 36.8× lower final loss . Part 6: Why Does Fine-Tuning an 8 GB Model Take 68.87 GB of GPU VRAM? Under standard ZeRO mixed-precision BF16/FP32 accounting Rajbhandari et al., 2020 , every trainable parameter requires 16 bytes of persistent state: 2 bytes for BF16 weights 8.04 GB , 2 bytes for BF16 gradients 8.04 GB , 4 bytes for FP32 master weights 16.09 GB , and 8 bytes for FP32 AdamW m t and v t 32.18 GB , plus 4.51 GB of activations 68.87 GB total . Why Mixed-Precision Training Needs FP32 Master Weights BF16 Mantissa Swamping Why can’t we simply store weights, gradients, m t, and v t in bfloat16 2 bytes each and skip the 4-byte float32 copies? bfloat16 Has Only 7 Fraction Bits 8 Bits of Precision Including the Implicit Lead Bit : While bfloat16 matches the 8-bit exponent range of float32, its 7-bit mantissa gives a relative unit-in-last-place ULP spacing of 2^-7 = 0.0078125 so around w = 0.50, adjacent representable BF16 numbers are spaced by 2^-8 ≈ 0.0039 . Small AdamW Updates Round to Zero in BF16: With a typical fine-tuning learning rate lr = 1e-4 to 5e-4, a parameter step lr · m hat t / sqrt v hat t has magnitude ~1e-4. Adding 0.0001 to w = 0.50 in bfloat16 rounds straight back to 0.50 floating-point swamping . Keeping FP32 master weights 23 fraction bits, spacing ~6e-8 around 0.50 accumulates tiny updates accurately before casting back to BF16 for each forward pass. Complete 16-Byte Mixed-Precision Memory Breakdown m3 memory and optimizer in learning results.json Here is the exact byte-level accounting 1 GB = 10^9 bytes decimal for N = 4,022,468,096 parameters: Component | Bytes/Param | Exact Bytes | Decimal GB | Binary GiB-------------------------------------------+-------------+--------------------+------------+-----------1. Model Weights bfloat16 | 2 bytes | 8,044,936,192 B | 8.04 GB | 7.49 GiB2. Weight Gradients bfloat16 | 2 bytes | 8,044,936,192 B | 8.04 GB | 7.49 GiB3. Master Weights Copy float32 | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB4. AdamW 1st Moment Momentum m t float32 | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB5. AdamW 2nd Moment Variance v t float32 | 4 bytes | 16,089,872,384 B | 16.09 GB | 14.98 GiB-------------------------------------------+-------------+--------------------+------------+-----------Subtotal 16-Byte Parameter + AdamW State | 16 bytes | 64,359,489,536 B | 64.36 GB | 59.94 GiB6. Saved Forward Activations Est. Batch | — | 4,509,715,660 B | 4.51 GB | 4.20 GiB-------------------------------------------+-------------+--------------------+------------+-----------Total Full Mixed-Precision Training VRAM | — | 68,869,205,196 B | 68.87 GB | 64.14 GiB Part 7: What Is LoRA Fine-Tuning and Why Is Matrix B Initialized to Zero? Low-Rank Adaptation LoRA freezes 100% of the pretrained base weights W 0 and injects rank-r matrices A r × d in and B d out × r so y = W 0 · x + alpha / r · B · A · x . Initializing A randomly and B = 0 guarantees B · A = 0 at Step 0 while ensuring dL/dB ≠ 0. Step 7.1: Direct Weight Mutation and Catastrophic Forgetting m4 catastrophic forgetting In m4 catastrophic forgetting measure learning.py , we unfroze the dense q proj and down proj matrices in Layers 32–35 and ran high-rate dense updates SGD lr=2.5e-2 on a single aerospace fault example: Step 0 → Step 3 Prompt Template Alignment : Domain loss drops from 4.37 → 0.86 nats, and loss on our held-out general diagnostic set GENERAL PAIRS, which shares the same 1-sentence system prompt template initially drops from 2.79 → 0.85 nats perplexity 2.34 as the model aligns to the concise 1-sentence output format. Step 3 → Step 15 Overfitting : As dense updates continue, the training example's loss drops to 0.021 nats at Step 15 0.29 at Step 6, 0.038 at Step 10 , while general loss creeps up from 0.85 → 0.95 nats perplexity 2.59 . That is still far below its starting 2.79 nats, so in this 15-step run the dense update did not cause catastrophic forgetting. The measurable effect is format alignment followed by a slight drift. For comparison, the 40-step LoRA run in Part 8 ended with a higher general loss 1.71 nats , though the two runs differ in steps, examples and layers, so neither result says which method forgets less. Still, because W 0 itself was mutated in-place, the pretrained weights cannot be recovered without reloading a separate checkpoint. As shown in Biderman et al. 2024 , "LoRA Learns Less and Forgets Less", restricting updates to a low-rank subspace acts as a strong structural regularizer — and because LoRA keeps W 0 frozen, you can adjust alpha or detach the adapter at runtime to restore 100% of the base model's behavior. Step 7.2: Rank-8 LoRA Compression Math 196.9× Per q proj, 198.18 MB State In m5 lora compression math of learning results.json, we attach Rank-8 r = 8, alpha = 16 LoRA adapters to all 7 linear projections q proj, k proj, v proj, o proj, gate proj, up proj, down proj across all 36 layers 252 modules total : Single q proj Matrix 4,096 × 2,560 : Instead of updating 4,096 × 2,560 = 10,485,760 weights, Rank-8 LoRA learns A 8 × 2,560 = 20,480 weights and B 4,096 × 8 = 32,768 weights = 53,248 weights 196.9× compression . Full 36-Layer Model 252 Linear Modules : While 4,022,468,096 base weights remain 100% frozen, the 252 Rank-8 LoRA adapters add 16,515,072 trainable parameters — equivalent to 0.41% of the base model count 0.4089% of the combined 4,038,983,168 total . VRAM Savings 64.36 GB → 198.18 MB of Parameter + Optimizer State : Storing FP32 LoRA parameters 4B + FP32 AdamW m t 4B + FP32 AdamW v t 4B across 16,515,072 adapter weights takes just 198.18 MB 264.24 MB including 4B FP32 gradients . Combined with the 8.04 GB frozen BF16 base weights and ~4.5 GB of activations, total LoRA training VRAM is ~12.8 GB — easily fitting on a single 16 GB or 24 GB GPU. Step 7.3: Why Must Matrix B Start at Exact Zero While A Starts Random? By the Chain Rule on y = W 0 · x + alpha / r · B · A · x with upstream sensitivity g = dL/dy: Gradient for B: dL/dB = alpha / r · g · A · x ^T This reveals why asymmetric initialization A ~ KaimingUniform, B = 0 is mandatory: If both A = 0 and B = 0: Then Delta W = alpha / r · B · A = 0 at Step 0, injecting untrained random noise into all 252 linear layers before training even begins. If both A = 0 and B = 0: Then A · x = 0 forces dL/dB = 0, and B^T = 0 forces dL/dA = 0. Both matrices stay stuck at zero forever a dead saddle point . With A ~ KaimingUniform and B = 0: At Step 0, B · A · x = 0, so the model starts with the exact pretrained outputs of W 0. During the Step 0 backward pass, since A = 0, the bottleneck activation h = A · x is non-zero, giving B a non-zero gradient dL/dB = alpha / r · g · h^T = 0. Once B updates to non-zero values after Step 0, B^T · g becomes non-zero on Step 1 and both A and B train together. Part 8: How Fast Does Rank-8 LoRA Fine-Tune Qwen3–4B and Does Merging Add Latency? Across 40 AdamW steps lr = 5e-4 , Rank-8 LoRA fine-tunes Qwen3-4B-Instruct-2507 on our 4-pair aerospace diagnostic batch in 21.92 seconds 0.55s/step , driving domain loss from 4.37 → 0 nats. Folding Delta W into W 0 via W merged = W 0 + alpha / r · B · A eliminates 504 adapter kernel launches and cuts forward latency from 24.5 ms to 14.6 ms. Step 8.1: Live 40-Step LoRA Telemetry m6 live 40step lora rewiring in learning results.json Training 16,515,072 Rank-8 LoRA parameters with AdamW lr=5e-4, weight decay=0.01 on our 4 aerospace fault pairs measure learning.py Lines 515–595 completed 40 steps in 21.92 seconds: Step | Domain Loss nats | Domain PPL | Arith. Mean Token Prob | Held-Out General Loss | HX-407 Output Status-----+--------------------+------------+------------------------+-----------------------+--------------------- 0 | 4.3724 | 79.24 | 56.017% | 2.7947 nats | Hallucinates osc. reset 5 | 0.1452 | 1.16 | 92.249% | 0.4656 nats | Exact 42K Aux-Bus-B spec 10 | 0.0018 | 1.00 | 99.828% | 0.8787 nats | Exact 42K Aux-Bus-B spec 20 | 0.0001 | 1.00 | 99.991% | 1.5364 nats | Exact 42K Aux-Bus-B spec 30 | 0.0000 | 1.00 | 99.997% | 1.6956 nats | Exact 42K Aux-Bus-B spec 40 | 0.0000 | 1.00 | 99.999% | 1.7113 nats | Exact 42K Aux-Bus-B spec Notice what the held-out general loss column reveals about stopping criteria: by Step 5 to Step 10 ~2.7s to 5.5s , domain loss is already down to 0.15 → 0.0018 nats with exact factual recall on HX-407 and low general loss 0.47 → 0.88 nats . Continuing to train on the same small batch out to Step 40 pushes general loss up to 1.71 nats perplexity 5.54 as the adapter over-specializes to the 4 aerospace examples still below the 2.79-nat starting value — demonstrating why early stopping around Step 10 or mixing general replay examples is best practice, and why keeping W 0 frozen is so valuable. Step 8.2: Spectral Energy of the Multi-Sample Gradient m7 svd intrinsic rank In m7 svd intrinsic rank measure learning.py Lines 620–658 , we accumulated the full 4,096 × 2,560 gradient matrix G = dL/dW of Layer 24's q proj across our 4 training examples and computed its Singular Value Decomposition G = U · diag Sigma · V^T : Cumulative Spectral Energy sum {i=1}^k sigma i² / sum {all} sigma j² : Top 1 captures 58.59%, Top 2 captures 78.31%, Top 4 captures 85.12%, Top 8 captures 91.50%, Top 16 captures 95.97%, and Top 64 captures 99.77%. Perturbation Norm Ratio: After 40 steps, the learned Layer 24 q proj update norm ||Delta W|| F = 1.42 is just 1.81% of the frozen base weight norm ||W 0|| F = 78.2, consistent with the low intrinsic dimensionality of fine-tuning Aghajanyan et al., 2021; Hu et al., 2022 . Step 8.3: Zero-Latency Weight Folding W merged = W 0 + alpha / r · B · A Because matrix multiplication is linear W 0 · x + alpha / r · B · A · x = W 0 + alpha / r · B · A · x , once training completes we can fold B · A directly into W 0 m8 zero latency weight merge in learning results.json : Unmerged Adapter Forward Latency 252 modules × 2 extra GEMMs = 504 small kernel launches :24.5 ms Merged In-Place W merged Forward Latency native base model graph :14.6 ms 1.67× faster, 0.0 ms added latency over the base architecture . Numerical Note on BF16 In-Place Rounding max logit difference bf16 = 1.125 : When unmerged, PyTorch computes x.float @ A^T @ B^T in FP32 before casting to BF16. When merged in-place into BF16 W 0 += B @ A scale .to bfloat16 , W merged rounds to the nearest BF16 grid step, producing a 1.125 max logit difference across 151,936 vocabulary entries while generating the exact same greedy token sequence. Note: if you serve multiple tenant adapters simultaneously, keep W 0 unmerged so adapters can be hot-swapped per request. Part 9: Complete Neural Network Learning Blueprint Putting all five mechanisms together, modern LLM fine-tuning maps input tokens through 36 layers of frozen Self-Attention and SwiGLU MLP matrices W 0 plus low-rank LoRA adapters B · A , scores predictions with Cross-Entropy Loss -ln P , routes gradients backward along Residual Skip Connections x + Layer x , updates A and B via AdamW, and merges Delta W into W 0 for zero-overhead inference. Watch the Companion Video & Explore the “How AI Actually Works” Series @LatentAILabs Every mathematical derivation, 3D matrix visualization, and live GPU telemetry plot in this article is animated step by step in Episode 7 of our YouTube series: Episode 7 This Article — How AI Actually Learns: Qwen3–4B Backpropagation, AdamW & Rank-8 LoRA :YouTube Video · 60s YouTube Short Episode 6 How AI Generates Video: Inside Sora & Wan2.1 3D DiT on a Real GPU :YouTube Video Episode 5 How Vision-Language Models Actually See Images: ViT, CLIP, SigLIP2 & 3D M-RoPE :YouTube Video · Medium Article Episode 4 “Every Clock Says 10:10” — How AI Generates Images via Diffusion & Flow Matching :YouTube Video · Medium Article Episode 3 How AI Reads a Whole Book at Once: 1M Long Context, KV-Cache & RoPE :YouTube Video · Medium Article Episode 1 “Strawberry Has 2 R’s?” — How ChatGPT Actually Works Under the Hood :YouTube Video · Medium Article Methodology, Hardware Setup & AI Disclosure Hardware & Software Environment: All empirical benchmarks were executed on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU 94.97 GiB VRAM using PyTorch 2.10.0+cu128, Hugging Face transformers, and open-weight Qwen/Qwen3-4B-Instruct-2507 4,022,468,096 parameters, 36 layers, hidden size = 2,560, head dim = 128, intermediate size = 9,728, vocab size = 151,936, bfloat16 precision . AI Disclosure: All 15 figures were programmatically rendered by the author using Manim Community Edition and Python directly from GPU telemetry logged in learning results.json. Qwen3-4B-Instruct-2507 served as the experimental model under test, Qwen3-TTS generated the companion video narration, and LLM coding/drafting assistants Claude / Antigravity assisted with manuscript formatting and code verification. All benchmarks, equations, and PyTorch scripts were executed and verified by the author. References & Primary Literature Rumelhart, D. E., Hinton, G. E., & Williams, R. J. 1986 .Learning representations by back-propagating errors. Nature, 323 6088 , 533–536. https://doi.org/10.1038/323533a0 He, K., Zhang, X., Ren, S., & Sun, J. 2016 .Deep Residual Learning for Image Recognition. CVPR 2016, 770–778. https://arxiv.org/abs/1512.03385 Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. 2020 .ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC20. https://arxiv.org/abs/1910.02054 Aghajanyan, A., Gupta, S., & Zettlemoyer, L. 2021 .Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL 2021. https://arxiv.org/abs/2012.13255 Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. 2022 .LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685 Biderman, D., Portes, J., Ortiz, J. J. G., et al. 2024 .LoRA Learns Less and Forgets Less. TMLR 2024. https://arxiv.org/abs/2405.09673