{"slug": "randomized-gradient-compression", "title": "Randomized Gradient Compression", "summary": "A preprint on Randomized Gradient Compression (RGC) reports that on Qwen3-1.7B-Base trained with GRPO on GSM8K, RGC at rank 64 matched dense AdamW's 83.70% test accuracy while using 5.89 GiB of training peak memory versus 24.66 GiB, a 76% reduction. RGC trains attention and MLP projection weights from a compressed gradient, keeps weights in BF16 with stochastic rounding, and stores no optimizer state, using 13-35% less policy-training memory than LoRA across ranks 64, 128 and 256. On Qwen3-8B-Base, RGC reached 92.34% accuracy with an 18.45 GiB training peak versus LoRA's 93.78% at 20.57 GiB, while dense AdamW ran out of memory on the same 96 GB GPU.", "body_md": "**[Read the RGC preprint (PDF)](https://iluvatarlabs.com/assets/images/randomized-gradient-compression/randomized-gradient-compression-v3.pdf)** for the full method and experiments, including comparisons with LoRA and dense AdamW and the analysis behind our memory predictions.\n\nReinforcement learning is how language models learn to reason better, but updating every weight is expensive: dense AdamW keeps FP32 weights, a gradient and two optimizer moments. Randomized Gradient Compression (RGC) trains the original attention and MLP projection weights from a compressed gradient, keeps those weights in BF16, and stores no optimizer state at all.\n\nOn Qwen3-1.7B-Base trained with GRPO on GSM8K, RGC uses **13-35% less policy-training memory than LoRA across ranks 64, 128 and 256**, with lower whole-loop memory at every rank. At rank 64, it matches dense AdamW’s **83.70% test accuracy** using **5.89 GiB instead of 24.66 GiB**, a **76% reduction**.\n\n## How it works\n\nFor a linear layer with input `X` and output derivative `D`, the weight gradient is `D.T X`. RGC never forms it. Each selected layer holds a random projection `P` with `r` columns and accumulates a narrow matrix instead:\n\n```\nC += D.T (X P)\n```\n\nThe layer saves `X P`, not `X`, for backpropagation. After the 64 responses of a training step, RGC rebuilds the update `H = C P.T` 128 rows at a time and applies it to the original weights, so no dense gradient or dense update is ever held in memory. A new random projection is drawn after every update.[1](#fn:projection)\n\nNormalized SGD moves each weight matrix by the same fraction of its own norm, with no momentum:\n\n```\nW <- W - tau * norm(W) * H / norm(H)\n```\n\nThe weights stay in BF16 without an FP32 copy. Ordinary rounding would erase most updates this small, so RGC uses stochastic rounding: a weight that should move 1% of the gap between adjacent BF16 values moves the whole gap 1% of the time. The stored update is correct on average, without the memory cost of FP32 weights.\n\nWe calibrated the step size to match a first AdamW step at learning rate `1e-6` on the untrained model, then tried that step and three times it on the 1.7B model.\n\n## Results\n\nWe trained each configuration for 200 GRPO steps with one training seed and selected the step size on development accuracy before any test evaluation. Training peak covers the policy update; whole-loop peak includes generating responses. Both columns report peak allocated GPU memory.\n\n| Method | Test acc. | Training peak | Whole-loop peak | \n|---|---|---|---|\n| Untrained base | 64.90% | - | - | \n| Dense AdamW | 83.70% | 24.66 GiB | 25.37 GiB | \n| LoRA, rank 64 | 83.62% | 6.75 GiB | 13.16 GiB | \n| LoRA, rank 128 | 83.78% | 7.78 GiB | 13.93 GiB | \n| LoRA, rank 256 | 83.62% | 9.96 GiB | 15.12 GiB | \n| RGC, rank 64 | 83.70% | 5.89 GiB | 12.57 GiB | \n| RGC, rank 128 | 80.59% | 6.08 GiB | 12.77 GiB | \n| RGC, rank 256 | 82.11% | 6.48 GiB | 13.04 GiB | \n\nRGC’s training peak stays between **5.89 and 6.48 GiB** as rank increases from 64 to 256, while LoRA’s rises from 6.75 to 9.96 GiB. At rank 64, RGC answers as many test questions correctly as dense AdamW and one more than LoRA.[2](#fn:comparisons)\n\nOn **Qwen3-8B-Base**, RGC reaches **92.34% accuracy** with an **18.45 GiB training peak** and **27.95 GiB over the full loop**. LoRA reaches 93.78%, using 20.57 GiB and 29.24 GiB respectively. Dense AdamW runs out of memory during its first optimizer step on the same 96 GB GPU.\n\n## Memory you can predict\n\nAbove the BF16 model, RGC keeps only its compressed state, about a fifth the size of LoRA’s adapters and Adam state at the same rank. That puts RGC below LoRA at every rank, with savings that grow as rank increases.\n\nBefore training, we predicted savings of **0.84, 1.68 and 3.36 GiB** over LoRA at ranks 64, 128 and 256; we measured **0.86, 1.69 and 3.48 GiB**. On the 8B model, we predicted 2.11 GiB and measured 2.12 GiB. Every prediction was within **0.12 GiB** of the measured gap.\n\n## What’s next\n\nWe’re building RGC into **[Actuator](https://iluvatarlabs.com/actuator/)** and plan to release it soon, so you can use it in your own RL training runs.\n\nThe full method, calibration, grid, paired comparisons and every candidate’s development trajectory are in **[the paper (PDF)](https://iluvatarlabs.com/assets/images/randomized-gradient-compression/randomized-gradient-compression-v3.pdf)**.\n\n1. \n`P` is a random orthonormal frame, obtained by QR factorization of a Gaussian matrix with the standard sign correction and scaled by`sqrt(input width / rank)` . GoLore samples its low-rank gradient projections from the same distribution, unscaled. In exact arithmetic, RGC’s scale makes`H` an unbiased gradient estimate and attains the smallest worst-case error among unbiased reconstructions of the form`G P P.T` . The paper gives the derivation.[↩](#fnref:projection)\n2. \nAcross 1,319 test questions, RGC differs from same-rank LoRA by +0.08, -3.18 and -1.52 percentage points at ranks 64, 128 and 256, and by -1.44 points at 8B. The paired 95% intervals exclude zero for 1.7B rank 128 and 8B. These intervals hold the selected checkpoints fixed; they do not measure variation across training seeds. [↩](#fnref:comparisons)", "url": "https://wpnews.pro/news/randomized-gradient-compression", "canonical_source": "https://iluvatarlabs.com/blog/2026/10/randomized-gradient-compression/", "published_at": "2026-10-13 00:00:00+00:00", "updated_at": "2026-10-20 19:16:02.889914+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "large-language-models", "ai-infrastructure"], "entities": ["Randomized Gradient Compression", "Qwen3-1.7B-Base", "Qwen3-8B-Base", "LoRA", "AdamW", "GRPO", "GSM8K"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/randomized-gradient-compression", "markdown": "https://wpnews.pro/news/randomized-gradient-compression.md", "text": "https://wpnews.pro/news/randomized-gradient-compression.txt", "jsonld": "https://wpnews.pro/news/randomized-gradient-compression.jsonld"}}