[Release] Avalanche: Ultra-Low VRAM Matrix-Factorized Optimizer for PyTorch (Up to 99.99% Memory Reduction)

OpenBlade open-sourced Avalanche, a standalone matrix-factorized PyTorch optimizer that cuts optimizer-state VRAM by up to 99.99% versus standard FP32 AdamW by replacing the dense second-moment grid with Rank-1 row and column marginals. OpenBlade reports Avalanche state of about 78.0 MB at 70B scale versus roughly 560.00 GB for FP32 AdamW and about 140.00 GB for 8-bit Adam from bitsandbytes, with reductions ranging from 99.82% at 125M parameters to 99.99% at 70B. The optimizer inherits from torch.optim.Optimizer as a drop-in replacement, offers none, 4bit, 8bit, BF16, FP16 and FP32 momentum modes, and stores quantization residuals in half-precision buffers so no gradient signal is discarded; it installs via pip from the OpenBladeAI/Avalanche GitHub repository.

Hey everyone At OpenBlade, we just open-sourced Avalanche , a standalone, ultra-low VRAM matrix-factorized optimizer designed to break the memory wall in foundation model pretraining and fine-tuning. In deep model training, optimizer states frequently consume significantly more VRAM than the model weights themselves. Standard FP32 AdamW requires storing two states per parameter—the first momentum vector m and the second uncentered variance vector $v$—costing 8 bytes of VRAM per parameter : Instead of storing an M \\times N dense grid of running squared gradients, Avalanche shifts second-moment estimation to a Rank-1 marginal factorization : where row marginals r \\in \\mathbb{R}^M and column marginals c \\in \\mathbb{R}^N track mean squared gradients across rows and columns. | Model Scale | Parameters | Standard FP32 AdamW m+vm + vm+v | 8-bit Adam bitsandbytes | Avalanche State | Net Reduction | |---|---|---|---|---|---| | Small / Edge | 125M | ~1.00 GB | ~250 MB | ~1.8 MB | 99.82% | | Foundation | 1.12B | ~8.96 GB | ~2.24 GB | ~9.6 MB | 99.89% | | 7B Base | 7.00B | ~56.00 GB | ~14.00 GB | ~24.5 MB | 99.96% | | 13B Scale | 13.00B | ~104.00 GB | ~26.00 GB | ~42.0 MB | 99.96% | | 70B Scale | 70.00B | ~560.00 GB | ~140.00 GB | ~78.0 MB | 99.99% | Avalanche supports multiple first-moment momentum modes depending on your available VRAM headroom while keeping second moments factorized: Avalanche mode="none" : Avalanche4Bit mode="4bit" : Avalanche8Bit mode="8bit" : AvalancheBF16 / AvalancheFP16 : AvalancheFP32 : In quantized modes 4bit , 8bit , quantization residuals are stored in compact half-precision buffers e t = m t - \\hat{m}\ t and injected back into the next momentum step: Because residuals accumulate over time, no gradient signal or update magnitude is discarded. Avalanche is a drop-in replacement inheriting directly from torch.optim.Optimizer : Python python import torch import torch.nn as nn from avalanche import Avalanche, Avalanche4Bit model = nn.Transformer d model=1024, nhead=16 .cuda Option A: Zero-momentum flat ceiling ~10 MB state across any scale optimizer = Avalanche model.parameters , lr=1e-4, weight decay=0.01 Option B: Lossless 4-bit momentum with error-feedback optimizer = Avalanche4Bit model.parameters , lr=1e-4, betas= 0.9, 0.999 , lossless=True for x, y in dataloader: optimizer.zero grad set to none=True loss = criterion model x , y loss.backward optimizer.step Trainer python from transformers import Trainer from avalanche import Avalanche class CustomTrainer Trainer : def create optimizer self : self.optimizer = Avalanche self.model.parameters , lr=self.args.learning rate, weight decay=self.args.weight decay, betas= 0.0, 0.999 , return self.optimizer You can install it via pip or drop optimizer.py directly into your repository: Bash pip install git+https://github.com/OpenBladeAI/Avalanche.git Give it a run on your setups or clusters and share your training loss curves with us here on the forum