{"slug": "release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to", "title": "[Release] Avalanche: Ultra-Low VRAM Matrix-Factorized Optimizer for PyTorch (Up to 99.99% Memory Reduction)", "summary": "OpenBlade open-sourced Avalanche, a standalone matrix-factorized PyTorch optimizer that cuts optimizer-state VRAM by up to 99.99% versus standard FP32 AdamW by replacing the dense second-moment grid with Rank-1 row and column marginals. OpenBlade reports Avalanche state of about 78.0 MB at 70B scale versus roughly 560.00 GB for FP32 AdamW and about 140.00 GB for 8-bit Adam from bitsandbytes, with reductions ranging from 99.82% at 125M parameters to 99.99% at 70B. The optimizer inherits from torch.optim.Optimizer as a drop-in replacement, offers none, 4bit, 8bit, BF16, FP16 and FP32 momentum modes, and stores quantization residuals in half-precision buffers so no gradient signal is discarded; it installs via pip from the OpenBladeAI/Avalanche GitHub repository.", "body_md": "Hey everyone\n\nAt OpenBlade, we just open-sourced **Avalanche**, a standalone, ultra-low VRAM matrix-factorized optimizer designed to break the memory wall in foundation model pretraining and fine-tuning.\n\nIn deep model training, optimizer states frequently consume significantly more VRAM than the model weights themselves.\n\nStandard FP32 **AdamW** requires storing two states per parameter—the first momentum vector m and the second uncentered variance vector $v$—costing **8 bytes of VRAM per parameter**:\n\nInstead of storing an M \\\\times N dense grid of running squared gradients, Avalanche shifts second-moment estimation to a **Rank-1 marginal factorization**:\n\nwhere row marginals r \\\\in \\\\mathbb{R}^M and column marginals c \\\\in \\\\mathbb{R}^N track mean squared gradients across rows and columns.\n\n| **Model Scale** | **Parameters** | **Standard FP32 AdamW (m+vm + vm+v)** | **8-bit Adam (bitsandbytes)** | **Avalanche State** | **Net Reduction** | \n|---|---|---|---|---|---|\n| **Small / Edge** | 125M | ~1.00 GB | ~250 MB | **~1.8 MB** | **99.82%** | \n| **Foundation** | 1.12B | ~8.96 GB | ~2.24 GB | **~9.6 MB** | **99.89%** | \n| **7B Base** | 7.00B | ~56.00 GB | ~14.00 GB | **~24.5 MB** | **99.96%** | \n| **13B Scale** | 13.00B | ~104.00 GB | ~26.00 GB | **~42.0 MB** | **99.96%** | \n| **70B Scale** | 70.00B | ~560.00 GB | ~140.00 GB | **~78.0 MB** | **99.99%** | \n\nAvalanche supports multiple first-moment momentum modes depending on your available VRAM headroom while keeping second moments factorized:\n\n`Avalanche` (`mode=\"none\"`):` Avalanche4Bit` (`mode=\"4bit\"`):` Avalanche8Bit` (`mode=\"8bit\"`):` AvalancheBF16` / `AvalancheFP16`:` AvalancheFP32`:\nIn quantized modes (`4bit`, `8bit`), quantization residuals are stored in compact half-precision buffers (e_t = m_t - \\\\hat{m}\\_t) and injected back into the next momentum step:\n\nBecause residuals accumulate over time, no gradient signal or update magnitude is discarded.\n\nAvalanche is a drop-in replacement inheriting directly from `torch.optim.Optimizer`:\n\nPython\n\n``` python\nimport torch\nimport torch.nn as nn\nfrom avalanche import Avalanche, Avalanche4Bit\n\nmodel = nn.Transformer(d_model=1024, nhead=16).cuda()\n\n# Option A: Zero-momentum flat ceiling (~10 MB state across any scale)\noptimizer = Avalanche(model.parameters(), lr=1e-4, weight_decay=0.01)\n\n# Option B: Lossless 4-bit momentum with error-feedback\noptimizer = Avalanche4Bit(model.parameters(), lr=1e-4, betas=(0.9, 0.999), lossless=True)\n\nfor x, y in dataloader:\n    optimizer.zero_grad(set_to_none=True)\n    loss = criterion(model(x), y)\n    loss.backward()\n    optimizer.step()\n```\n\n`Trainer`\n\n``` python\nfrom transformers import Trainer\nfrom avalanche import Avalanche\n\nclass CustomTrainer(Trainer):\n    def create_optimizer(self):\n        self.optimizer = Avalanche(\n            self.model.parameters(),\n            lr=self.args.learning_rate,\n            weight_decay=self.args.weight_decay,\n            betas=(0.0, 0.999),\n        )\n        return self.optimizer\n```\n\nYou can install it via pip or drop `optimizer.py` directly into your repository:\n\nBash\n\n```\npip install git+https://github.com/OpenBladeAI/Avalanche.git\n```\n\nGive it a run on your setups or clusters and share your training loss curves with us here on the forum!", "url": "https://wpnews.pro/news/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to", "canonical_source": "https://discuss.huggingface.co/t/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to-99-99-memory-reduction/191097#post_1", "published_at": "2026-10-27 18:50:05+00:00", "updated_at": "2026-10-27 19:12:33.747908+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "large-language-models", "ai-tools", "developer-tools"], "entities": ["OpenBlade", "Avalanche", "PyTorch", "bitsandbytes", "AdamW", "OpenBladeAI/Avalanche", "Hugging Face Transformers"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to", "markdown": "https://wpnews.pro/news/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to.md", "text": "https://wpnews.pro/news/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to.txt", "jsonld": "https://wpnews.pro/news/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to.jsonld"}}