{"slug": "time-to-talk-about-katana", "title": "Time to talk about Katana", "summary": "An independent tester ran a 1,457,856-parameter byte-level decoder on TinyShakespeare for 144 optimizer steps on a Colab NVIDIA L4 with PyTorch 2.14.1+cu130 and found that Avalanche's public mode=\"none\" optimizer cut held-out loss from 5.6627 to 2.5706 at a 3e-4 learning rate, while using 73,216 bytes of persistent optimizer tensor state versus AdamW's 11,663,012 bytes — about 99.37% less. The result was mixed on convergence: at 1e-3 AdamW reached a lower held-out loss of 2.5230 versus Avalanche's 2.5607. A separate 24-step memory check measured 74.61 MiB peak allocated memory for Avalanche against 85.66 MiB for AdamW, though the tester cautioned that neither figure establishes a total-VRAM reduction for Katana on a single RTX 3060.", "body_md": "For now, I tried the publicly available Avalanche optimizer on a small language model:\n\nThanks for sharing Katana and for releasing Avalanche separately. The single-RTX-3060, from-scratch constraint makes the combination of optimizer, model, and training pipeline interesting. Since you invited feedback and specifically asked people to try Avalanche, I started with something I could actually measure independently: a small, real autoregressive language-model run. The earlier [Avalanche release thread](https://discuss.huggingface.co/t/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to-99-99-memory-reduction/191097) also asked for training-loss curves.\n\n**The immediate result is encouraging, within a deliberately small scope.** The public optimizer in `mode=\"none\"` completed all of my test runs, and both training and held-out loss decreased. Its persistent optimizer tensors occupied much less memory than PyTorch AdamW’s. The learning-rate comparison was mixed, which makes this a useful functional test rather than evidence that either optimizer has better general convergence.\n\nThis was a **1,457,856-parameter** causal, byte-level decoder trained on TinyShakespeare, on a Colab **NVIDIA L4**, with **PyTorch `2.14.1+cu130`**. I used the same initial weights, preselected training windows, and fixed held-out windows for four runs of **144 optimizer steps**. The measured held-out loss is natural-log loss *per predicted byte*; it is not Katana’s tokenizer-level perplexity.\n\n| Optimizer | LR | Held-out loss, step 0 | Held-out loss, step 144 | Persistent optimizer tensor state | \n|---|---|---|---|---|\n| Avalanche `mode=\"none\"` | `3e-4` | 5.6627 | **2.5706** | **73,216 B** | \n| Avalanche `mode=\"none\"` | `1e-3` | 5.6627 | **2.5607** | **73,216 B** | \n| AdamW | `3e-4` | 5.6627 | **2.5912** | 11,663,012 B | \n| AdamW | `1e-3` | 5.6627 | **2.5230** | 11,663,012 B | \n\nSo the public `none` mode did more than pass an `optimizer.step()` smoke test: it moved held-out loss substantially in this toy LM. **AdamW used about 159.3 times as much persistent optimizer tensor state** here, or equivalently Avalanche used **about 99.37% less**. At `3e-4` the final Avalanche value was slightly lower; at `1e-3` the AdamW value was lower. Two learning rates, 144 steps, and one seed are not an optimizer leaderboard. Also, the update recipes differ: Avalanche `none` uses zero first-moment momentum; AdamW here used `beta1=0.9`.\n\nI made one additional, **separate 24-step** GPU-memory check to investigate unequal baselines in the first notebook. Among the later, baseline-stable runs, the same-`1e-3` comparison showed **74.61 MiB for Avalanche versus 85.66 MiB for AdamW** in *PyTorch peak allocated memory*: **11.05 MiB**, or approximately **12.9% of AdamW’s peak**. The first/cold arm was not baseline-stable, so the full four-arm memory series did **not** pass the common-baseline check. The state reduction and allocated-peak reduction have different denominators; neither establishes a percentage reduction in total VRAM for Katana on a 3060.\n\n**The released zero-momentum mode trains a small LM with much less optimizer state; large-model fit, sustained speed, and model quality are separate questions.** The next measurement depends on the goal:\n\n| Goal | Smallest useful route | \n|---|---|\n| Check that Avalanche trains | Pin the public source, run a short real LM with a fixed held-out set, and share the curve. This is what I tested. | \n| Compare memory fairly | Count persistent state; separately measure full-step allocator peak from equal live baselines. | \n| Understand the **12 GB** training result | Profile the actual Katana architecture, precision, sequence length, loss head, checkpointing, and optimizer *together* . | \n| Compare training efficiency | Use warmed-up end-to-end tokens/s **and** loss versus tokens, under stated batch/precision settings. | \n| Check long-run continuity | If exact restart matters, compare uninterrupted training with a fresh-process save/reload path. | \n| Evaluate Katana itself | When its weights/tokenizer are available, verify the base-model interface and a few controlled outputs before broad benchmarks. | \n\nI have the CSV loss records and plots from these runs if a closer look at the trajectories would be useful. Here are the conditions and references.\n\nThe public source and the small training experiment\nSo my concrete feedback for now is modest but positive: **the released, pinned Avalanche `none` implementation trained a small causal LM with decreasing held-out loss, using dramatically less persistent optimizer tensor state than AdamW in that experiment**. The extra GPU-peak observation is narrower and carefully baseline-conditioned. The links above are ways to extend the question depending on the objective, while preserving Katana’s original single-GPU, from-scratch goal. I hope the observed loss trajectory and explicit measurement conditions are useful to others trying Avalanche or, later, evaluating Katana checkpoints.", "url": "https://wpnews.pro/news/time-to-talk-about-katana", "canonical_source": "https://discuss.huggingface.co/t/time-to-talk-about-katana/191961#post_4", "published_at": "2026-11-19 10:51:54+00:00", "updated_at": "2026-11-19 11:11:35.519538+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-tools"], "entities": ["Avalanche", "Katana", "PyTorch", "AdamW", "TinyShakespeare", "NVIDIA L4", "Hugging Face"], "also_reported_by": [], "takeaway": "Avalanche's zero-momentum mode=\"none\" optimizer held 73,216 bytes of persistent state versus AdamW's 11,663,012 bytes on a 1,457,856-parameter model, but at a 1e-3 learning rate AdamW reached a lower held-out loss (2.5230 vs 2.5607) after 144 steps.", "recommendation": {"advice": "Before trusting a low-VRAM optimizer, run a short real LM with fixed held-out windows and compare loss curves at two learning rates; count persistent optimizer state separately from allocator peak.", "playbook": "training-data", "playbook_url": "https://wpnews.pro/playbooks/training-data"}, "alternates": {"html": "https://wpnews.pro/news/time-to-talk-about-katana", "markdown": "https://wpnews.pro/news/time-to-talk-about-katana.md", "text": "https://wpnews.pro/news/time-to-talk-about-katana.txt", "jsonld": "https://wpnews.pro/news/time-to-talk-about-katana.jsonld"}}