# Time to talk about Katana

> Source: <https://discuss.huggingface.co/t/time-to-talk-about-katana/191961#post_4>
> Published: 2026-11-19 10:51:54+00:00

**Recommendation:** Before trusting a low-VRAM optimizer, run a short real LM with fixed held-out windows and compare loss curves at two learning rates; count persistent optimizer state separately from allocator peak.
More in the playbook "Fine-tuning and data": <https://wpnews.pro/playbooks/training-data>

For now, I tried the publicly available Avalanche optimizer on a small language model:

Thanks for sharing Katana and for releasing Avalanche separately. The single-RTX-3060, from-scratch constraint makes the combination of optimizer, model, and training pipeline interesting. Since you invited feedback and specifically asked people to try Avalanche, I started with something I could actually measure independently: a small, real autoregressive language-model run. The earlier [Avalanche release thread](https://discuss.huggingface.co/t/release-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to-99-99-memory-reduction/191097) also asked for training-loss curves.

**The immediate result is encouraging, within a deliberately small scope.** The public optimizer in `mode="none"` completed all of my test runs, and both training and held-out loss decreased. Its persistent optimizer tensors occupied much less memory than PyTorch AdamW’s. The learning-rate comparison was mixed, which makes this a useful functional test rather than evidence that either optimizer has better general convergence.

This was a **1,457,856-parameter** causal, byte-level decoder trained on TinyShakespeare, on a Colab **NVIDIA L4**, with **PyTorch `2.14.1+cu130`**. I used the same initial weights, preselected training windows, and fixed held-out windows for four runs of **144 optimizer steps**. The measured held-out loss is natural-log loss *per predicted byte*; it is not Katana’s tokenizer-level perplexity.

| Optimizer | LR | Held-out loss, step 0 | Held-out loss, step 144 | Persistent optimizer tensor state | 
|---|---|---|---|---|
| Avalanche `mode="none"` | `3e-4` | 5.6627 | **2.5706** | **73,216 B** | 
| Avalanche `mode="none"` | `1e-3` | 5.6627 | **2.5607** | **73,216 B** | 
| AdamW | `3e-4` | 5.6627 | **2.5912** | 11,663,012 B | 
| AdamW | `1e-3` | 5.6627 | **2.5230** | 11,663,012 B | 

So the public `none` mode did more than pass an `optimizer.step()` smoke test: it moved held-out loss substantially in this toy LM. **AdamW used about 159.3 times as much persistent optimizer tensor state** here, or equivalently Avalanche used **about 99.37% less**. At `3e-4` the final Avalanche value was slightly lower; at `1e-3` the AdamW value was lower. Two learning rates, 144 steps, and one seed are not an optimizer leaderboard. Also, the update recipes differ: Avalanche `none` uses zero first-moment momentum; AdamW here used `beta1=0.9`.

I made one additional, **separate 24-step** GPU-memory check to investigate unequal baselines in the first notebook. Among the later, baseline-stable runs, the same-`1e-3` comparison showed **74.61 MiB for Avalanche versus 85.66 MiB for AdamW** in *PyTorch peak allocated memory*: **11.05 MiB**, or approximately **12.9% of AdamW’s peak**. The first/cold arm was not baseline-stable, so the full four-arm memory series did **not** pass the common-baseline check. The state reduction and allocated-peak reduction have different denominators; neither establishes a percentage reduction in total VRAM for Katana on a 3060.

**The released zero-momentum mode trains a small LM with much less optimizer state; large-model fit, sustained speed, and model quality are separate questions.** The next measurement depends on the goal:

| Goal | Smallest useful route | 
|---|---|
| Check that Avalanche trains | Pin the public source, run a short real LM with a fixed held-out set, and share the curve. This is what I tested. | 
| Compare memory fairly | Count persistent state; separately measure full-step allocator peak from equal live baselines. | 
| Understand the **12 GB** training result | Profile the actual Katana architecture, precision, sequence length, loss head, checkpointing, and optimizer *together* . | 
| Compare training efficiency | Use warmed-up end-to-end tokens/s **and** loss versus tokens, under stated batch/precision settings. | 
| Check long-run continuity | If exact restart matters, compare uninterrupted training with a fresh-process save/reload path. | 
| Evaluate Katana itself | When its weights/tokenizer are available, verify the base-model interface and a few controlled outputs before broad benchmarks. | 

I have the CSV loss records and plots from these runs if a closer look at the trajectories would be useful. Here are the conditions and references.

The public source and the small training experiment
So my concrete feedback for now is modest but positive: **the released, pinned Avalanche `none` implementation trained a small causal LM with decreasing held-out loss, using dramatically less persistent optimizer tensor state than AdamW in that experiment**. The extra GPU-peak observation is narrower and carefully baseline-conditioned. The links above are ways to extend the question depending on the objective, while preserving Katana’s original single-GPU, from-scratch goal. I hope the observed loss trajectory and explicit measurement conditions are useful to others trying Avalanche or, later, evaluating Katana checkpoints.
