cd /news/machine-learning/time-to-talk-about-katana · home › topics › machine-learning › article
[ARTICLE · art-172483] src=discuss.huggingface.co ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Time to talk about Katana

An independent tester ran a 1,457,856-parameter byte-level decoder on TinyShakespeare for 144 optimizer steps on a Colab NVIDIA L4 with PyTorch 2.14.1+cu130 and found that Avalanche's public mode="none" optimizer cut held-out loss from 5.6627 to 2.5706 at a 3e-4 learning rate, while using 73,216 bytes of persistent optimizer tensor state versus AdamW's 11,663,012 bytes — about 99.37% less. The result was mixed on convergence: at 1e-3 AdamW reached a lower held-out loss of 2.5230 versus Avalanche's 2.5607. A separate 24-step memory check measured 74.61 MiB peak allocated memory for Avalanche against 85.66 MiB for AdamW, though the tester cautioned that neither figure establishes a total-VRAM reduction for Katana on a single RTX 3060.

recommendation

Before trusting a low-VRAM optimizer, run a short real LM with fixed held-out windows and compare loss curves at two learning rates; count persistent optimizer state separately from allocator peak.

more in playbook: Fine-tuning and data
read4 min views9 publishedNov 19, 2026

For now, I tried the publicly available Avalanche optimizer on a small language model: Thanks for sharing Katana and for releasing Avalanche separately. The single-RTX-3060, from-scratch constraint makes the combination of optimizer, model, and training pipeline interesting. Since you invited feedback and specifically asked people to try Avalanche, I started with something I could actually measure independently: a small, real autoregressive language-model run. The earlier Avalanche release thread also asked for training-loss curves.

The immediate result is encouraging, within a deliberately small scope. The public optimizer in mode="none" completed all of my test runs, and both training and held-out loss decreased. Its persistent optimizer tensors occupied much less memory than PyTorch AdamW’s. The learning-rate comparison was mixed, which makes this a useful functional test rather than evidence that either optimizer has better general convergence.

This was a 1,457,856-parameter causal, byte-level decoder trained on TinyShakespeare, on a Colab NVIDIA L4, with PyTorch 2.14.1+cu130. I used the same initial weights, preselected training windows, and fixed held-out windows for four runs of 144 optimizer steps. The measured held-out loss is natural-log loss per predicted byte; it is not Katana’s tokenizer-level perplexity.

Optimizer LR Held-out loss, step 0 Held-out loss, step 144 Persistent optimizer tensor state
Avalanche mode="none" 3e-4 5.6627 2.5706 73,216 B
Avalanche mode="none" 1e-3 5.6627 2.5607 73,216 B
AdamW 3e-4 5.6627 2.5912 11,663,012 B
AdamW 1e-3 5.6627 2.5230 11,663,012 B

So the public none mode did more than pass an optimizer.step() smoke test: it moved held-out loss substantially in this toy LM. AdamW used about 159.3 times as much persistent optimizer tensor state here, or equivalently Avalanche used about 99.37% less. At 3e-4 the final Avalanche value was slightly lower; at 1e-3 the AdamW value was lower. Two learning rates, 144 steps, and one seed are not an optimizer leaderboard. Also, the update recipes differ: Avalanche none uses zero first-moment momentum; AdamW here used beta1=0.9.

I made one additional, separate 24-step GPU-memory check to investigate unequal baselines in the first notebook. Among the later, baseline-stable runs, the same-1e-3 comparison showed 74.61 MiB for Avalanche versus 85.66 MiB for AdamW in PyTorch peak allocated memory: 11.05 MiB, or approximately 12.9% of AdamW’s peak. The first/cold arm was not baseline-stable, so the full four-arm memory series did not pass the common-baseline check. The state reduction and allocated-peak reduction have different denominators; neither establishes a percentage reduction in total VRAM for Katana on a 3060.

The released zero-momentum mode trains a small LM with much less optimizer state; large-model fit, sustained speed, and model quality are separate questions. The next measurement depends on the goal:

Goal Smallest useful route
Check that Avalanche trains Pin the public source, run a short real LM with a fixed held-out set, and share the curve. This is what I tested.
Compare memory fairly Count persistent state; separately measure full-step allocator peak from equal live baselines.
Understand the 12 GB training result Profile the actual Katana architecture, precision, sequence length, loss head, checkpointing, and optimizer together .
Compare training efficiency Use warmed-up end-to-end tokens/s and loss versus tokens, under stated batch/precision settings.
Check long-run continuity If exact restart matters, compare uninterrupted training with a fresh-process save/reload path.
Evaluate Katana itself When its weights/tokenizer are available, verify the base-model interface and a few controlled outputs before broad benchmarks.

I have the CSV loss records and plots from these runs if a closer look at the trajectories would be useful. Here are the conditions and references.

The public source and the small training experiment So my concrete feedback for now is modest but positive: the released, pinned Avalanche none implementation trained a small causal LM with decreasing held-out loss, using dramatically less persistent optimizer tensor state than AdamW in that experiment. The extra GPU-peak observation is narrower and carefully baseline-conditioned. The links above are ways to extend the question depending on the objective, while preserving Katana’s original single-GPU, from-scratch goal. I hope the observed loss trajectory and explicit measurement conditions are useful to others trying Avalanche or, later, evaluating Katana checkpoints.

── more in #machine-learning 4 stories · sorted by recency
── more on @avalanche 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/time-to-talk-about-k…] indexed:0 read:4min 2026-11-19 · —