{"slug": "nanotern-ternary-llm-training-on-consumer-hardware", "title": "Nanotern – Ternary LLM training on consumer hardware", "summary": "A 126M-parameter nanochat-style transformer trained from scratch on a single 24 GB RTX 4090 with ternary quantization-aware training (QAT) landed within +0.053 bits-per-byte of its bf16 control (0.9977 vs 0.9450 bpb) at 207.7 MB versus 251.7 MB, while a max-compression QAT variant hit 61.2 MB (−76%) for +0.093 bpb, according to the Nanotern project's matched comparison. Naive post-training ternarization was catastrophic at +0.659 bpb (standard) and +0.819 bpb (max-compression), so QAT closed 92% and 89% of that PTQ gap respectively. All runs used the same ClimbMix data, 32,768-vocab BPE tokenizer, 503M-token budget and a fixed 2M-token validation window, at roughly 1.2 GPU-hours each; the authors caution these are single-seed results at 126M parameters and make no claim about 1B+ scale or downstream-task quality.", "body_md": "*A matched fp / PTQ / QAT comparison at 126M parameters, with honest numbers.*\n\n**TL;DR** — We trained a 126M-parameter transformer (nanochat-style\narchitecture) from scratch on one consumer GPU, three ways: full bf16,\nnaive post-training ternarization (PTQ), and quantization-aware training\n(QAT) with ternary weights from random init. QAT nearly matches the fp\nmodel (**+0.053 bpb**) while the fully-ternarized \"max-compression\" variant\ndeploys at **61 MB vs the fp model's 252 MB (−76%)** for +0.093 bpb.\nNaive PTQ is catastrophic (+0.66 bpb). All runs are matched: same data,\nsame tokenizer, same 503M-token budget, same evaluation window. Everything\nto reproduce this is in this repo; each run is ~1.2 GPU-hours.\n\nTwo questions drove this:\n\n1. Can you do *real* ternary-training research on a single consumer GPU?\n(A 24 GB RTX 4090, in a personal workstation, total cost of these\nexperiments: an afternoon of electricity.)\n2. What does ternary quantization actually cost when you measure it honestly — equal token budgets, equal evaluation, deployed bytes counted including everything that stays floating point?\n\nThe weight format is BitNet b1.58-style ternary: every weight in {-1, 0, +1} with a per-output-row fp16 scale, signs packed 2 bits each. \"1.57-bit\" ≈ log₂(3).\n\n- **Model** : 126M-param decoder-only transformer, faithful\nreimplementation of the nanochat architecture (RoPE, QK-norm, relu² MLP,\nvalue embeddings, sliding-window attention). Config`smoke` : 8 layers,\n512 dim.\n- **Data** : ClimbMix (`karpathy/climbmix-400b-shuffle` ), pre-tokenized\ninto packed 2048-token rows; validation is the held-out last shard.\n- **Tokenizer** : 32,768-vocab BPE trained on 1B chars of the training\ndistribution (rustbpe + tiktoken), 4.68 chars/token on English.\n- **Training** : Muon (matrices) + AdamW (embeddings/scalars), nanochat's\nhyperparameter scaling system, 503M tokens = 12× the matmul parameter\ncount (compute-optimal by the nanochat rule of thumb), 1,920 steps of\n262,144 tokens, ~1.2 h per run on the 4090 (~120k tokens/s).\n- **Quantizer** : per-row threshold`t = mean(|row|)` , sign ±1 outside, 0\ninside; row scale`‖row‖₂ / √nnz` . QAT uses the same quantizer in the\nforward pass with a straight-through estimator; master weights stay\nfloating point during training. Two variants:**standard** (transformer\nmatrices ternary; embeddings/head float) and**max-compression** (embedding tables ternary too; lm_head stays float).\n- **Metric** : bits-per-byte (bpb) on a fixed 2M-token window at the start\nof the validation shard. Deployed bytes count packed signs + fp16 scales\n  - all float tensors in bf16. No cherry-picked generations.\n\n| model | val bpb | Δ vs fp | deployed size | vs fp size | \n|---|---|---|---|---|\n| fp control (bf16) | 0.9450 | — | 251.7 MB | 1× | \n| PTQ standard | 1.6040 | +0.659 | — | — | \n| PTQ max-compression | 1.7644 | +0.819 | — | — | \n| **QAT standard** | **0.9977** | **+0.053** | 207.7 MB | 0.83× | \n| **QAT max-compression** | **1.0378** | **+0.093** | **61.2 MB** | **0.24×** | \n\n- QAT closes **92%** of the PTQ gap (standard) and**89%** (max-comp).\n- Max-compression pays only +0.040 bpb over standard QAT and is **3.4×\nsmaller** — the clear quality-per-byte winner.\n- QAT quantizer telemetry was healthy throughout: per-layer trit fractions steady at ~19% / 62% / 19% (−/0/+), no divergence, no Muon+STE instability, ~4% throughput overhead vs the fp run.\n\n**Does:** at matched tokens/data/eval, ternary QAT from random init is\nviable on commodity hardware and lands within ~0.05–0.09 bpb of its fp\ntwin; naive PTQ of a model this small/undertrained is not viable;\nquality-per-byte strongly favors ternarizing embeddings too.\n\n**Doesn't (yet):** these are single-seed runs at 126M params / 503M tokens\n(seed-2 replicates are in flight; the repo will be updated). This is not a\nclaim about 1B+ scale, about downstream-task quality (bpb only), or about\nmaster-weight-free training (that's the TTF experiment, next on our\nqueue). The fp model itself is a research-grade 126M, not a chat model.\n\n```\nuv venv --python 3.13 .venv\nuv pip install --python .venv/bin/python -r requirements.txt\npython -m nanotern.dataset -n 12                 # ~1.2 GB ClimbMix shards\npython -m nanotern.tok_train --max-chars 1000000000\npython -m nanotern.tokenize --split train --n-shards 10\npython -m nanotern.tokenize --split val\npython -m nanotern.train --config smoke --run-name fp-control            # fp arm\npython -m nanotern.train --config smoke --run-name qat-std --qat         # QAT arm\npython -m nanotern.train --config smoke --run-name qat-max --qat \\\n    --qat-variant max-compression                                        # max-comp arm\npython eval_bpb.py --checkpoint <ckpt.pt>                                # eval any arm\npython -m nanotern.ternary --from-checkpoint <ckpt.pt> --out tern.pt     # PTQ export\n```\n\nFull work log with every decision and its rationale: [`DECISIONS.md`](https://github.com/comalice/nanotern-public/blob/master/DECISIONS.md).\nThings that occurred to the agents along the way: [`INSIGHTS.md`](https://github.com/comalice/nanotern-public/blob/master/INSIGHTS.md).\n\n- Architecture and training recipe: reimplemented from\n[karpathy/nanochat](https://github.com/karpathy/nanochat) (MIT) as a\nspec; no nanochat code is imported.\n- Quantizer: BitNet b1.58-style per-row ternarization.\n- Muon optimizer: Keller Jordan's modded-nanogpt lineage (Polar Express / NorMuon variants), ported single-GPU.\n- This project is developed in the open by a human + LLM agents (the commit log is honest about which commits are agent-authored).", "url": "https://wpnews.pro/news/nanotern-ternary-llm-training-on-consumer-hardware", "canonical_source": "https://github.com/comalice/nanotern-public/blob/master/REPLICATION_REPORT.md", "published_at": "2026-11-10 17:35:04+00:00", "updated_at": "2026-11-10 17:53:45.256469+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Nanotern", "RTX 4090", "nanochat", "ClimbMix", "Muon", "AdamW", "BitNet b1.58"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/nanotern-ternary-llm-training-on-consumer-hardware", "markdown": "https://wpnews.pro/news/nanotern-ternary-llm-training-on-consumer-hardware.md", "text": "https://wpnews.pro/news/nanotern-ternary-llm-training-on-consumer-hardware.txt", "jsonld": "https://wpnews.pro/news/nanotern-ternary-llm-training-on-consumer-hardware.jsonld"}}