Nanotern – Ternary LLM training on consumer hardware A 126M-parameter nanochat-style transformer trained from scratch on a single 24 GB RTX 4090 with ternary quantization-aware training (QAT) landed within +0.053 bits-per-byte of its bf16 control (0.9977 vs 0.9450 bpb) at 207.7 MB versus 251.7 MB, while a max-compression QAT variant hit 61.2 MB (−76%) for +0.093 bpb, according to the Nanotern project's matched comparison. Naive post-training ternarization was catastrophic at +0.659 bpb (standard) and +0.819 bpb (max-compression), so QAT closed 92% and 89% of that PTQ gap respectively. All runs used the same ClimbMix data, 32,768-vocab BPE tokenizer, 503M-token budget and a fixed 2M-token validation window, at roughly 1.2 GPU-hours each; the authors caution these are single-seed results at 126M parameters and make no claim about 1B+ scale or downstream-task quality. A matched fp / PTQ / QAT comparison at 126M parameters, with honest numbers. TL;DR — We trained a 126M-parameter transformer nanochat-style architecture from scratch on one consumer GPU, three ways: full bf16, naive post-training ternarization PTQ , and quantization-aware training QAT with ternary weights from random init. QAT nearly matches the fp model +0.053 bpb while the fully-ternarized "max-compression" variant deploys at 61 MB vs the fp model's 252 MB −76% for +0.093 bpb. Naive PTQ is catastrophic +0.66 bpb . All runs are matched: same data, same tokenizer, same 503M-token budget, same evaluation window. Everything to reproduce this is in this repo; each run is ~1.2 GPU-hours. Two questions drove this: 1. Can you do real ternary-training research on a single consumer GPU? A 24 GB RTX 4090, in a personal workstation, total cost of these experiments: an afternoon of electricity. 2. What does ternary quantization actually cost when you measure it honestly — equal token budgets, equal evaluation, deployed bytes counted including everything that stays floating point? The weight format is BitNet b1.58-style ternary: every weight in {-1, 0, +1} with a per-output-row fp16 scale, signs packed 2 bits each. "1.57-bit" ≈ log₂ 3 . - Model : 126M-param decoder-only transformer, faithful reimplementation of the nanochat architecture RoPE, QK-norm, relu² MLP, value embeddings, sliding-window attention . Config smoke : 8 layers, 512 dim. - Data : ClimbMix karpathy/climbmix-400b-shuffle , pre-tokenized into packed 2048-token rows; validation is the held-out last shard. - Tokenizer : 32,768-vocab BPE trained on 1B chars of the training distribution rustbpe + tiktoken , 4.68 chars/token on English. - Training : Muon matrices + AdamW embeddings/scalars , nanochat's hyperparameter scaling system, 503M tokens = 12× the matmul parameter count compute-optimal by the nanochat rule of thumb , 1,920 steps of 262,144 tokens, ~1.2 h per run on the 4090 ~120k tokens/s . - Quantizer : per-row threshold t = mean |row| , sign ±1 outside, 0 inside; row scale ‖row‖₂ / √nnz . QAT uses the same quantizer in the forward pass with a straight-through estimator; master weights stay floating point during training. Two variants: standard transformer matrices ternary; embeddings/head float and max-compression embedding tables ternary too; lm head stays float . - Metric : bits-per-byte bpb on a fixed 2M-token window at the start of the validation shard. Deployed bytes count packed signs + fp16 scales - all float tensors in bf16. No cherry-picked generations. | model | val bpb | Δ vs fp | deployed size | vs fp size | |---|---|---|---|---| | fp control bf16 | 0.9450 | — | 251.7 MB | 1× | | PTQ standard | 1.6040 | +0.659 | — | — | | PTQ max-compression | 1.7644 | +0.819 | — | — | | QAT standard | 0.9977 | +0.053 | 207.7 MB | 0.83× | | QAT max-compression | 1.0378 | +0.093 | 61.2 MB | 0.24× | - QAT closes 92% of the PTQ gap standard and 89% max-comp . - Max-compression pays only +0.040 bpb over standard QAT and is 3.4× smaller — the clear quality-per-byte winner. - QAT quantizer telemetry was healthy throughout: per-layer trit fractions steady at ~19% / 62% / 19% −/0/+ , no divergence, no Muon+STE instability, ~4% throughput overhead vs the fp run. Does: at matched tokens/data/eval, ternary QAT from random init is viable on commodity hardware and lands within ~0.05–0.09 bpb of its fp twin; naive PTQ of a model this small/undertrained is not viable; quality-per-byte strongly favors ternarizing embeddings too. Doesn't yet : these are single-seed runs at 126M params / 503M tokens seed-2 replicates are in flight; the repo will be updated . This is not a claim about 1B+ scale, about downstream-task quality bpb only , or about master-weight-free training that's the TTF experiment, next on our queue . The fp model itself is a research-grade 126M, not a chat model. uv venv --python 3.13 .venv uv pip install --python .venv/bin/python -r requirements.txt python -m nanotern.dataset -n 12 ~1.2 GB ClimbMix shards python -m nanotern.tok train --max-chars 1000000000 python -m nanotern.tokenize --split train --n-shards 10 python -m nanotern.tokenize --split val python -m nanotern.train --config smoke --run-name fp-control fp arm python -m nanotern.train --config smoke --run-name qat-std --qat QAT arm python -m nanotern.train --config smoke --run-name qat-max --qat \ --qat-variant max-compression max-comp arm python eval bpb.py --checkpoint