Paper-explained Series: 12
If you’ve read my earlier deep-dives on HRM, Mamba, and TRM, you know the recurring theme: the frontier isn’t only about making models bigger. Sometimes the most interesting research asks whether we’ve been paying for precision we never actually needed. BitNet is the purest expression of that idea. It asks a heretical question — what if every weight in a large language model were just −1, 0, or +1? — and then answers it by training models that match full-precision baselines while shrinking memory by an order of magnitude and turning the most expensive operation in deep learning, matrix multiplication, into plain addition.
Let me walk you through the whole story, from why a 70B model is so painful to run, through the math of quantization, to the ternary trick, and finally to the 2-billion-parameter model Microsoft trained on 4 trillion tokens and put on Hugging Face in 2025.
Let’s ground this in hardware you can actually buy. The NVIDIA RTX 4090 is the top consumer GPU, and it ships with 24 GB of VRAM. That’s your budget.
Now take a 70-billion-parameter model like Llama-3 70B. In FP16 (half precision), every parameter occupies 2 bytes. So the weights alone require:
70 × 10⁹ params × 2 bytes = 140 GB
That’s just the static weights. It doesn’t fit in 24 GB — it doesn’t fit in six RTX 4090s (144 GB), and once you account for overhead you need a seventh card. APXML’s Llama-3 70B VRAM guide is explicit: at FP16 with 1,024 tokens of context the model needs 148.85 GB → 7× RTX 4090 (24 GB each), rising to 151.32 GB → 8× RTX 4090 at 8,192 tokens (or, in the data center, “2× NVIDIA A100 · 80 GB”).
And weights are only part of the bill. During inference, transformers cache the key and value vectors for every past token — the KV cache — so they don’t recompute attention from scratch at each step. This cache grows linearly with sequence length and batch size, and for a 70B model at long context it can rival or exceed the model weights themselves. On top of that you have activations flowing through the network. A frontier 70B model is fundamentally a data-center citizen: two 80 GB A100s or H100s, minimum, wired together with fast interconnect.
For anyone who wants to run a capable model on a laptop, a phone, or a single consumer card, 140 GB of FP16 weights is simply a wall.
The obvious fix is to stop storing weights in 16 bits. This is quantization — mapping high-precision floats into a smaller set of low-precision values.
The dominant flavor is post-training quantization (PTQ): you take an already-trained FP16 model and compress its weights after the fact, with no retraining (or just a short calibration pass over sample data). Two milestones defined the field:
When researchers first tried compressing models larger than 6.7 billion parameters into 8-bit (INT8), the models suddenly broke down and output gibberish. The LLM.int8() paper discovered why: Outlier Features.
Compressing from 16-bit to 8-bit is hard, but going to 4-bit is brutal. In 4-bit, you only have 16 possible numbers to represent a weight. You have to be incredibly clever about how you round things. The text highlights three distinct strategies for surviving 4-bit compression:
A. GPTQ (The “Compensation” Trick):
When you round a weight down, you introduce a tiny mathematical error. GPTQ uses advanced calculus (Hessian matrices) to look at how that error ripples through a specific layer of the network.
B. AWQ (The “VIP Protection” Trick):
Not all weights are equally important. AWQ (Activation-aware Weight Quantization) runs sample text (activations) through the model to see which weights are actually doing the most work.
C. QLoRA and NF4 (The “Bell Curve” Trick):
When you compress a model down to 4-bit, you only have exactly 16 available numbers to represent the billions of weights inside the AI.
If you look at the raw data of a neural network, the vast majority of its weights are tiny fractions clustered incredibly close to zero (e.g., 0.015, -0.04, 0.002). Very few weights are large numbers. They form a natural bell curve.
If you space your 16 available numbers evenly across a number line, you create a major problem:
NF4 (4-bit NormalFloat) solves this by changing the spacing. Instead of spacing the 16 numbers evenly, it clusters the majority of them tightly around zero and spaces them much further apart at the extreme edges. Because the spacing perfectly matches the bell-curve distribution of the data, the model retains maximum precision exactly where the bulk of its “brain” actually operates.
Adding LoRA to the Mix
Once you compress a massive model into 4-bit using NF4, it becomes “read-only.” The math is too restricted to teach it anything new (a process called fine-tuning). This is where LoRA (Low-Rank Adaptation) comes in.
To teach the AI a new skill without uncompressing it, researchers use a clever workaround:
Think of the 4-bit base model like a massive, printed encyclopedia. You can’t change the printed text, but LoRA allows you to slap a small sticky note of new, specialized instructions onto the page.
QLoRA is simply the combination of these two techniques: using N F4 to shrink the base model so it fits on your graphics card, and using L oRA to train it on new data without running out of memory.
Here’s the counterintuitive part that motivates everything after: given a fixed memory budget, you’re almost always better off taking a big model and quantizing it hard than training a small model at full precision.
The definitive study is Dettmers & Zettlemoyer’s “The case for 4-bit precision: k-bit inference scaling laws” (2023). In their words, they “run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2.” The headline: 4-bit precision is almost universally optimal for the trade-off between total model bits and zero-shot accuracy. In other words, for a fixed number of bits, a 4-bit model with more parameters beats an 8-bit or 16-bit model with fewer.
But there’s a floor. Their scaling curves show the trend reverses below 3 bits — at 3-bit and lower, quality can fall off a cliff rather than degrade gracefully, and several models (OPT, Pythia) became unstable. PTQ, being a lossy afterthought applied to a model that never “knew” it would be quantized, simply can’t push reliably past that barrier.
That’s the gap BitNet aims at. If 4-bit is the PTQ sweet spot and sub-3-bit is a minefield for PTQ, maybe the way to reach 1-ish bits isn’t to compress after training — it’s to train in low precision from the start, so the network learns to live within the constraint.
Before we get to ternary, let’s make the mechanics concrete, because BitNet reuses exactly these primitives.
Here is the step-by-step breakdown of how modern quantization works.
The most common method is absmax (symmetric) quantization. It assumes your data is roughly centered around zero, and it calculates a single multiplier to stretch or shrink your weights so they perfectly fit into your integer limits.
Let’s assume you want to compress your weights into b-bit integers.
Example: If you are using 8-bit quantization, the max integer is 127. If the largest weight in your network is 2.5, your scale is 127 / 2.5 = 50.8.
scale s = (2^(b−1) − 1) / max(|W|)W_quant = round(s · W) # integers in [−(2^(b−1)−1), +(2^(b−1)−1)]
To use the weights, you dequantize by dividing back out:
W_dequant = W_quant / s
Because rounding throws information away, W_dequant ≠ W. The gap is the quantization error, and minimizing it is the whole game.
Symmetric quantization forces the “zero” of the integer buckets to line up perfectly with the mathematical 0.0. This works beautifully if your weights form a bell curve centered on zero.
But what if your data is purely positive (like activations that have passed through a ReLU function, which deletes negative numbers)? If you use symmetric quantization, you will waste half of your integer buckets on negative numbers that don’t exist.
Asymmetric quantization introduces a “zero-point” — an offset that shifts the integer buckets left or right. Instead of centering on zero, it perfectly maps the minimum value of your data to the lowest bucket, and the maximum value to the highest bucket. No buckets are wasted.
The math above requires calculating a scale (s). The critical question is: how much data do you apply that single scale to?
If you use one scale for millions of weights, a single massive outlier will ruin the scale for everything else (like a billionaire ruining the income scale for a normal neighborhood). Granularity determines how you group the weights before calculating their scale.
Keep these three ideas — scale, rounding error, and granularity — in your head. BitNet’s ternary quantizer is just a very aggressive, very cleverly-scaled version of them.
In October 2023, Wang et al. published “BitNet: Scaling 1-bit Transformers for Large Language Models” (arXiv:2310.11453). Their claim to fame: the first architecture to do quantization-aware training (QAT) for 1-bit LLMs from scratch, using a drop-in module called BitLinear to replace nn.Linear.
In this first version, weights were binary — just {−1, +1}. The recipe: center the weights to zero mean, then take the sign.
W̃ = Sign(W − α), where α = (1/nm) Σ W_ij (Eqs. 1–3)
α is simply the mean of the weight matrix. Subtracting it before applying Sign — rather than signing the raw weights directly — is the detail that makes this work well rather than just work.
Sign(x) = +1 if x > 0, −1 otherwise
Here’s why centering matters: a single bit carries the most information when its two outcomes are equally likely. If the raw weight distribution has any skew (and trained weight matrices almost always do), signing them directly produces an imbalanced mix of +1s and −1s — some of the 1-bit budget gets wasted encoding a bias that a scalar could have captured for free. Centering to zero mean before signing pushes the split toward 50/50, which maximizes the entropy of the resulting binary code. The mean α is discarded after it’s done its job of setting the decision boundary — it isn’t added back anywhere downstream.
Binarization throws away magnitude entirely. A weight of 0.002 and a weight of 4.8 both collapse to the same +1 if they’re on the same side of α. To partially undo this, BitNet computes one scalar per weight matrix:
β = (1/nm) Σ|W_ij| = ‖W‖₁ / (nm)
This is just the mean absolute weight — but it isn’t an arbitrary choice of “some average.” It’s the exact solution to a least-squares problem:
min_β ‖W − βW̃‖²
Given that the sign pattern W̃ is already fixed, β is the one number that makes βW̃ the closest possible reconstruction of the real-valued W, in the least-squares sense. So the two-step recipe — sign first, then scale — isn't heuristic layering; each step is separately optimal given the one before it.
Weights collapse to 1 bit, but activations are left considerably more precision — 8 bits in BitNet’s experiments — because activations carry the actual input-dependent signal flowing through the network and are far more sensitive to aggressive rounding. The scheme is standard absmax quantization
γ = max(|x|)Quant(x) = Clip(round(x · Q_b / γ), −Q_b, Q_b − 1), Q_b = 2^(b−1)
γ is the largest-magnitude value in the activation vector. Scaling by Q_b / γ stretches the activations so the biggest value uses the full available integer range, then rounding and clipping produces signed 8-bit integers in [−128, 127] for b = 8.
Right before the quantization step, activations pass through a LayerNorm placed inside the sublayer (hence “Sub-LN”), rather than only at block boundaries the way a standard Pre-LN transformer does it.
This does two jobs at once. First, it keeps the input to Quant() well-scaled: without it, activation magnitudes can drift as they propagate through many binarized layers, and a single outlier value would inflate γ and crush the resolution available to every other entry in that vector. Second, it stabilizes training — a network built from 1-bit weights has meaningfully different variance dynamics than a full-precision one, and the extra normalization compensates for that so gradients don't blow up or vanish across depth.
y = W̃ · Quant(LN(x)) × (βγ / Q_b)
Walking through it left to right:
So the O(nm) part of the computation — the part that scales with model size — runs in cheap integer ops, and the only float math is a single scalar multiply on the output.
Importantly, BitNet keeps itsgradients and optimizer states in full precisionduring training and uses adeliberately large learning rate— a small nudge to a latent weight often won’t flip a binary value, so aggressive steps are needed to make progress.
In February 2024, the same group published the paper that made the field sit up: “The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits” (Ma et al., arXiv:2402.17764). The change is deceptively small: add a third value, 0, so every weight is now ternary: {−1, 0, +1}.
The quantizer switches from sign to absmean: scale the weight matrix by its mean absolute value, then round each weight to the nearest of {−1, 0, +1}:
W̃ = RoundClip(W / (γ + ε), −1, 1)RoundClip(x, a, b) = max(a, min(b, round(x)))γ = (1/nm) Σ |W_ij|
Why does adding zero matter so much? Because 0 lets the network switch a connection completely off. A weight of 0 means “this input feature does not contribute to this output” — it’s built-in feature filtering and sparsity, learned during training. Binary {−1, +1} forces every connection to vote either for or against; ternary lets the model abstain. That extra expressive freedom is exactly why b1.58 closes the gap with full precision that binary BitNet couldn’t. You still keep the multiplication-free property — a ternary matmul is just conditional adds and skips.
BitNet b1.58 was compared against a reproduced FP16 LLaMA, both trained from scratch on 100 billion tokens of RedPajama at matched sizes. The numbers:
The crossover is the headline: at 3B parameters, BitNet b1.58 matches full-precision LLaMA in perplexity while using 3.55× less GPU memory and running 2.71× faster. Zero-shot task accuracy tells the same story — the gap narrows with scale and b1.58 actually matches or exceeds FP16 from 3B onward (b1.58 3B scored 50.2 average vs LLaMA 3B’s 49.7 across seven tasks). And the 3.9B ternary model beats the 3B FP16 model on accuracy while being cheaper on every axis — a genuine Pareto improvement.
On energy, the paper estimates BitNet b1.58 saves 71.4× the arithmetic energy of matrix multiplication versus FP16 on a 7 nm chip (because it’s almost all INT8 addition, no FP16 multiplication). End-to-end energy savings grow with size, from ~18.6× at 1.3B to ~41.2× at 70B.
On throughput, comparing two 70B models on A100–80GB cards: BitNet’s smaller memory footprint let it run 11× the batch size and deliver 8.9× the throughput (2,977 vs 333 tokens/sec) of FP16 LLaMA.
The team also verified token-scalability: a b1.58 3B trained on 2 trillion tokens beat StableLM-3B (trained on the same 2T tokens) on every reported end task.
The most quotable consequence is the “equivalence table” the paper derives from its latency/memory/energy curves. Because the savings compound as models grow, they claim:
Read that last line slowly. A 70-billion-parameter ternary model is cheaper to run than a 13-billion-parameter FP16 model — while being vastly more capable, since it has more than five times the parameters. This is the inversion of everything the 140 GB wall implied. The bottleneck for LLM inference has historically been memory bandwidth (shuttling weights from DRAM to on-chip SRAM); when your weights are 1.58 bits instead of 16, that bandwidth cost mostly evaporates. (Note this is a projection from the paper’s scaling curves — see the caveats.)
This trips people up, so let’s nail it. A single ternary weight can be in one of 3 states. The information content of choosing among 3 equally likely options is:
log₂(3) ≈ 1.585 bits
That’s it. Two states (binary) = log₂(2) = 1 bit. Three states (ternary) = log₂(3) ≈ 1.58 bits. The name “b1.58” is a precise, slightly nerdy way of saying “ternary.” You can’t store 1.58 bits directly, of course — in practice the inference code packs multiple ternary weights into one byte. Microsoft’s GPU kernel packs four ternary values into a single INT8 (four values × ~1.58 ≈ 6.3 bits, comfortably under 8).
Here’s the catch that the papers are refreshingly honest about. Today’s GPUs — with their cuBLAS libraries and tensor cores — are exquisitely optimized for FP16, BF16, and INT8/INT4 GEMM (general matrix multiply). There is no native hardware instruction for a “1.58-bit × 8-bit” matrix multiply (W1.58A8). So even though ternary weights should be dramatically faster, off-the-shelf GPUs can’t fully cash in the theoretical win.
The workaround is custom kernels. For GPUs, Microsoft wrote a bespoke CUDA kernel that stores four packed ternary weights per INT8 in high-bandwidth memory, loads them into fast on-chip SRAM, and unpacks-then-computes (“pack-store-load-unpack-compute”). For CPUs, they built bitnet.cpp, an official inference framework (forked from llama.cpp, using lookup-table kernels pioneered in T-MAC). Per Microsoft’s own benchmarks, bitnet.cpp delivers 2.37×–6.17× speedups on x86 CPUs with 71.9%–82.2% energy reduction, and 1.37×–5.07× on ARM with 55.4%–70.0% energy reduction — all “lossless” relative to the training procedure. Strikingly, “bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5–7 tokens per second).”
Both papers end with the same call to action: this is a new computation paradigm that “opens the door for designing specific hardware optimized for 1-bit LLMs.” They explicitly point to Groq’s LPU-style dedicated inference hardware as evidence that purpose-built silicon delivers, and argue for chips designed around ternary addition from the ground up.
A common misconception is that BitNet ternarizes everything. It doesn’t. Only the big matrix multiplications become BitLinear:
Replaced with 1.58-bit BitLinear:
Kept in higher precision:
The paper’s justification is pragmatic: residual connections and layer norm contribute negligible compute at scale, and the QKV transformation cost shrinks relative to the parametric projections as the model grows. So you ternarize where the FLOPs and the memory actually live, and leave the cheap, sensitive parts alone.
Once weights are ternary, as in BitNet b1.58, the memory cost of weights mostly disappears. What’s left is the matmul itself — the activation-times-weight arithmetic — which was still running on 8-bit activations. In November 2024, Wang, Ma, and Wei released BitNet a4.8: 4-bit Activations for 1-bit LLMs (arXiv:2411.04965), aimed squarely at that remaining cost. Weights stay ternary; the goal is to push activations from 8-bit down to 4-bit so inference can run on fast INT4/FP4 kernels.
The obstacle is the one that haunts every activation-quantization scheme: outlier channels. A handful of unusually large activation values force the quantization scale wide enough that everything else gets crushed into just a few usable integer levels. BitNet a4.8’s contribution, based on studying activation distributions across a 7B BitNet b1.58 model, was noticing that different parts of the network behave very differently — so rather than one blanket policy, it applies a hybrid quantization-and-sparsification strategy:
To push sparsity further, a4.8 also swaps the FFN’s SwiGLU nonlinearity for a gated squared-ReLU variant (ReLU²GLU). Because squared ReLU hard-clamps every negative input to exactly zero, it drives the down-projection’s inputs to over 80% exact-zero entries at 7B scale, with the gate branch alone reaching around 67.5% zeros.
That last detail is what actually produces the paper’s headline number. Every time an activation entry lands on exactly zero — whether from ReLU² or from top-K masking — the weight column it would have multiplied contributes nothing to the output for that token, and can be skipped outright. Averaged across all the sublayers, weighted by each layer’s share of the model’s parameters, this works out to roughly half the network per token: for a 7B model, about 3.4B of 6.5B parameters actually get touched on a given forward pass — close to the paper’s rounded headline that only around 55% of parameters are active at inference.
It’s worth keeping the two levers separate, since it’s easy to conflate them: sparsification (from ReLU² and top-K masking) is what creates the zeros and shrinks the active fraction below 100% in the first place; quantization (4-bit vs. 8-bit) is a separate axis that just determines how cheaply the surviving ~55% gets computed, via fast INT4/FP4 kernels. Both matter, but only one of them is why not all the parameters get used.
a4.8 also enables a 3-bit KV cache. KV cache stores past keys and values and grows linearly with sequence length — for long contexts it becomes the memory bottleneck, often exceeding the weights themselves (for a model like LLaMA3–70B, serving 32 requests at 128K context can need over 1.2 TB of KV cache). a4.8 quantizes the KV cache down to 3 bits: after applying RoPE, the K and V heads are quantized directly with absmax to unsigned low-bit integers, no calibration needed, with one small exception — the bos (beginning-of-sequence) token's heads are kept at 4-bit because they carry the most extreme outlier features. The result: negligible accuracy loss even at 3-bit KV, roughly halving KV-cache memory versus 8-bit and letting you serve far longer contexts on the same hardware. Training-wise, a4.8 is continue-trained from a b1.58 checkpoint in two stages (W1.58A8 → W1.58A4), needing only a few billion extra tokens, and matches b1.58 accuracy at every size from 700M to 7B.
For all its promise, there was a nagging asterisk on the early BitNet results. The b1.58 and a4.8 experiments were run at a research scale of 100 billion tokens (with a single 2-trillion-token comparison against StableLM). Frontier models are trained on trillions — Llama 3 on 15T, Qwen2.5 on 18T. It was genuinely unknown whether the “matches full precision” story would survive when you trained a native 1-bit model at true frontier data scale, or whether some subtle instability would emerge over trillions of tokens. Training a ternary model at full scale remained the field’s big open question.
In April 2025, Microsoft Research answered the open question with “BitNet b1.58 2B4T Technical Report” (arXiv:2504.12285): a 2-billion-parameter native ternary model trained on 4 trillion tokens — the “2B4T” name. It’s the first open-source, native 1-bit LLM at this scale, and the weights are on Hugging Face (microsoft/bitnet-b1.58-2B-4T).
Architecture. A LLaMA-3-style decoder: 30 layers, hidden size 2,560, 20 attention heads with grouped-query attention (5 KV heads), intermediate size 6,912, squared-ReLU FFN, RoPE positional embeddings, SubLN normalization, no bias terms, and the LLaMA-3 tokenizer (128,256 vocab). All the big linear layers are BitLinear (ternary weights, absmean); activations are 8-bit (absmax, per-token). It’s a W1.58A8 model.
Training recipe. Three phases:
Results. Benchmarked against LLaMA 3.2 1B, Gemma-3 1B, Qwen2.5 1.5B, SmolLM2 1.7B, and MiniCPM 2B (all instruction-tuned, full precision), BitNet b1.58 2B4T posts an average of 54.19 across 16 benchmarks — essentially tied with Qwen2.5 1.5B’s 55.23 and ahead of everything else — while leading the pack on several: ARC-Challenge (49.91), WinoGrande (71.90), GSM8K math (58.38), and PIQA (77.09).
The efficiency, though, is the whole point:
*To actually get these numbers you must run it through bitnet.cpp (or the custom CUDA kernel); the vanilla *transformers library has no ternary kernels and will show none of the speed or energy benefits.
The deepest challenge in training a ternary network is that calculus and rounding do not mix. The derivative of a rounding function is zero on the flat parts and undefined at the jumps. If you try to run standard backpropagation through a rounding step, the gradient zeroes out. The learning signal dies instantly, and the model learns nothing.
To survive this, the system maintains two separate versions of every weight simultaneously:
Because of this dual-weight setup, the training process relies on a strict division of labor. The bridge that makes this possible is the Straight-Through Estimator (STE).
Here is exactly how the loop flows:
The Result: The gradients are calculated based on the ternary weights, but applied to the latent weights. Over thousands of steps, these tiny updates accumulate in the full-precision latent weights. Once a latent weight drifts far enough across a threshold, its rounded ternary shadow flips to a new value (-1, 0, or +1), and the network successfully learns.
This is also why the “1-bit LLMs training tips” guidance emphasizes a large learning rate and the two-stage weight-decay schedule: with weight decay applied to the latent weights, its magnitude acts like a confidence score for each ternary weight, so decay is disabled in the second half of training to let the model settle.
BitNet keeps gradients and optimizer states in full precision for exactly this reason — accumulating updates in low precision would vanish or explode. And the beautiful punchline: once training is done, you throw the latent weights away. They exist only to make learning possible. At inference you keep only the ternary weights — which is why the shipped inference model is 0.4 GB while the BF16 “master” checkpoint (also on Hugging Face, for fine-tuning) is much larger.
Until next time, folks…El Psy Congroo
How BitNet Run a Transformer With (Almost) No Multiplication? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.