My interest in LLMs has increasingly shifted towards smaller models, small in terms of size and their ability to run on modest local machines. For some time now, I have been using models with fewer than 40 B parameters on my machines in some quantized way. Then I came across BitNet, a project that introduced me to the concept of using just 1 bit per weight.
Quantization
Weights are represented by numbers in a model. A normal model might store each weight using 16 bits, i.e** **FP16 (half-precision floating-point). Quantization reduces that dramatically, where weights are represented using INT8 (8-bit integer) or INT4 (4-bit integer). They reduce precision but increase inference speed and reduce hardware requirements. Most of the time, quantization is performed post-training, which reduces model accuracy. But there is INT4 QAT (4-bit Integer Quantization-Aware Training), which does this during training, thereby reducing the extreme accuracy drops common with standard 4-bit quantization.
| Approach | Training | Inference weights |
|---|---|---|
| FP16 model | Full precision | FP16 |
| INT8 post-training quantization | Train normally, quantize later | INT8 |
| INT4 QAT | Quantization simulated during training | INT4 |
| BitNet | Designed and trained around extreme low-bit weights | ternary or binary |
1 bit weight
1-bit goes one step further and asks: what if the weights were just one bit, and if that were done during training? With exactly 1 bit per weight, each weight can have only 2 possible values. 0 or 1. 0 represents -1 and 1 represents +1. i.e., instead of storing numbers like 0.72, -0.13, 1.24, -0.56, we will store +1, -1, +1, -1. That is at least 16 times less storage for weights theoretically.
1.58 bit weight
In 1.58-bit weights (ternary bit weights), there are three possible values: {-1, 0, +1}. Theoretically, we need log2(3) bits of information to distinguish between 3 possible states : 2x=3. i.e. log2(3)≈1.585. Hence the name.
For our interpretation and understanding:
- 0: the input is effectively ignored for that connection. i.e., no influence at all, which also gives you built-in sparsity.
- +1: the input contributes positively to the output. i.e., an increase in the input pushes the weighted sum up. This is “same-direction” influence.
- -1: the input contributes negatively. i.e., an increase in the input pushes the weighted sum down. This is “opposite-direction” influence.
1.125-bit weight
1.125-bit applies different logic for compression and calculation. One can’t literally allocate 1.125 bits per weight. The way this works is that the average bit allocation is used when they are packed. For example, it takes 9 bits to store 8 weights; thus, the average number of bits per weight is 1.125. One important distinction here is that bits per weight doesn’t necessarily tell you how many values each individual weight can take. 1.58 has a clean theoretical connection to three states, while 1.125 bpw usually describes the overall encoding/storage scheme. The table below should make a bit clear
| Format | Possible basic values | Approx storage / 1B weights |
|---|---|---|
| FP16 | ~65,536 | 2 GB |
| INT8 | 256 | 1 GB |
| INT4 | 16 | 500 MB |
| Ternary | 3 | ~198 MB at theoretical 1.58 bpw |
| 1.125-bit | special packed encoding | ~141 MB |
| Binary | 2 | 125 MB |
BitNet b1.58 2B4T
This is the first open-source, native 1-bit Large Language Model (LLM) at the 2-billion parameter scale, developed by Microsoft Research. You can find the model on the hugging face, it needs a specific inference code called bitnet.cpp.
Bonsai models
Bonsai 1-bit models are probably the first set of production-ready binary/ternary models. They have 1.7B, 4B, 8B, and 27B parameter models.
I have been running the 8B-parameter version locally. My experience with it has been positive. I have explored it for text summaries, code explanations, etc. I will try to install the 27B version and do some coding to see how it goes.
curl -LsSf https://llama.app/install.sh | sh
llama serve -hf prism-ml/Bonsai-1.7B-gguf:Q1_0
llama serve -hf prism-ml/Bonsai-8B-gguf:Q1_0
Note: Yes, I have moved away from Ollama and now mostly use llama.cpp. Also I sometimes Jan.ai along with OpenWebUI