Exploring 1-Bit LLMs

Microsoft Research released BitNet b1.58 2B4T, the first open-source native 1-bit LLM at the 2-billion parameter scale, which uses ternary weights (-1, 0, +1) requiring about 1.58 bits per weight, reducing storage to roughly 198 MB per billion weights compared to 2 GB for FP16. The model is available on Hugging Face and requires the bitnet.cpp inference code. The article also explains 1.125-bit packing, where 8 weights are stored in 9 bits, averaging 1.125 bits per weight, and contrasts these approaches with traditional quantization methods like INT8 and INT4 QAT.

Exploring 1-Bit LLMs My interest in LLMs has increasingly shifted towards smaller models, small in terms of size and their ability to run on modest local machines. For some time now, I have been using models with fewer than 40 B parameters on my machines in some quantized way. Then I came across BitNet, a project that introduced me to the concept of using just 1 bit per weight. Quantization Weights are represented https://hai.stanford.edu/ai-definitions/what-are-weights by numbers in a model. A normal model might store each weight using 16 bits, i.e FP16 half-precision floating-point . Quantization reduces that dramatically, where weights are represented using INT8 8-bit integer or INT4 4-bit integer . They reduce precision but increase inference speed https://developer.nvidia.com/blog/int4-for-ai-inference/ and reduce hardware requirements. Most of the time, quantization is performed post-training, which reduces model accuracy. But there is INT4 QAT 4-bit Integer Quantization-Aware Training https://www.ibm.com/think/topics/quantization-aware-training , which does this during training, thereby reducing the extreme accuracy drops common with standard 4-bit quantization. | Approach | Training | Inference weights | |---|---|---| | FP16 model | Full precision | FP16 | | INT8 post-training quantization | Train normally, quantize later | INT8 | | INT4 QAT | Quantization simulated during training | INT4 | | BitNet | Designed and trained around extreme low-bit weights | ternary or binary | 1 bit weight 1-bit goes one step further and asks: what if the weights were just one bit, and if that were done during training? With exactly 1 bit per weight, each weight can have only 2 possible values. 0 or 1. 0 represents -1 and 1 represents +1. i.e., instead of storing numbers like 0.72, -0.13, 1.24, -0.56, we will store +1, -1, +1, -1. That is at least 16 times less storage for weights theoretically. 1.58 bit weight In 1.58-bit weights https://arxiv.org/abs/2504.12285v2 ternary bit weights , there are three possible values: {-1, 0, +1}. Theoretically, we need log2 3 bits of information to distinguish between 3 possible states : 2x=3. i.e. log2 3 ≈1.585. Hence the name. For our interpretation and understanding: - 0: the input is effectively ignored for that connection. i.e., no influence at all, which also gives you built-in sparsity. - +1: the input contributes positively to the output. i.e., an increase in the input pushes the weighted sum up. This is “same-direction” influence. - -1: the input contributes negatively. i.e., an increase in the input pushes the weighted sum down. This is “opposite-direction” influence. 1.125-bit weight 1.125-bit applies different logic for compression and calculation. One can’t literally allocate 1.125 bits per weight. The way this works is that the average bit allocation is used when they are packed. For example, it takes 9 bits to store 8 weights; thus, the average number of bits per weight is 1.125. One important distinction here is that bits per weight doesn’t necessarily tell you how many values each individual weight can take. 1.58 has a clean theoretical connection to three states, while 1.125 bpw usually describes the overall encoding/storage scheme. The table below should make a bit clear | Format | Possible basic values | Approx storage / 1B weights | |---|---|---| | FP16 | ~65,536 | 2 GB | | INT8 | 256 | 1 GB | | INT4 | 16 | 500 MB | | Ternary | 3 | ~198 MB at theoretical 1.58 bpw | | 1.125-bit | special packed encoding | ~141 MB | | Binary | 2 | 125 MB | BitNet b1.58 2B4T This is the first open-source, native 1-bit Large Language Model LLM at the 2-billion parameter scale, developed by Microsoft Research. You can find the model on the hugging face https://huggingface.co/microsoft/bitnet-b1.58-2B-4T , it needs a specific inference code called bitnet.cpp https://github.com/microsoft/BitNet . Bonsai models Bonsai 1-bit models https://huggingface.co/collections/prism-ml/bonsai are probably the first set of production-ready binary/ternary models. They have 1.7B https://huggingface.co/prism-ml/Bonsai-1.7B-gguf , 4B https://huggingface.co/prism-ml/Bonsai-4B-gguf , 8B https://huggingface.co/prism-ml/Bonsai-8B-gguf , and 27B https://huggingface.co/prism-ml/Bonsai-27B-gguf parameter models. I have been running the 8B-parameter version locally. My experience with it has been positive. I have explored it for text summaries, code explanations, etc. I will try to install the 27B version and do some coding to see how it goes. Install llama curl -LsSf https://llama.app/install.sh | sh Serve the model using OpenAI-compatible API and a web ui llama serve -hf prism-ml/Bonsai-1.7B-gguf:Q1 0 or 8B parameter version llama serve -hf prism-ml/Bonsai-8B-gguf:Q1 0 Note: Yes, I have moved away from Ollama and now mostly use llama.cpp https://llama.app/ . Also I sometimes Jan.ai https://www.jan.ai/ along with OpenWebUI https://openwebui.com/