# LittleBit: How I made a 164kb encoder at 1bpw with higher accuracy than the original, and how you can make Qwen 27b run on a 6gb card at 0.1bpw

> Source: <https://discuss.huggingface.co/t/littlebit-how-i-made-a-164kb-encoder-at-1bpw-with-higher-accuracy-than-the-original-and-how-you-can-make-qwen-27b-run-on-a-6gb-card-at-0-1bpw/192239#post_1>
> Published: 2026-11-20 21:29:56+00:00

**Recommendation:** For encoders below about 2 MB, quantize with LittleBit at 1.0 BPW and retrain against the batch pairwise similarity matrix rather than per-vector cosine loss, which scored 0.714 versus 0.660 STS-B at 0.55 BPW.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

“LittleBit: Ultra Low-Bit Quantization via Latent Factorization”,

Lee, Kim, You, Kim (Samsung Research), NeurIPS 2025, arXiv:2506.13771.

**Potion-base-8m**: ~30mb original

**LittlePotion-1BPW**: 931kb compressed version: ~98% original accuracy

**

Potion 8m Model2Vec**: ~700kb original

**LittleLittlePotion-164kb**: compressed version: ~130% original accuracy.

Did I do what the original paper did? No. Not exactly.

First I used an embedding model, proving it is possible. I also beat an original version. I retrained but I didn’t only distill from the base - I think it needs retraining based on what it lost, not just an average of the model.

My result is:

Very similar to the 8mb at just 931kb

Better than the 731kb version at 164kb.

Hopefully LaTeX doesn’t mess up here.

The idea is pretty simple and honestly I’m surprised others didn’t think of it earlier. It’s similar to JPEG. Or quantum bits (but not quite).  It’s similar to jpeg in that it:

Starts as patterns, keeps only the important parts, stored crudely, restores magnitudes, and uses variable quality.

Basically it uses the binaries themselves to hold the data. This means it can squeeze a weight into a much smaller space. It’s extremely lossy, but I’ve gotten 98% at 0.55 bpw with retraining and with embedding tools, I’ve gotten BETTER than the original at 1bpw with retraining from a new source. I did this by using my own embedding tool which I plan to release soon, but that’s not the point. You can also use model2vec and get a similar effect.

So you squeeze mutliple characters into a single bit by letting a sequence of bits, for a weight matrix W:

Don’t believe it? Prove it to yourself. Try OPT-125m. You can do this on a bad GPU. Or quickly on a good one. It takes minutes to retrain.

My results after a few mins of retraining:

| BPW | Before training | After training | 
|---|---|---|
| 1.0 | 11,551 | 212 | 
| 0.55 | 24,821 | 268 | 
| 0.3 | 127,546 | 322 | 
| 0.1 | 39,384 | 528 | 

After this matched the paper exactly I thought that we’re on to a winner here.

Model2Vec-style encoders are just a lookup table (token to vector). A sentence embedding is the mean of its token vectors, so the “model” *is* one matrix, and LittleBit applies directly.

STS-B Spearman; the original fp16 is 15.1 MB and scores 0.753:

| Method | Size | STS-B | 
|---|---|---|
| standard 8-bit | 7.8 MB | 0.754 | 
| standard 4-bit | 4.0 MB | 0.753 | 
| standard 2-bit | 2.1 MB | 0.745 | 
| LittleBit 2.0 BPW | 1.9 MB | 0.737 | 
| **LittleBit 1.0 BPW** | **0.95 MB** | **0.737** | 
| LittleBit 0.55 BPW | 0.51 MB | 0.715 | 
| LittleBit 0.3 BPW | 0.29 MB | 0.636 | 
| LittleBit 0.1 BPW (1 path) | 0.12 MB | 0.489 | 

Above 2 MB, use normal quantisation. 4-bit is lossless here, with no training.

Below ~2 MB, LittleBit is the only game in town: 16× smaller than fp16 at ~98% of the score.

The loss function matters a lot. Distilling each embedding to match the teacher’s (cosine loss) hurt STS-B (0.660 at 0.55 BPW). Matching the pairwise similarity matrix within each batch gave 0.714. Mean-pooled vectors share a big common direction, and per-vector matching mostly learns that direction instead of the differences that matter.

A 748 KB vocab-quantised model becomes 164 KB

This model is already compressed: 2,000 shared int8 vectors (2000×256), a token to vector mapping (int32), and a per-token scale weights(fp32). Expanding it to a full table and LittleBit-ing that would make it *bigger*, so I kept the structure:

| Part | Original | Compressed | 
|---|---|---|
| 2000x256 shared vectors | int8, 512 KB | **LittleBit 1.0 BPW** , rank 96, 2 paths, around 64 KB | 
| token to vector mapping | int32, 118 KB | **11-bit packed** (2,000 < 2¹¹), lossless, ~41 KB | 
| per-token scales | fp32, 118 KB | fp16, ~59 KB | 
| **Total** | **748 KB** | **164 KB** | 

Only the shared-vector table is at 1 bit per weight. Over the whole file it works out to ~0.17 bits per entry of the equivalent dense 29,525 x 256 table.

Round 1, distilling from the original: STS-B unchanged (0.724), but SciFact retrieval dropped 11% (0.373 down to 0.333). Lesson learned, check more than one benchmark. Similarity-only training didn’t keep what retrieval needs so I moved on.

Round 2, distilling from a retrieval pipeline. Measured on SciFact (nDCG@10):

| Retrieval setup | nDCG@10 | 
|---|---|
| BM25f results embedded by bge-small into Ettin-32m reranker | 0.732 | 
| BM25F results reranked by Ettin-32m | 0.726 | 
| bge-small alone | 0.713 | 
| BM25F alone | 0.671 | 
| the 748 KB static model alone | 0.373 | 

The re-rankers are far stronger teachers than the static model itself, so the student learns to rank candidates the way the re-ranker does:

**Data:** 5,020 generated real world queries, each with a positive passage and 4 hard negatives. Split 90/5/5, no overlap with SciFact or STS-B.

**Candidates:** the positive, the dataset’s hard negative, and BM25-mined negatives.

Teachers, in 250-step stages: Ettin rerankers 17m to 32m to 68m to 150m (Apache-2.0, cross-encoder/ettin-reranker), then bge-small (MIT).

**Loss:** listwise KL to the teacher’s candidate distribution, in-batch contrastive, and an anchor that keeps sentence similarities close to the original.

**Anti-overfitting:** each stage is kept only if in-domain validation improves, SciFact validation doesn’t drop, and STS-B validation drops ≤ 0.01. Otherwise it’s rolled back.

An earlier non-staged run on Natural Questions beat the original after 500 steps, then over-fit and regressed back down while STS-B fell from 0.72 to 0.65, teaching me why on a tiny model you teach less.

Final test scores:

| Model | Size | In-domain test | SciFact | STS-B | 
|---|---|---|---|---|
| Original | 748 KB | 0.426 | **0.373** | **0.724** | 
| LittleBit, similarity-only | 164 KB | — | 0.333 | 0.724 | 
| **LittleBit, staged rerankers** | **164 KB** | **0.454** | 0.363 | 0.716 | 

All five stages passed the keep rule, and each bigger reranker added a bit more. So the 164 KB model is **better than the original on the kind of queries it was trained for**, and within a few percent elsewhere.

Before getting too excited, right now this wouldn’t be great. Qwen 27b at 1bpw would probably suffer in coding tasks. There’s a lot of work to be done before this would be great. BUT it could be great and I have some ideas how.

Anyway, here’s my analysis:

0.55 BPW: around 1.7 GB of layers.

0.24 BPW: around 0.75 GB.

0.1 BPW: ~0.3 GB.

The embeddings and output layer dominate. The paper keeps them at 16-bit, which adds maybe 3-4GB depending on vocabulary and width.

At standard 4-bit they’re much smaller at 0.4 to 1.3 GB.

Realistic totals with 4-bit embeddings:

~2–3 GB at 0.55 BPW

~1.2–2 GB at 0.24

~0.7–1.6 GB at 0.1

That fits 6 GB with room for the KV cache.

Cost to make it. Retraining needs the original model in memory as teacher (~54 GB in 16-bit), so think 2x 80 GB GPUs (A100/H100) for ~8–36 hours per bit rate. That’s roughly $40–60 per level. These are estimates.

Quality. From the paper’s own results (lower perplexity is better):

QwQ-32B: 6.34 original  13.57 at 0.55 BPW, 16.48 at 0.3, 35.26 at 0.1.

Phi-4 at 0.1 BPW still averages ~44% on common-sense benchmarks.

0.3–0.55 is the sweet spot; at 0.1 it’s a curiosity, not an assistant. At ~1–2 GB, a normal 4-bit 3B model may beat a heavily compressed 27B.

No llama.cpp/Ollama/GGUF support. They have no code for this layer type, so you’d run it through custom PyTorch code which does work for all the models I’ve tried so far (I’ve made six models at 1bpw or under, all at 95%+). Currently, the models would be slow even through custom PyTorch. I know how the runtime would be made to work (I think) and I’m currently running tests to see if it’s feasible. Let me know if you have ideas. If these models are made, I WILL make a custom runtime. In fact, I’ve already made an embedding runtime which I think people will like that’s probably about 75% done now.

My suggested path: prove the method at small scale first to yourself like OPT-1.3B. If you can retrain and newly train, you may get better numbers.

| Part needed | How we can approach it | 
|---|---|
| Weights | LittleBit and around 1 BPW on the transformer layers (~3–4.7 GB) | 
| Embeddings | compressed (palette/LittleBit) or kept in system RAM | 
| Context memory | 4–8-bit KV cache, or LittleBit’s factorized K/V, which the paper says shrinks the cache ~4–20× | 
| Training data | the original Qwen 27B’s own agentic trajectories: run it as an agent on many real coding tasks, keep the successful runs, train the compressed model to reproduce them (tool calls included) AND, importantly, use other data too. | 
| Teacher signal | match the original’s token probabilities on those trajectories, not just the final text | 
| Evaluation | task *success rate* on agentic coding tasks (e.g. a SWE-bench style subset or your own repo tasks), compared directly to the original | 

My potion 8m:

| Model | Size | STS-B | SICK-R | SciFact (nDCG@10) | 
|---|---|---|---|---|
| Original potion-base-8M | 30 MB (fp32) / 15 MB (fp16) | 0.754 | 0.647 | 0.505 | 
| **LittleBit 1 BPW** | **931 KB** | **0.737** | **0.625** | **0.453** | 
| Share of original | ~16× smaller than fp16 | **98%** | **97%** | **90%** |
