cd /news/machine-learning/littlebit-how-i-made-a-164kb-encoder… · home › topics › machine-learning › article
[ARTICLE · art-173493] src=discuss.huggingface.co ↗ pub= topic=machine-learning verified=true sentiment=↑ positive

LittleBit: How I made a 164kb encoder at 1bpw with higher accuracy than the original, and how you can make Qwen 27b run on a 6gb card at 0.1bpw

A developer's write-up of the NeurIPS 2025 paper "LittleBit: Ultra Low-Bit Quantization via Latent Factorization" (Lee, Kim, You, Kim, Samsung Research, arXiv:2506.13771) reports that the technique compresses a 748 KB Model2Vec-style Potion-base-8m encoder to 164 KB while scoring about 130% of the original's accuracy, and that a 931 KB 1-bit-per-weight version retains roughly 98% of the 30 MB original's accuracy. On STS-B Spearman, the author reports LittleBit at 1.0 BPW reaching 0.737 from a 0.95 MB file versus 0.753 for the 15.1 MB fp16 original, and states that retraining OPT-125m after quantization cut perplexity from 11,551 to 212 at 1.0 BPW and from 39,384 to 528 at 0.1 BPW. The author claims the approach could let Qwen 27B run on a 6 GB card at 0.1 BPW, and reports that matching the pairwise similarity matrix within each batch (0.714 STS-B at 0.55 BPW) beat per-vector cosine distillation (0.660).

recommendation

For encoders below about 2 MB, quantize with LittleBit at 1.0 BPW and retrain against the batch pairwise similarity matrix rather than per-vector cosine loss, which scored 0.714 versus 0.660 STS-B at 0.55 BPW.

more in playbook: Inference and self-hosting
read8 min views1 publishedNov 20, 2026

“LittleBit: Ultra Low-Bit Quantization via Latent Factorization”,

Lee, Kim, You, Kim (Samsung Research), NeurIPS 2025, arXiv:2506.13771.

Potion-base-8m: ~30mb original LittlePotion-1BPW: 931kb compressed version: ~98% original accuracy

**

Potion 8m Model2Vec**: ~700kb original

LittleLittlePotion-164kb: compressed version: ~130% original accuracy.

Did I do what the original paper did? No. Not exactly.

First I used an embedding model, proving it is possible. I also beat an original version. I retrained but I didn’t only distill from the base - I think it needs retraining based on what it lost, not just an average of the model.

My result is:

Very similar to the 8mb at just 931kb

Better than the 731kb version at 164kb.

Hopefully LaTeX doesn’t mess up here.

The idea is pretty simple and honestly I’m surprised others didn’t think of it earlier. It’s similar to JPEG. Or quantum bits (but not quite). It’s similar to jpeg in that it:

Starts as patterns, keeps only the important parts, stored crudely, restores magnitudes, and uses variable quality.

Basically it uses the binaries themselves to hold the data. This means it can squeeze a weight into a much smaller space. It’s extremely lossy, but I’ve gotten 98% at 0.55 bpw with retraining and with embedding tools, I’ve gotten BETTER than the original at 1bpw with retraining from a new source. I did this by using my own embedding tool which I plan to release soon, but that’s not the point. You can also use model2vec and get a similar effect.

So you squeeze mutliple characters into a single bit by letting a sequence of bits, for a weight matrix W:

Don’t believe it? Prove it to yourself. Try OPT-125m. You can do this on a bad GPU. Or quickly on a good one. It takes minutes to retrain.

My results after a few mins of retraining:

BPW Before training After training
1.0 11,551 212
0.55 24,821 268
0.3 127,546 322
0.1 39,384 528

After this matched the paper exactly I thought that we’re on to a winner here.

Model2Vec-style encoders are just a lookup table (token to vector). A sentence embedding is the mean of its token vectors, so the “model” is one matrix, and LittleBit applies directly.

STS-B Spearman; the original fp16 is 15.1 MB and scores 0.753:

Method Size STS-B
standard 8-bit 7.8 MB 0.754
standard 4-bit 4.0 MB 0.753
standard 2-bit 2.1 MB 0.745
LittleBit 2.0 BPW 1.9 MB 0.737
LittleBit 1.0 BPW 0.95 MB 0.737
LittleBit 0.55 BPW 0.51 MB 0.715
LittleBit 0.3 BPW 0.29 MB 0.636
LittleBit 0.1 BPW (1 path) 0.12 MB 0.489

Above 2 MB, use normal quantisation. 4-bit is lossless here, with no training.

Below ~2 MB, LittleBit is the only game in town: 16× smaller than fp16 at ~98% of the score.

The loss function matters a lot. Distilling each embedding to match the teacher’s (cosine loss) hurt STS-B (0.660 at 0.55 BPW). Matching the pairwise similarity matrix within each batch gave 0.714. Mean-pooled vectors share a big common direction, and per-vector matching mostly learns that direction instead of the differences that matter.

A 748 KB vocab-quantised model becomes 164 KB

This model is already compressed: 2,000 shared int8 vectors (2000×256), a token to vector mapping (int32), and a per-token scale weights(fp32). Expanding it to a full table and LittleBit-ing that would make it bigger, so I kept the structure:

Part Original Compressed
2000x256 shared vectors int8, 512 KB LittleBit 1.0 BPW , rank 96, 2 paths, around 64 KB
token to vector mapping int32, 118 KB 11-bit packed (2,000 < 2¹¹), lossless, ~41 KB
per-token scales fp32, 118 KB fp16, ~59 KB
Total 748 KB 164 KB

Only the shared-vector table is at 1 bit per weight. Over the whole file it works out to ~0.17 bits per entry of the equivalent dense 29,525 x 256 table.

Round 1, distilling from the original: STS-B unchanged (0.724), but SciFact retrieval dropped 11% (0.373 down to 0.333). Lesson learned, check more than one benchmark. Similarity-only training didn’t keep what retrieval needs so I moved on.

Round 2, distilling from a retrieval pipeline. Measured on SciFact (nDCG@10):

Retrieval setup nDCG@10
BM25f results embedded by bge-small into Ettin-32m reranker 0.732
BM25F results reranked by Ettin-32m 0.726
bge-small alone 0.713
BM25F alone 0.671
the 748 KB static model alone 0.373

The re-rankers are far stronger teachers than the static model itself, so the student learns to rank candidates the way the re-ranker does:

Data: 5,020 generated real world queries, each with a positive passage and 4 hard negatives. Split 90/5/5, no overlap with SciFact or STS-B.

Candidates: the positive, the dataset’s hard negative, and BM25-mined negatives.

Teachers, in 250-step stages: Ettin rerankers 17m to 32m to 68m to 150m (Apache-2.0, cross-encoder/ettin-reranker), then bge-small (MIT). Loss: listwise KL to the teacher’s candidate distribution, in-batch contrastive, and an anchor that keeps sentence similarities close to the original.

Anti-overfitting: each stage is kept only if in-domain validation improves, SciFact validation doesn’t drop, and STS-B validation drops ≤ 0.01. Otherwise it’s rolled back.

An earlier non-staged run on Natural Questions beat the original after 500 steps, then over-fit and regressed back down while STS-B fell from 0.72 to 0.65, teaching me why on a tiny model you teach less.

Final test scores:

Model Size In-domain test SciFact STS-B
Original 748 KB 0.426 0.373 0.724
LittleBit, similarity-only 164 KB — 0.333 0.724
LittleBit, staged rerankers 164 KB 0.454 0.363 0.716

All five stages passed the keep rule, and each bigger reranker added a bit more. So the 164 KB model is better than the original on the kind of queries it was trained for, and within a few percent elsewhere.

Before getting too excited, right now this wouldn’t be great. Qwen 27b at 1bpw would probably suffer in coding tasks. There’s a lot of work to be done before this would be great. BUT it could be great and I have some ideas how.

Anyway, here’s my analysis:

0.55 BPW: around 1.7 GB of layers.

0.24 BPW: around 0.75 GB.

0.1 BPW: ~0.3 GB.

The embeddings and output layer dominate. The paper keeps them at 16-bit, which adds maybe 3-4GB depending on vocabulary and width.

At standard 4-bit they’re much smaller at 0.4 to 1.3 GB.

Realistic totals with 4-bit embeddings: ~2–3 GB at 0.55 BPW

~1.2–2 GB at 0.24

~0.7–1.6 GB at 0.1

That fits 6 GB with room for the KV cache.

Cost to make it. Retraining needs the original model in memory as teacher (~54 GB in 16-bit), so think 2x 80 GB GPUs (A100/H100) for ~8–36 hours per bit rate. That’s roughly $40–60 per level. These are estimates.

Quality. From the paper’s own results (lower perplexity is better):

QwQ-32B: 6.34 original 13.57 at 0.55 BPW, 16.48 at 0.3, 35.26 at 0.1. Phi-4 at 0.1 BPW still averages ~44% on common-sense benchmarks.

0.3–0.55 is the sweet spot; at 0.1 it’s a curiosity, not an assistant. At ~1–2 GB, a normal 4-bit 3B model may beat a heavily compressed 27B.

No llama.cpp/Ollama/GGUF support. They have no code for this layer type, so you’d run it through custom PyTorch code which does work for all the models I’ve tried so far (I’ve made six models at 1bpw or under, all at 95%+). Currently, the models would be slow even through custom PyTorch. I know how the runtime would be made to work (I think) and I’m currently running tests to see if it’s feasible. Let me know if you have ideas. If these models are made, I WILL make a custom runtime. In fact, I’ve already made an embedding runtime which I think people will like that’s probably about 75% done now.

My suggested path: prove the method at small scale first to yourself like OPT-1.3B. If you can retrain and newly train, you may get better numbers.

Part needed How we can approach it
Weights LittleBit and around 1 BPW on the transformer layers (~3–4.7 GB)
Embeddings compressed (palette/LittleBit) or kept in system RAM
Context memory 4–8-bit KV cache, or LittleBit’s factorized K/V, which the paper says shrinks the cache ~4–20×
Training data the original Qwen 27B’s own agentic trajectories: run it as an agent on many real coding tasks, keep the successful runs, train the compressed model to reproduce them (tool calls included) AND, importantly, use other data too.
Teacher signal match the original’s token probabilities on those trajectories, not just the final text
Evaluation task success rate on agentic coding tasks (e.g. a SWE-bench style subset or your own repo tasks), compared directly to the original

My potion 8m:

| Model | Size | STS-B | SICK-R | SciFact (nDCG@10) | 
|---|---|---|---|---|
| Original potion-base-8M | 30 MB (fp32) / 15 MB (fp16) | 0.754 | 0.647 | 0.505 | 

| LittleBit 1 BPW | 931 KB | 0.737 | 0.625 | 0.453 | | Share of original | ~16× smaller than fp16 | 98% | 97% | 90% |

── more in #machine-learning 4 stories · sorted by recency
── more on @littlebit 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/littlebit-how-i-made…] indexed:0 read:8min 2026-11-20 · —