# LittleBit: Miniturising models past known limits. The steps involved in getting an embedding tool 30x and 200x smaller than the base. A better way to compress LLMs, SLMs and embedding tools

> Source: <https://discuss.huggingface.co/t/littlebit-miniturising-models-past-known-limits-the-steps-involved-in-getting-an-embedding-tool-30x-and-200x-smaller-than-the-base-a-better-way-to-compress-llms-slms-and-embedding-tools/192239#post_3>
> Published: 2026-11-21 16:09:51+00:00

**Recommendation:** When retrieval quality matters, use the larger LittlePotion-931KB checkpoint or fuse the small model's scores with BM25F keyword search, which lifted SciFact nDCG@10 from 0.363 to 0.669.
More in the playbook "Retrieval and RAG": <https://wpnews.pro/playbooks/rag-retrieval>

Thanks, this is a really careful write up, and the reproduction is appreciated.

**Quality.** I can confirm your numbers independently. Scoring the 164 KB checkpoint through a separate CPU runtime (not model2vec or pytorch) gives STS-B 0.7165, SICK-R 0.6213 and SciFact 0.3631. NFCorpus comes out at 0.1551 under MTEBs graded scoring, against your 0.1550 binary. So the gap you describe shows for me too: pairwise similarity survives the compression much better than retrieval does. And yes, LittlePotion-931KB is the better pick when retrieval matters. The 164 KB one is a proof of concept.

At this size: the retrieval gap is simply rare term matching. a 2,000 vector shared table can’t do this. Adding a keyword score is cheap (all nDCG@10 + standardised score fusion + weights not tuned)

|  | SciFact | nfcorpus | ArguAna | 
|---|---|---|---|
| 164 KB model alone | 0.363 | 0.155 | 0.288 | 
| BM25F alone | 0.669 | 0.305 | 0.344 | 
| model + BM25F | 0.669 | 0.317 | 0.373 | 

So the only deployment that shows an insignificant advantage is hybrid search, and even then it isn’t worth it imo. Again, as I mentioned, proof of concept. However the larger model scores pretty well, like actually well.

On the GPU idea. I agree with the main point about choosing kernels by batch size, shape and precision. Verifying each one against FP32. Keeping fallback. For embedding models, the operator isn’t the bottleneck. The table can be rebuilt and then just looked up, so a per query cost is just a gather and a mean. On cpu that runs approx 30k-60k texts per second on a single thread, depending on the CPU. As the text is turned into tokens, it’s 256 value roows are looked up and averaged. The texts don’t depend on each other. So the work splits evenly across the cores, and the limits become the tokenizer speed OR memory latency. But the CPU work is what speeds it up, like splitting each text into tokens, which is the largest share. Looking up each token’s 256 runtime numbers in the table also speeds it up. Averaging them, normalising them, etc. etc. etc.

I would imagine a top CPU for a consumer workstation could reach 500k+ words per second and a server 64core easily in the multiple millions (from quick napkin math). One caveat is that looking up each tokens 256 numbers in a table is a memory read, not a calculation, so it scales with cores while the table stays in cache and only when it stays in cache. So the 30mb table (or 7.5mb int8) would have to stay in cache.

Where your batching work really matters is LLMs. I think 90%+ of the original quality at under 1 bpw is realistic for larger models, and even more with further tuning. That’s still a belief, not a result. If it holds, strong models become usable on a 6 GB GPU like the one in my laptop. That’s my goal: even though I have access to better hardware, I want these models to reach more people.

I already have a runtime that runs these embedding models. It works, but it’s messy and badly under-documented, and it needs cleaning up before anyone else can use it. On embeddings, it encodes about 3 to 13× faster than the Python loader I shipped, with identical scores, and I have ideas to push it further. The scripts in the repo are only there so people can try the models today; they aren’t the runtime.

I’ll reply to your GPU points properly when I have more time. BUT for now I will say this: it’s very interesting, and some of those ideas are already in my runtime in some form. I’m happy to message you and show you what I’ve done if you’re interested. A fair amount of it is novel. I plan to release it all as open source, most likely under MIT, or Apache 2.0 if vendored Apache code makes that unavoidable. But I want it cleaned up and documented first, so that what I release is something people can actually use and check for themselves.

Another thing I’ve made: A patch for Llama.cpp that allows it to run Little Qwen, Z, Deepseek, and more.
