Native binary embeddings experiment: curious about your thoughts

I spent a few days testing a simple hypothesis: does training a binary embedding model natively (with a binary loss) produce better retrieval than just binarizing a float model post-hoc?

The setup is deliberately small : bert-mini (~11M params), CPU-only training on a Mac Mini M4 Pro, NLI 550k pairs, 3 epochs.

Key results on SciFact Recall@10:

And at 1M vectors with FAISS (AVX2+POPCNT on x86): The three things that made the binary model actually converge:

Models and code on GitHub / HuggingFace (korben99/bne-binary-2048). Happy to hear if you’ve seen similar or contradictory results, especially at larger scales or with bigger backbones. Also curious whether the 2048-dim sweet spot holds with e.g. MiniLM.