Rembed – Pure-Go text embeddings, no ONNX Runtime, no cgo

Rostam Labs released Rembed, a pure-Go text embedding inference engine that runs BERT-style and decoder-derived embedding models without ONNX Runtime or cgo, achieving statistical parity with ONNX Runtime and faster int8 performance. The library loads models directly from Hugging Face Hub, supports eight architectures including ModernBERT and EmbeddingGemma, and offers features like int8 quantization and Matryoshka dimension truncation.

Pure-Go embedding inference engine for text-embedding models — BERT-style encoders and decoder-derived embedders. Text in, L2-normalized embedding vectors out — no cgo, no ONNX Runtime, one static binary. 📚 Documentation — Go library, CLI & server, Python/C bindings, supported models, architecture, and benchmarks. // Loads straight from the Hugging Face Hub pure Go, cached locally — // no Python, no conversion step: emb, err := rembed.Load "sentence-transformers/all-MiniLM-L6-v2" vecs, err := emb.Embed ctx, string{"hello world"} // vecs 0 is a float32 of emb.Dim 384 for MiniLM-L6-v2 EmbedTokens returns per-token hidden states ONNX Runtime's last hidden state for rerankers and late-interaction retrieval, and a multi-text Embed call fans out across texts for near-linear batch throughput bit-identical to one-at-a-time results . Load accepts a Hub model id downloaded into $REMBED CACHE , default the user cache dir; HF TOKEN honored , a git-cloned HF repo directory, or a converted model dir. Options: rembed.WithInt8 weight-only quantization, ~4× less weight traffic, cosine ≥ 0.999 vs fp32 , rembed.WithWorkers n CPU cap for servers , and rembed.WithDim d Matryoshka: truncate to d dims and re-normalize — EmbeddingGemma 768→512/256/128 — for cheaper storage and search; CLI -dim . Status: the optimization ladder is complete — naive baseline to statistical parity with and, with int8, consistently ahead of ONNX Runtime on the reference laptop: ~45× → 0.89× across six rungs, every step measured against a golden ONNX reference within 1e-4 int8: cosine ≥ 0.999 . See DESIGN.md /rostamlabs/rembed/blob/main/DESIGN.md for the architecture and bench/RESULTS.md /rostamlabs/rembed/blob/main/bench/RESULTS.md for the full measured ladder, including the failed experiments. Weight-only int8 is opt-in via rembed.WithInt8 ; rembed.WithWorkers n caps per-call CPU for throughput-saturated servers. Eight architectures: BERT-family, DistilBERT, MPNet, RoBERTa including XLM-RoBERTa — multilingual-e5-base / -large , bge-m3 , the same encoder with the SentencePiece tokenizer , ModernBERT, and nomic-embed a post-norm BERT with RoPE + SwiGLU encoders, plus two decoder-derived embedders: Qwen3-Embedding a causal decoder and EmbeddingGemma a bidirectional Gemma 3 backbone — the current MMTEB state of the art for its size . sentence-transformers format: mean, CLS, or last-token pooling, with an optional Dense projection head EmbeddingGemma ; WordPiece, byte-level BPE, SentencePiece Unigram the XLM-R tokenizer — multilingual models work, 100+ languages , or the Gemma byte-fallback BPE; absolute positions plus MPNet's bucketed relative-position bias OR rotary positions RoPE, single- or dual-theta — ModernBERT, Qwen3, EmbeddingGemma ; alternating global/local sliding-window attention ModernBERT, EmbeddingGemma , full causal attention Qwen3 , or bidirectional attention with grouped-query attention and QK-norm Qwen3, EmbeddingGemma ; exact GELU, tanh-GELU, GeGLU, and SwiGLU; LayerNorm and RMSNorm unit-offset for Gemma ; F32/F16/BF16 safetensors. Validated end-to-end against each model's own ONNX Runtime reference ModernBERT and Qwen3 against the canonical PyTorch ModernBertModel / Qwen3Model , since their ONNX exports bundle or omit the pooling rembed reproduces; XLM-RoBERTa against PyTorch XLMRobertaModel , and EmbeddingGemma against PyTorch Gemma3TextModel with the sentence-transformers pool+Dense+normalize head, since neither reliably ships ONNX : | model | pooling | dtype | fp32 vs ONNX | int8 | |---|---|---|---|---| | sentence-transformers/all-MiniLM-L6-v2 | mean | F32 | 1.5e-7 | cosine ≥ 0.9991 | | sentence-transformers/all-MiniLM-L12-v2 | mean | F32 | 1.9e-7 | in bounds | | sentence-transformers/paraphrase-MiniLM-L3-v2 | mean | F32 | < 1e-4 | — | | BAAI/bge-small-en-v1.5 | cls | F32 | < 1e-4 | in bounds | | sentence-transformers/all-mpnet-base-v2 | mean | F32 | 3.3e-7 | cosine ≥ 0.9978 | | sentence-transformers/all-distilroberta-v1 | mean | F32 | 3.2e-7 | cosine ≥ 0.9985 | | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | mean | F32 | 7e-7 | cosine ≥ 0.9995 | | intfloat/multilingual-e5-small | mean | F32 | 1.7e-7 | cosine ≥ 0.9995 | | intfloat/multilingual-e5-base | mean | F32 | 2.9e-7 vs PyTorch | cosine ≥ 0.999 | | BAAI/bge-base-en-v1.5 | cls | F32 | 7.4e-7 | cosine ≥ 0.995 | | thenlper/gte-base | mean | F32 | 3.8e-6 | cosine ≥ 0.988 | | sentence-transformers/paraphrase-mpnet-base-v2 | mean | F32 | 1.2e-6 | cosine ≥ 0.9945 | | sentence-transformers/multi-qa-MiniLM-L6-cos-v1 | mean | F32 | 2.1e-7 | cosine ≥ 0.998 | | Snowflake/snowflake-arctic-embed-s | cls | F32 | 2.5e-7 | cosine ≥ 0.995 | | sentence-transformers/multi-qa-distilbert-cos-v1 | mean | F32 | 2.5e-7 | cosine ≥ 0.999 | | nomic-ai/modernbert-embed-base | mean | F32 | < 1e-4 vs PyTorch | cosine ≥ 0.998 | | Qwen/Qwen3-Embedding-0.6B | lasttoken | BF16 | < 1e-4 vs PyTorch | cosine ≥ 0.997 | | google/embeddinggemma-300m | mean + Dense | F32 | < 1e-4 vs PyTorch | cosine ≥ 0.998 | | nomic-ai/nomic-embed-text-v1.5 | mean | F32 | < 1e-4 | cosine ≥ 0.996 | | thenlper/gte-small | mean | F16 | 2e-3 maxAbs + cosine ≥ 0.9999 + meanAbs ≤ 2e-4 the repo's ONNX export is fp32 while its safetensors are f16, so maxAbs is dominated by the checkpoint's own rounding; the cosine/mean bounds are what actually constrain rembed | — | On CPUs with AVX-VNNI — Intel Alder Lake 2021 onward and Sapphire Rapids+ servers, AMD Zen 5+; note that AVX-512-VNNI-only parts like Ice Lake-SP and Zen 4 do NOT have it — WithInt8Activations selects full int8 inference u8 activations × s8 weights via VPDPBUSD for a further ~1.3× over weight-only int8. The accuracy trade is real, PER-MODEL, and test-enforced worst golden cosine, full int8 vs weight-only : | model | full int8 | weight-only int8 | |---|---|---| | MiniLM-L6 / L12 / L3 | 0.9917 / 0.9932 / 0.9979 | ≥ 0.9990 | | mpnet-base / paraphrase-mpnet | 0.9912 / 0.9867 | ≥ 0.9945 | | multilingual MiniLM / multilingual-e5 | 0.9982 / 0.9988 | ≥ 0.9995 | | multilingual-e5-base xlm-roberta | 0.9849 | 0.9992 | | gte-small / gte-base | 0.9991 / 0.9741 | ≥ 0.9880 | | multi-qa MiniLM / distilbert | 0.9949 / 0.9854 | ≥ 0.9940 | | arctic-embed-s | 0.9932 | 0.9953 | | distilroberta | 0.9747 | 0.9987 | | modernbert-embed | 0.9660 | 0.9984 | | qwen3-embedding-0.6B | 0.9747 | 0.9978 | | embeddinggemma-300m | 0.9938 | 0.9981 | | nomic-embed-text-v1.5 | 0.9530 | 0.9968 | bge-base | 0.9593 | 0.9957 | Activation outliers are a PER-CHECKPOINT property, not an architecture one: nomic-embed SwiGLU, and an un-normalized output measures worst at 0.9530 and bge-base a plain BERT next at 0.9593, below distilroberta's 0.9747 and modernbert-embed's 0.9660 whose GeGLU gate activations have a range the per-row u8 scale can't hold , while bge-base's sibling bge-small is unremarkable. Qwen3-Embedding compounds this: last-token pooling reads a single position, so there is no averaging across tokens to soften activation-quantization error — prefer WithInt8 weight-only there. Check the table before enabling full int8 for a model — anything below ~0.99 is a real retrieval-quality risk — and prefer WithInt8 weight-only, ≥ 0.988 everywhere when in doubt. Every figure above is enforced in the golden matrix. Cross-engine, measured on a Zen 4 cloud box with a both-orders/median protocol bench/RESULTS.md has the full data and every noise flag : rembed fp32 sits at parity with ONNX Runtime fp32, and rembed full int8 beat ORT fp32 in every round — the flag-free rounds measured 0.70× and 0.75× 5.9 ms vs 7.9 ms on mpnet — while trading blows at parity with ORT's own AVX-512-VNNI int8 graphs. Disk-backed weights run larger than RAM . WithDiskWeights memory-maps the weights from a pack file instead of loading them into RAM: the OS pages weights in on access and evicts under pressure, so resident memory tracks the working set and a model larger than RAM runs disk-bandwidth-bound when it does not fit, full speed with a warm page cache when it does — the same trade ORT's mmap mode makes . On first use the safetensors single-file or sharded are streamed to a pack file one tensor at a time, so even the pack step fits a small box. This is what lets Qwen3-Embedding-4B run cgo-free on a laptop that cannot hold it in fp32 RAM. Close the Embedder to unmap. Numerics are unchanged — only where the bytes live. Currently wired for qwen3. Expected compatible same architecture, no committed golden yet : the remaining e5 sizes, the largest BGE/GTE variants, the msmarco families, other BERT/DistilBERT-based sentence-transformers checkpoints, and the larger Qwen3-Embedding sizes 4B/8B — same architecture, far larger . Caveat for retrieval models: e5 requires "query: "/"passage: " prefixes, Qwen3-Embedding expects an instruction on queries only "Instruct: {task}\nQuery:{text}", with documents left bare , and some models e.g. arctic declare prompt handling in their pooling config — rembed embeds exactly the text you pass and does not add prefixes; add them yourself or retrieval quality silently degrades. Validated models in that category: the e5 family "query: "/"passage: " , bge "Represent this sentence for searching relevant passages: " on queries , and arctic-embed its own query prefix — rembed embeds exactly the text you pass. One deliberate tokenizer divergence: on NFD decomposed Hangul/kana — routine output from macOS — HF's fast tokenizer skips ≥6-byte grapheme clusters during normalization and shreds Korean into jamo; rembed matches the sentencepiece C++ reference instead, which composes NFD back so decomposed and composed text embed identically. 65k-input fuzzing against the reference: zero mismatches. The validation harness's golden files come from ONNX Runtime in Python ModernBERT from the canonical PyTorch ModernBertModel instead; this is a dev-time tool; users never need it : cd models python3 -m venv .venv && .venv/bin/pip install -r requirements.txt .venv/bin/python convert.py sentence-transformers/all-MiniLM-L6-v2 The same in-process engine is callable from Python through a C-shared build ctypes, ~µs call overhead; the Go library itself stays cgo-free — the shared object is a separate artifact for foreign callers : python/build.sh needs a C toolchain; produces python/rembed/librembed.so python import sys; sys.path.insert 0, "python" from rembed import Embedder emb = Embedder "models/all-MiniLM-L6-v2" fp32 emb = Embedder "models/all-MiniLM-L6-v2", int8=True weight-only int8 vecs = emb.embed "hello world" n, dim float32 numpy Validated against the same golden reference as the Go tests python/test rembed.py ; vectors cross the ABI bit-identically. go run ./cmd/rembed embed -model models/all-MiniLM-L6-v2 "some text" go run ./cmd/rembed validate -model models/all-MiniLM-L6-v2 go run ./cmd/rembed bench -model models/all-MiniLM-L6-v2 Apache-2.0