{"slug": "rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo", "title": "Rembed – Pure-Go text embeddings, no ONNX Runtime, no cgo", "summary": "Rostam Labs released Rembed, a pure-Go text embedding inference engine that runs BERT-style and decoder-derived embedding models without ONNX Runtime or cgo, achieving statistical parity with ONNX Runtime and faster int8 performance. The library loads models directly from Hugging Face Hub, supports eight architectures including ModernBERT and EmbeddingGemma, and offers features like int8 quantization and Matryoshka dimension truncation.", "body_md": "Pure-Go embedding inference engine for text-embedding models — BERT-style encoders and decoder-derived embedders. Text in, L2-normalized embedding vectors out — no cgo, no ONNX Runtime, one static binary.\n\n📚 ** Documentation** — Go library, CLI & server, Python/C\nbindings, supported models, architecture, and benchmarks.\n\n```\n// Loads straight from the Hugging Face Hub (pure Go, cached locally) —\n// no Python, no conversion step:\nemb, err := rembed.Load(\"sentence-transformers/all-MiniLM-L6-v2\")\nvecs, err := emb.Embed(ctx, []string{\"hello world\"})\n// vecs[0] is a []float32 of emb.Dim() (384 for MiniLM-L6-v2)\n```\n\n`EmbedTokens`\n\nreturns per-token hidden states (ONNX Runtime's\n`last_hidden_state`\n\n) for rerankers and late-interaction retrieval, and a\nmulti-text `Embed`\n\ncall fans out across texts for near-linear batch\nthroughput (bit-identical to one-at-a-time results).\n\n`Load`\n\naccepts a Hub model id (downloaded into `$REMBED_CACHE`\n\n, default\nthe user cache dir; `HF_TOKEN`\n\nhonored), a git-cloned HF repo directory,\nor a converted model dir. Options: `rembed.WithInt8()`\n\n(weight-only\nquantization, ~4× less weight traffic, cosine ≥ 0.999 vs fp32),\n`rembed.WithWorkers(n)`\n\n(CPU cap for servers), and `rembed.WithDim(d)`\n\n(Matryoshka: truncate to d dims and re-normalize — EmbeddingGemma\n768→512/256/128 — for cheaper storage and search; CLI `-dim`\n\n).\n\nStatus: the optimization ladder is complete — naive baseline to\n**statistical parity with (and, with int8, consistently ahead of) ONNX\nRuntime** on the reference laptop: ~45× → 0.89× across six rungs, every\nstep measured against a golden ONNX reference within 1e-4 (int8: cosine\n≥ 0.999). See [DESIGN.md](/rostamlabs/rembed/blob/main/DESIGN.md) for the architecture and\n[bench/RESULTS.md](/rostamlabs/rembed/blob/main/bench/RESULTS.md) for the full measured ladder,\nincluding the failed experiments. Weight-only int8 is opt-in via\n`rembed.WithInt8()`\n\n; `rembed.WithWorkers(n)`\n\ncaps per-call CPU for\nthroughput-saturated servers.\n\nEight architectures: BERT-family, DistilBERT, MPNet, RoBERTa (including\n**XLM-RoBERTa** — `multilingual-e5-base`\n\n/`-large`\n\n, `bge-m3`\n\n, the same\nencoder with the SentencePiece tokenizer), ModernBERT, and **nomic-embed**\n(a post-norm BERT with RoPE + SwiGLU) encoders, plus two decoder-derived\nembedders: **Qwen3-Embedding** (a causal decoder) and **EmbeddingGemma**\n(a bidirectional Gemma 3 backbone — the current MMTEB state of the art for\nits size). sentence-transformers format: mean, CLS,\nor last-token pooling, with an optional Dense projection head (EmbeddingGemma);\nWordPiece, byte-level BPE, SentencePiece Unigram (the XLM-R tokenizer —\nmultilingual models work, 100+ languages), or the Gemma byte-fallback BPE;\nabsolute positions\n(plus MPNet's bucketed relative-position bias) OR rotary positions (RoPE,\nsingle- or dual-theta — ModernBERT, Qwen3, EmbeddingGemma); alternating\nglobal/local sliding-window attention (ModernBERT, EmbeddingGemma), full\ncausal attention (Qwen3), or bidirectional attention with grouped-query\nattention and QK-norm (Qwen3, EmbeddingGemma); exact GELU, tanh-GELU,\nGeGLU, and SwiGLU; LayerNorm and RMSNorm (unit-offset for Gemma);\nF32/F16/BF16 safetensors. Validated end-to-end against each model's own\nONNX Runtime reference (ModernBERT and Qwen3 against the canonical PyTorch\n`ModernBertModel`\n\n/ `Qwen3Model`\n\n, since their ONNX exports bundle or omit\nthe pooling rembed reproduces; XLM-RoBERTa against PyTorch\n`XLMRobertaModel`\n\n, and EmbeddingGemma against PyTorch `Gemma3TextModel`\n\nwith the sentence-transformers pool+Dense+normalize head, since neither\nreliably ships ONNX):\n\n| model | pooling | dtype | fp32 vs ONNX | int8 |\n|---|---|---|---|---|\n| sentence-transformers/all-MiniLM-L6-v2 | mean | F32 | 1.5e-7 | cosine ≥ 0.9991 |\n| sentence-transformers/all-MiniLM-L12-v2 | mean | F32 | 1.9e-7 | in bounds |\n| sentence-transformers/paraphrase-MiniLM-L3-v2 | mean | F32 | < 1e-4 | — |\n| BAAI/bge-small-en-v1.5 | cls | F32 | < 1e-4 | in bounds |\n| sentence-transformers/all-mpnet-base-v2 | mean | F32 | 3.3e-7 | cosine ≥ 0.9978 |\n| sentence-transformers/all-distilroberta-v1 | mean | F32 | 3.2e-7 | cosine ≥ 0.9985 |\n| sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | mean | F32 | 7e-7 | cosine ≥ 0.9995 |\n| intfloat/multilingual-e5-small | mean | F32 | 1.7e-7 | cosine ≥ 0.9995 |\n| intfloat/multilingual-e5-base | mean | F32 | 2.9e-7 (vs PyTorch) | cosine ≥ 0.999 |\n| BAAI/bge-base-en-v1.5 | cls | F32 | 7.4e-7 | cosine ≥ 0.995 |\n| thenlper/gte-base | mean | F32 | 3.8e-6 | cosine ≥ 0.988 |\n| sentence-transformers/paraphrase-mpnet-base-v2 | mean | F32 | 1.2e-6 | cosine ≥ 0.9945 |\n| sentence-transformers/multi-qa-MiniLM-L6-cos-v1 | mean | F32 | 2.1e-7 | cosine ≥ 0.998 |\n| Snowflake/snowflake-arctic-embed-s | cls | F32 | 2.5e-7 | cosine ≥ 0.995 |\n| sentence-transformers/multi-qa-distilbert-cos-v1 | mean | F32 | 2.5e-7 | cosine ≥ 0.999 |\n| nomic-ai/modernbert-embed-base | mean | F32 | < 1e-4 (vs PyTorch) | cosine ≥ 0.998 |\n| Qwen/Qwen3-Embedding-0.6B | lasttoken | BF16 | < 1e-4 (vs PyTorch) | cosine ≥ 0.997 |\n| google/embeddinggemma-300m | mean + Dense | F32 | < 1e-4 (vs PyTorch) | cosine ≥ 0.998 |\n| nomic-ai/nomic-embed-text-v1.5 | mean | F32 | < 1e-4 | cosine ≥ 0.996 |\n| thenlper/gte-small | mean | F16 | 2e-3 maxAbs + cosine ≥ 0.9999 + meanAbs ≤ 2e-4 (the repo's ONNX export is fp32 while its safetensors are f16, so maxAbs is dominated by the checkpoint's own rounding; the cosine/mean bounds are what actually constrain rembed) | — |\n\nOn CPUs with AVX-VNNI — Intel Alder Lake (2021) onward and Sapphire\nRapids+ servers, AMD Zen 5+; note that AVX-512-VNNI-only parts like Ice\nLake-SP and Zen 4 do NOT have it — `WithInt8Activations`\n\nselects full\nint8 inference (u8 activations × s8 weights via VPDPBUSD) for a further\n~1.3× over weight-only int8. The accuracy trade is real, PER-MODEL, and\ntest-enforced (worst golden cosine, full int8 vs weight-only):\n\n| model | full int8 | weight-only int8 |\n|---|---|---|\n| MiniLM-L6 / L12 / L3 | 0.9917 / 0.9932 / 0.9979 | ≥ 0.9990 |\n| mpnet-base / paraphrase-mpnet | 0.9912 / 0.9867 | ≥ 0.9945 |\n| multilingual MiniLM / multilingual-e5 | 0.9982 / 0.9988 | ≥ 0.9995 |\n| multilingual-e5-base (xlm-roberta) | 0.9849 | 0.9992 |\n| gte-small / gte-base | 0.9991 / 0.9741 | ≥ 0.9880 |\n| multi-qa MiniLM / distilbert | 0.9949 / 0.9854 | ≥ 0.9940 |\n| arctic-embed-s | 0.9932 | 0.9953 |\n| distilroberta | 0.9747 | 0.9987 |\n| modernbert-embed | 0.9660 | 0.9984 |\n| qwen3-embedding-0.6B | 0.9747 | 0.9978 |\n| embeddinggemma-300m | 0.9938 | 0.9981 |\n| nomic-embed-text-v1.5 | 0.9530 | 0.9968 |\nbge-base |\n0.9593 |\n0.9957 |\n\nActivation outliers are a PER-CHECKPOINT property, not an architecture\none: nomic-embed (SwiGLU, and an un-normalized output) measures worst at\n0.9530 and bge-base (a plain BERT) next at 0.9593, below distilroberta's\n0.9747 and modernbert-embed's 0.9660 (whose GeGLU gate activations have a\nrange the per-row u8 scale can't hold), while bge-base's sibling bge-small\nis unremarkable. Qwen3-Embedding compounds\nthis: last-token pooling reads a single position, so there is no\naveraging across tokens to soften activation-quantization error — prefer\n`WithInt8`\n\n(weight-only) there.\nCheck the table before enabling full int8 for a model — anything below\n~0.99 is a real retrieval-quality risk — and prefer `WithInt8`\n\n(weight-only, ≥ 0.988 everywhere) when in doubt. Every figure above is\nenforced in the golden matrix.\n\nCross-engine, measured on a Zen 4 cloud box with a both-orders/median protocol (bench/RESULTS.md has the full data and every noise flag): rembed fp32 sits at parity with ONNX Runtime fp32, and rembed full int8 beat ORT fp32 in every round — the flag-free rounds measured 0.70× and 0.75× (5.9 ms vs 7.9 ms on mpnet) — while trading blows at parity with ORT's own AVX-512-VNNI int8 graphs.\n\n**Disk-backed weights (run larger than RAM).** `WithDiskWeights()`\n\nmemory-maps the weights from a pack file instead of loading them into RAM:\nthe OS pages weights in on access and evicts under pressure, so resident\nmemory tracks the working set and a model larger than RAM runs\n(disk-bandwidth-bound when it does not fit, full speed with a warm page\ncache when it does — the same trade ORT's mmap mode makes). On first use\nthe safetensors (single-file or sharded) are streamed to a pack file one\ntensor at a time, so even the pack step fits a small box. This is what\nlets Qwen3-Embedding-4B run cgo-free on a laptop that cannot hold it in\nfp32 RAM. Close the Embedder to unmap. Numerics are unchanged — only\nwhere the bytes live. (Currently wired for qwen3.)\n\nExpected compatible (same architecture, no committed golden yet): the remaining e5 sizes, the largest BGE/GTE variants, the msmarco families, other BERT/DistilBERT-based sentence-transformers checkpoints, and the larger Qwen3-Embedding sizes (4B/8B — same architecture, far larger). Caveat for retrieval models: e5 requires \"query: \"/\"passage: \" prefixes, Qwen3-Embedding expects an instruction on queries only (\"Instruct: {task}\\nQuery:{text}\", with documents left bare), and some models (e.g. arctic) declare prompt handling in their pooling config — rembed embeds exactly the text you pass and does not add prefixes; add them yourself or retrieval quality silently degrades. Validated models in that category: the e5 family (\"query: \"/\"passage: \"), bge (\"Represent this sentence for searching relevant passages: \" on queries), and arctic-embed (its own query prefix) — rembed embeds exactly the text you pass.\n\nOne deliberate tokenizer divergence: on NFD (decomposed) Hangul/kana — routine output from macOS — HF's fast tokenizer skips ≥6-byte grapheme clusters during normalization and shreds Korean into jamo; rembed matches the sentencepiece C++ reference instead, which composes NFD back so decomposed and composed text embed identically. 65k-input fuzzing against the reference: zero mismatches.\n\nThe validation harness's golden files come from ONNX Runtime in Python\n(ModernBERT from the canonical PyTorch `ModernBertModel`\n\ninstead; this is\na dev-time tool; users never need it):\n\n```\ncd models\npython3 -m venv .venv && .venv/bin/pip install -r requirements.txt\n.venv/bin/python convert.py sentence-transformers/all-MiniLM-L6-v2\n```\n\nThe same in-process engine is callable from Python through a C-shared build (ctypes, ~µs call overhead; the Go library itself stays cgo-free — the shared object is a separate artifact for foreign callers):\n\n```\npython/build.sh   # needs a C toolchain; produces python/rembed/librembed.so\npython\nimport sys; sys.path.insert(0, \"python\")\nfrom rembed import Embedder\n\nemb = Embedder(\"models/all-MiniLM-L6-v2\")           # fp32\nemb = Embedder(\"models/all-MiniLM-L6-v2\", int8=True)  # weight-only int8\nvecs = emb.embed([\"hello world\"])                    # (n, dim) float32 numpy\n```\n\nValidated against the same golden reference as the Go tests\n(`python/test_rembed.py`\n\n); vectors cross the ABI bit-identically.\n\n```\ngo run ./cmd/rembed embed    -model models/all-MiniLM-L6-v2 \"some text\"\ngo run ./cmd/rembed validate -model models/all-MiniLM-L6-v2\ngo run ./cmd/rembed bench    -model models/all-MiniLM-L6-v2\n```\n\nApache-2.0", "url": "https://wpnews.pro/news/rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo", "canonical_source": "https://github.com/rostamlabs/rembed", "published_at": "2026-08-27 07:14:32+00:00", "updated_at": "2026-08-27 07:49:07.165950+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Rostam Labs", "Rembed", "Hugging Face Hub", "ONNX Runtime", "ModernBERT", "EmbeddingGemma", "Qwen3-Embedding", "nomic-embed"], "alternates": {"html": "https://wpnews.pro/news/rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo", "markdown": "https://wpnews.pro/news/rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo.md", "text": "https://wpnews.pro/news/rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo.txt", "jsonld": "https://wpnews.pro/news/rembed-pure-go-text-embeddings-no-onnx-runtime-no-cgo.jsonld"}}