{"slug": "nemotron-embed-and-rerank-nvidia-s-local-rag-stack", "title": "Nemotron Embed and Rerank: NVIDIA's local RAG stack", "summary": "NVIDIA released Nemotron 3.5 Embed (1B and 8B) and Nemotron 3.5 Rerank (8B) on Hugging Face under the openmdw-1.1 license, ungated, with official NVFP4 checkpoints and community GGUF conversions. The bi-encoder embedding models accept images, text, or both, with the 8B producing 4,096-dimension vectors over a 32,768-token context and the 1B producing 2,048-dim vectors in a shared text embedding space; the reranker was benchmarked reranking Embed-1B's top-50 results. NVIDIA built the models on Mistral's Ministral-3-3B and 8B-Instruct bases, and the memory footprint — about 2 GB for Embed-1B at BF16 and roughly 16 GB for Rerank-8B at BF16 or near 8 GB at NVFP4 — lets a 24 GB machine run the full retrieval layer alongside an answering model.", "body_md": "NVIDIA just shipped the two pieces local RAG always had to steal from different vendors: an open embedding model and an open reranker, trained to work together, multimodal, and small enough that the retrieval layer stops eating the VRAM your answering model needs. **Nemotron 3.5 Embed** (1B and 8B) and **Nemotron 3.5 Rerank** (8B) are up on Hugging Face under openmdw-1.1 licenses, ungated, with official NVFP4 checkpoints and community GGUF conversions already pulling thousands of downloads.\n\n## What you are actually getting\n\nAn embedding model with one job: turning a document page into one vector that means “what this page says.” Both sizes are bi-encoders, and the difference from the usual embedding story is the input: a page can go in as an image, as text, or both. Scanned PDFs, charts, infographics and whiteboard photos go into the index without an OCR pass. The 8B model produces 4,096-dimension vectors over a 32,768-token context; the 1B produces 2,048-dim vectors and the two share the same text embedding space, which means you can start on 1B and re-rank your way to better answers without re-architecting.\n\nThe reranker is the part most local stacks skip, and skipping it is why local RAG often disappoints. Embeddings do fast approximate search; a cross-encoder reads the question and the candidate page TOGETHER and scores the pair. That second pass is where “found 50 pages that might help” becomes “here are the 3 pages that answer it.” The charts tell the story in NVIDIA’s own measured numbers.\n\n## Why NVIDIA shipping this matters more than the scores\n\nThree things in this release are about the ecosystem, not the leaderboard:\n\n**The whole pipeline is finally one vendor’s coherent set.** Until now a local RAG stack assembled parts from four shops: BAAI’s bge for embeddings, a closed Cohere rerank API, Jina for multilingual, and your own glue. NVIDIA now ships embed + rerank designed as a pair (the reranker’s benchmark was run reranking Embed-1B’s top-50, not some other embedder’s), plus Parse (their 7.6k-download document extraction model) upstream. Version-matching risk across the pipeline disappears.\n\n**NVIDIA built it on Mistral’s bases** - the model cards list mistralai/Ministral-3-3B and 8B-Instruct as the base models, architecture Mistral3Model. The RAG layer of the most pro-NVIDIA stack there is runs on a European open-weights foundation, and llama.cpp supports the arch natively, which is why GGUFs appeared within days.\n\n**The memory math closes the loop.** Embed-1B is ~2 GB at BF16 (4-bit under 1.5 GB); Rerank-8B is ~16 GB BF16 or near 8 GB at NVFP4. So a 24 GB box runs the full retrieval floor plus a real answering model; a 48 GB Mac runs everything with room to spare.\n\n## What it does not do\n\nNo text-only trickery here, but real limits: it is retrieval, not answering - the reranker returns scores, the chat model still writes the answer. NVFP4 needs an RTX-class card, so Mac users take the GGUF/MLX path. And openmdw-1.1 is an open-weights license with use terms, not Apache: fine for personal and most commercial use, but read it before embedding this in a product you sell.\n\n## The week in context\n\nThe same seven days added Voxtral 3 transcription family announcements from Mistral (no weights yet), closed realtime duplex voice from ElevenLabs, and IBM’s CEO telling The Register that open weights are “irresistibly good and cheap on-prem” (258 points on HN). The RAG-stack release is the concrete version of that claim: the parts that used to justify subscriptions - the search layer, the rerank layer, the PDF understanding - now ship as weights you can pull tonight. The answering layer has been open for months. What remains closed gets smaller every week, and this release takes another visible slice off it.\n\nSources: NVIDIA model cards for Embed-1B/8B-BF16, Embed-1B-NVFP4, Rerank-8B-BF16 (benchmarks, license, base models); W4D GGUF conversions and compatibility tables; The Register’s Oct 22 IBM interview for the on-prem quote; the Nemotron 3.5 Oct 20 release event.\n\nRelated on this site: the model catalog carries the rest of the Nemotron 3.5 family (Bolt E4B runs on 4 GB cards), and the Kimi K3.1 Pro piece covers why 1M-context models don’t remove the need for retrieval. Rerank quality is also why “one big model” approaches waste VRAM: the retrieval floor here costs less than one Bolt layer.===", "url": "https://wpnews.pro/news/nemotron-embed-and-rerank-nvidia-s-local-rag-stack", "canonical_source": "https://tokenstead.ai/guides/nemotron-rag-stack", "published_at": "2026-10-26 11:52:39+00:00", "updated_at": "2026-10-26 12:40:59.407236+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "machine-learning"], "entities": ["NVIDIA", "Nemotron 3.5 Embed", "Nemotron 3.5 Rerank", "Hugging Face", "Mistral", "Ministral-3-3B", "llama.cpp", "IBM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/nemotron-embed-and-rerank-nvidia-s-local-rag-stack", "markdown": "https://wpnews.pro/news/nemotron-embed-and-rerank-nvidia-s-local-rag-stack.md", "text": "https://wpnews.pro/news/nemotron-embed-and-rerank-nvidia-s-local-rag-stack.txt", "jsonld": "https://wpnews.pro/news/nemotron-embed-and-rerank-nvidia-s-local-rag-stack.jsonld"}}