Do you still need embeddings for semantic search? - my little experiment

An independent experiment testing whether semantic search can work without embeddings found that embeddings still won or tied in every fair comparison, with keyword search (BM25) coming within about 1 percentage point on a 0-to-100 nDCG@10 score when questions and answers shared words or close synonyms but dropping 8 to 9 points when the answer used completely different words. The author tested BM25 against the Qwen3-Embedding-0.6B embedding model on three public datasets — SciFact, NFCorpus and FiQA — and found that LLM query rewriting without documents barely helped, while grounding the rewrite in the top five keyword results improved it. The result matters for RAG builders weighing whether to skip vector databases: the author reports embeddings remained hard to beat across the comparisons.

Last week on X aka Twitter , I came across a product that claimed to perform semantic search without using vector databases. Semantic search without vectors? It was enough to spike my curiosity They did not claim how they did it, so I wanted to try building my own version. Goal: Build a Semantic Search system without using Embeddings. In this post, I share about my trials and errors in this attempt This was more of an experiment or series of experiments . So, before each test - I wrote down my expectations of outcomes, so I couldn't adjust the story afterwards. TL;DR: Embeddings are still hard to beat; embeddings won or tied in every fair comparison. On a 0 to 100 score, keyword search came within about 1% point when the question and the answer shared words or close synonyms. It dropped 8 to 9% when the answer used completely different words. A quick note before we start: I'm assuming you've built a RAG system, or at least poked at one, and have a rough idea of what embeddings and BM25 are. If BM25 is new to you, don't worry, there's a short explainer when it first shows up. The idea In a typical RAG setup, the heavy lifting happens when documents are loaded: every document is chunked and converted into an embeddings and stored as a vector, so search can match on meaning. My approach saves that work for query time. Keyword search only matches exact words, so an LLM expands the question with other words a matching document might use, such as synonyms, related phrases and technical terms. The expansion is meant to do what embeddings usually do, match on meaning/semantics. Here’s my idea described in few steps: 1. Search the keyword index with the question. 2. Let an LLM read the top results and rewrite the question with better search words. 3. Search again with the rewritten question. 4. Later in the post, a reranker reorders the best candidates. The first three steps are about finding the right documents. The reranker is about putting them in the right order. Most of what follows comes down to which of those two jobs is failing. The setup The keyword side runs on BM25, the same algorithm OpenSearch and Elasticsearch use by default. ↓ Internals: BM25 BM25 scores a document by the words it shares with the question. Rare words count much more than common ones, and repeating a word helps less each time. It can't tell that "tiredness" and "fatigue" mean the same thing, and that blind spot drives most of this post. I tested on three public datasets that behave differently: - SciFact: scientific claims matched to paper abstracts - NFCorpus: short medical questions matched to technical papers - FiQA: everyday finance questions matched to forum answers I compared it against an embedding model, Qwen3-Embedding-0.6B, and in the final comparisons both sides got the same help. I made every decision on a set of development questions, then scored a separate set of test questions once, at the end. The two main comparison tables use the test questions; the deeper analyses use the development ones. ↓ Internals: how I score results nDCG@10 rates the top 10 results, and the top spots count most. It normally runs from 0 to 1; I show it from 0 to 100, so one point is 0.01. Higher is better. I compare setups on the same questions and report a 95% interval. If it crosses zero, I can't tell the two apart. Rewriting the query I started by testing the rewrite on its own, before adding any reranker. My first version was simple: ask the LLM for synonyms and related terms, without showing it any documents. It barely helped. The LLM added so many words that the original question got lost in the query. When I asked for fewer words, it picked ones that drifted off topic. So I tried grounding it in the corpus itself. Now the LLM first reads the top five keyword search results and borrows words from them, so its suggestions come from documents that actually exist. I also repeat the original question three times in the final query, so it doesn't get drowned out again. Query2doc, a 2023 paper, does the same thing with five repeats . The rewrite helped on only one of my three datasets. On NFCorpus, questions use everyday words and papers use technical ones, and the LLM bridged that. One question about glycotoxins found nothing until the rewrite added "advanced glycation endproducts". FiQA is where it broke. One question asks, "What are the labels associated with a share called?" The right answer talks about "ordinary" and "preference" shares and never says "label". No synonym leads there; you'd have to know the answer. Embeddings ranked it first. The rewrite read five wrong results, copied their vocabulary stock splits, options , and searched further in the wrong direction. A rewrite can supply synonyms, but it can't guess what an answer will talk about. Reranking keyword results Next, I wanted to know why keyword search loses. Does it fail to find the right documents, or find them and rank them too low? To check, I took keyword search's top 100 and scored it as if it had been sorted using the answer key, with every relevant document moved to the top, on the development questions. Sorted perfectly, keyword search's own candidates would beat embeddings, so ordering was a big part of the problem. Embeddings' candidates would score even higher, which matters later. ↓ Internals: rerankers BM25 and embeddings score the question and each document separately, which is fast. A reranker reads the question and a document together and judges the pair. It's slower and better, so it only reorders the top results. I used one over the top 50. When I added the reranker, keyword search jumped, and on SciFact it matched embeddings. For a while I thought that settled it. Giving embeddings the same reranker Then I noticed I had given keyword search a reranker and embeddings nothing, which isn't a fair comparison. So I gave embeddings the same reranker. Embeddings with the reranker stayed ahead everywhere: about a point on SciFact and NFCorpus, eight to nine on FiQA. On SciFact and NFCorpus the gap is small enough that a tie isn't ruled out, but I couldn't show one. The test questions held one surprise. The reranker only helped embeddings on SciFact. On NFCorpus and FiQA, plain embeddings did as well or slightly better without it, and still beat both reranked keyword setups. On the development questions the reranker had helped embeddings on FiQA by about four points, so I wouldn't read much into either direction. I also tried a much bigger LLM as the reranker. It didn't change the picture, and it cost about four times as much per search. One thing surprised me: once the reranker was in, the rewrite barely mattered, within about half a point on SciFact and NFCorpus, and slightly worse on FiQA. The reranker already does the reordering the rewrite was helping with. I wouldn't put an LLM rewrite in front of a reranked pipeline by default anymore. The one place I'd still try it is short queries full of rare jargon. Made-up names Codenames should favour keyword search. By a codename I mean a name a company invents, like a billing system called Atlas. These names fill internal docs but rarely appear on the web. This is the case I wrote about in 3: an embedding model maps text by how words were used on the web, so names that never appeared there land somewhere arbitrary. Keyword search doesn't care what a word means, only how rare it is. To test it, I swapped 54 real names in FiQA companies, currencies, account types for made-up words, so "Roth IRA" became "Kevodran". They're public names standing in for internal ones. First, users type the codename: Embeddings lost about 10 points, while keyword search barely moved. The gap between the two reranked setups shrank from 7.5 points to 1.3, too close to call on 94 questions. But it didn't flip. When users describe the thing instead of naming it, keyword search has little left to match, and embeddings win clearly. I had written down that keyword search would win this one outright. It narrowed the gap a lot and still finished behind. Missing candidates So why couldn't keyword search catch up, even with the same reranker? The reranker can only reorder what it's given: same 50 documents in, same order out. So the gap has to come from which documents each side hands it. Embeddings hand it better ones. On FiQA, even when both lists had a relevant document, embeddings' list held more of them. The ones keyword search missed were mostly still there, ranked somewhere between 100th and 500th, too deep for a reranker reading the top 50. On NFCorpus, most weren't in its top 1,000 at all. So I let the reranker read deeper, on 150 FiQA development questions: Reading the top 200 closes about half the gap, for about four times the reranking cost. After that it flattens, and even at 500, embeddings with a reranker stay 3.5 points ahead. Deep keyword reranking catches up with plain embeddings, but not with embeddings plus a reranker. One example: a FiQA question asks, "Am I able to conduct a private sale of public shares at a price that I determine?" Its answer talks about shares "issued in your name" and "street name", and keyword search ranks it 274th. A reranker reading the top 50 never sees it; one reading the top 500 moves it to first. Embeddings had it at 5th. Cost The two approaches pay at different times. Embeddings pay up front: every document goes through the model, and again whenever you switch models. Keyword search costs almost nothing to index, but pays on every search for the reranker, and more if it has to read deeper. At these prices, embedding all of FiQA costs about as much as 200 to 1,200 reranked keyword searches. A collection that changes constantly and is searched rarely favours keyword search; one that's searched all day pays off its embeddings quickly. Choosing between them From what I saw, keyword search with a reranker is a reasonable choice when questions and documents use the same words, or when users type exact names the embedding model has never seen. I'd pick embeddings when the answer uses different words from the question, when users describe things instead of naming them, or when you can't afford a reranker at all. If I were starting a search system today, I'd begin with BM25 and a reranker, collect a small set of real questions, and add embeddings once those questions show people asking in words the documents don't use. I'd also measure before assuming hybrid search helps; my one hybrid run didn't. In my single hybrid search test, combining BM25 and embeddings didn’t yield an improvement. However, because production systems frequently rely on Reciprocal Rank Fusion RRF to blend keyword precision with semantic breadth, tuning candidate pool depth and rank fusion is an area worth testing further. Mistakes along the way Writing down what I expected before every test kept me honest. A lot of my guesses were wrong, and every one of them is in the repo's journal. I built this with a coding agent doing most of the runs. It was good at the tedious parts, like freezing files before a test and checking a claim before writing it down. It also named the wrong cause for the reranker result, and I only caught it because the mechanism didn't allow that reading. I also used AI to help edit and tighten my writing for this post; the ideas, the decisions and the mistakes are mine. Next post: can that candidate gap be closed without embeddings? I'll try searching with a guessed answer instead of the question, and running several short searches instead of one long one. INTERNALS.md is a technical series on how production AI systems actually work. No tutorials. No framework evangelism. Just the layer beneath. References - CodePackage associated with this experiment: Thedu - https://github.com/laxmena/thedu https://github.com/laxmena/thedu Pronounced Thay-doo - BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models https://arxiv.org/abs/2104.08663 , Thakur et al., 2021 - When do Generative Query and Document Expansions Fail? https://arxiv.org/abs/2309.08541 , Weller et al., 2023 - Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents https://arxiv.org/abs/2304.09542 , Sun et al., 2023 - Query2doc: Query Expansion with Large Language Models https://arxiv.org/abs/2303.07678 , Wang et al., 2023 - Pyserini BEIR baselines https://castorini.github.io/pyserini/2cr/beir.html - Your embedding model doesn't understand your data https://internals.laxmena.com/p/your-embedding-model-doesnt-understand , INTERNALS.md 3 - https://github.com/allenai/scifact - https://huggingface.co/datasets/BeIR/nfcorpus - https://huggingface.co/datasets/BeIR/fiqa