# Everyone’s Obsessed With Embeddings. BM25 Is Still Doing the Heavy Lifting. > Source: > Published: 2026-09-15 07:59:49+00:00 In the previous [part](https://medium.com/ai-in-plain-english/i-built-a-rag-system-to-understand-what-rag-actually-does-3e8b8a8d907b), I talked about why I started building a small RAG system instead of trying to memorize RAG architecture diagrams. I started with a very simple idea: *Before asking an LLM anything, I need to find the relevant part of my documents.* That led me to retrieval. And before I touched embeddings, I wanted to understand something much simpler: **Can I just search for the words in the user’s question?** The answer is yes. But it turns out that “search for matching words” is not quite enough. That’s where BM25 came in. And honestly, BM25 was one of those things I had seen many times before actually understanding it. **But if someone had asked me:** *“Why BM25? What is it actually calculating?”* I would have struggled to explain it. So I decided to break it down from the beginning. Imagine I have these three chunks: ``` Chunk 1:To reset your password, open Account Settingsand select Reset Password. Chunk 2:Passwords are encrypted using our security system.Users must authenticate before accessing their account. Chunk 3:To configure email notifications, open Settingsand select Notifications. ``` And the user asks: *How do I reset my password?* The simplest thing I can do is look for: ``` resetpassword ``` So intuitively: ``` php Chunk 1 ->very relevantChunk 2 ->somewhat relevantChunk 3 ->probably irrelevant ``` That’s already the beginning of a search ranking system. But there are some problems. Suppose I have 10,000 chunks. The word: ``` password ``` might appear in 7,000 of them. If a chunk contains password, that's not particularly surprising. Now imagine the word: ``` reset ``` appears in only 100 chunks. Finding reset tells me much more. This was one of the first BM25 ideas that really clicked for me: ***A word becomes more useful as evidence when it is relatively rare across the collection.*** Think about a search for: password is useful. But reset might be a stronger signal. This is where **IDF** comes in. Let’s start with something simpler. Suppose this is our chunk: ``` To reset your password, go to Settings.You can reset your password from there. ``` The word reset appears twice. So: ``` TF(reset) = 2 ``` TF means: ***Term Frequency*** Basically: *How many times does this term appear in this particular chunk?* If password appears twice: ``` TF(password) = 2 ``` If settings appears once: ``` TF(settings) = 1 ``` If kubernetes doesn't appear: ``` TF(kubernetes) = 0 ``` This sounds straightforward. But there’s an important distinction coming next. This confused me initially. Imagine we have 100 chunks. The word password appears three times in Chunk 1. It appears once in Chunk 2. And it appears once in Chunk 3. Then: ``` TF(password, Chunk 1) = 3 ``` But the number of chunks containing password is: ``` DF(password) = 3 ``` DF means: ***Document Frequency*** ``` php TF -> How many times does the word appear in this chunk? php DF -> How many chunks contain the word? ``` That’s an important distinction. You can think of it like this: ``` password ┌─────────────────────┐ │ │ ↓ ↓ TF DF │ │ ↓ ↓ Inside one chunk Across all chunks ``` Once I separated those two concepts, BM25 became much easier to follow. DF tells us how common a word is. IDF tells us how much that commonness should affect its importance. Suppose we have 1,000 chunks. password appears in 800 chunks. Very common. reset appears in 50 chunks. Much less common. So we’d like something roughly like: ``` php password -> lower importancereset -> higher importance ``` That’s the intuition behind **Inverse Document Frequency**. *The less frequently a term appears across the collection, the more useful it can be for distinguishing relevant documents.* This is where BM25 starts becoming more than simple keyword matching. It isn’t just saying: *“The word exists.”* It’s asking: *“How meaningful is the presence of this word?”* Imagine searching for: *database connection timeout* If a document contains: ``` database ``` that’s not enough. If your entire documentation is about databases, database may appear everywhere. But if a document contains: ``` connection timeout ``` that’s much more specific. Those terms give us stronger evidence. So BM25 naturally gives more importance to terms that are useful for distinguishing one chunk from another. Suppose I have two chunks. ``` reset password reset password reset password reset passwordreset password reset password reset passwordreset password reset password ``` If I only use term frequency, Chunk B wins easily. It contains the words many more times. But does that mean it is actually more relevant? Not necessarily. Maybe the document is simply much longer. This is where BM25 does something clever. It **doesn’t let term frequency increase the score forever.** The benefit of seeing a word again starts to diminish. This is called **TF saturation**. Imagine a word appears once: ``` TF = 1 ``` That’s useful. It appears again: ``` TF = 2 ``` That should increase confidence. Again: ``` TF = 3 ``` Still useful. But going from: ``` TF = 100 ``` to: ``` TF = 101 ``` shouldn’t make a huge difference. Otherwise, a document that repeats a keyword hundreds of times would always win. BM25 prevents that. *That’s one reason BM25 is more useful than simply counting keyword occurrences.* Here’s another situation. Imagine two chunks both contain the word password five times. ``` php Chunk A -> 100 charactersChunk B -> 5,000 characters ``` Five occurrences mean something different in those two chunks. In the short chunk, the word is a much larger part of the content. In the huge chunk, it might just be one of many words. BM25 accounts for this using **length normalization**. The intuition is: *Don’t automatically reward long chunks just because they naturally have more opportunities to contain a word.* This is controlled by a parameter called b. In my implementation: ``` js const b = 0.75; ``` Again, I wouldn’t start by memorizing 0.75. The important thing is understanding what b controls: ``` b↓How strongly should document length affect the score? ``` The implementation I used in my MVP looks like this: ``` js const idf = Math.log( (chunks.length - df + 0.5) / (df + 0.5) + 1); js const tf = termFrequency(term, text); js const lengthNorm = 1 - b + (b * docLength) / avgLength; score += idf * ( (tf * (k1 + 1)) / (tf + k1 * lengthNorm) ); ``` **Think of k1 and b like configuration knobs** Imagine your retrieval system has: ``` php k1│├── Low -> repeated words saturate quickly│└── High ->repeated words continue contributingb│├── 0 -> document length doesn't matter│└── 1 -> document length matters strongly ``` When I first saw this, it looked like a wall of math. Now I can break it into three questions: ``` BM25 │ ┌─────────────┼─────────────┐ ↓ ↓ ↓ IDF TF Length │ │ │ How rare is How often How long is the word? does it this chunk? appear? ``` That’s basically the heart of it. My function starts with: ``` export function bm25Score( query: string, text: string, chunks: Chunk[]) {} ``` There are three things here: ``` query ↓What did the user ask? text ↓Which chunk am I scoring? chunks ↓What does the entire collection look like? ``` That last parameter is important. BM25 doesn’t just need the current chunk. It needs the entire collection to calculate things like: *How common is this word across all chunks?* Before scoring anything, I tokenize the query: ``` js const terms = tokenize(query); ``` My tokenizer is intentionally simple: ``` function tokenize(value: string) { return [ ...new Set( value .toLowerCase() .match(/[a-z0-9]{2,}/g) || [] ) ];} ``` For: ``` "How do I reset my password?" ``` I get something roughly like: ``` ["how", "do", "reset", "my", "password"] ``` The Set removes duplicates. ``` "reset reset password" ``` becomes: ``` ["reset", "password"] ``` This isn’t a production-grade tokenizer. And that’s okay. This is an MVP. The goal was to understand retrieval, not build the next Google search engine. My term frequency function looks for the actual word inside the current chunk. Conceptually: ``` Query term:reset Current chunk:"To reset your password..." ``` The function answers: ``` TF(reset) = 1 ``` If the word appears three times: ``` TF(reset) = 3 ``` And if it doesn’t appear: ``` TF(reset) = 0 ``` That last case matters because a query can contain several terms, but a particular chunk might only match some of them. Then I calculate document frequency: ``` function termDocumentFrequency( term: string, chunks: Chunk[]) { return chunks.filter( (chunk) => containsWord(chunk.text, term) ).length;} ``` This is basically asking every chunk: *“Do you contain this word?”* If 30 chunks say yes: ``` DF = 30 ``` Notice that I’m not counting how many times the word appears. I’m counting **how many chunks contain it**. That’s why it’s called document frequency. Suppose the query is: ``` reset password ``` BM25 doesn’t treat the entire query as one giant object. It processes: ``` reset ``` and: ``` password ``` separately. For each term: ``` term ↓DF ↓IDF ↓TF in current chunk ↓length normalization ↓term contribution ``` Then the contributions are added together. So conceptually: ``` BM25(query, chunk) = score(reset) + score(password) ``` If the chunk contains both important terms, it gets a stronger score. Let’s imagine our query is: ``` reset password ``` And we have these chunks: ``` Chunk A:To reset your password from Account Settings... Chunk B:Our system stores password information securely... Chunk C:Configure email notifications in Settings... ``` Now imagine: ``` reset password Chunk A ✓ ✓Chunk B ✗ ✓Chunk C ✗ ✗ ``` Chunk A gets contributions from both terms. Chunk B gets a contribution from only password. Chunk C gets nothing. So we’d expect: ``` php Chunk A -> highestChunk B -> lowerChunk C -> zero ``` That’s already a useful ranking. It doesn’t require the query and document to be converted into embeddings. There is no model involved in this part. It’s just: ``` text ↓tokenize ↓count ↓calculate statistics ↓rank ``` That makes BM25 relatively easy to reason about. And that’s exactly why I wanted it in my MVP. I could actually see what was happening. If a result ranked highly, I could ask: *Which words caused this?* With embeddings, that question becomes much harder to answer directly. Let’s go back to the example: Query: *How do I change my login credentials?* Document: *To reset your password, open Account Settings.* A human can see the connection. But BM25 mostly sees: ``` change ≠ resetlogin ≠ accountcredentials ≠ password ``` The meaning is related. The words aren’t. That’s the limitation of keyword-based retrieval. BM25 is very good at answering: ***“Do these important words match?”*** But it isn’t designed to answer: ***“Do these two pieces of text mean roughly the same thing?”*** And that’s exactly the problem I ran into next. I initially thought: *“If embeddings can understand meaning, why don’t I just use embeddings for everything?”* But after working with BM25, I realized something important. BM25 and semantic search solve slightly different problems. BM25 is great when the exact word matters. For example: ``` error code: HTTP 429 ``` If the user searches for: ``` HTTP 429 ``` I don’t necessarily want a semantic interpretation. I want the exact thing. On the other hand, if the user asks: *“How do I change my login credentials?”* I may want a result talking about: *“resetting your password”* even though the words aren’t identical. So instead of thinking: *BM25 vs embeddings* I started thinking: ***BM25 + embeddings*** And that became the next step in my MVP. The biggest thing I took away wasn’t the formula. It was this: **Search isn’t simply about finding words.** It’s about deciding how much evidence a matching word provides. BM25 considers: ``` TF How often does the term appear here?DF How many chunks contain it?IDF How useful is that term for distinguishing chunks?Length normalization Is this chunk unusually long?TF saturation Should repeating the same word 100 times really make it 100x more relevant? ``` Once I understood those questions, the BM25 formula stopped looking completely random. It became a compact way of expressing those decisions. If I had to explain BM25 to someone without showing the formula, I’d say: ***BM25 is trying to answer: “How strong is the keyword-based evidence that this chunk is relevant to the query?”*** ``` BM25= ∑IDF × (TF(k1+1) / TF+k1(1−b+b×dl/avgdl) )in ``` That’s enough to make the formula worth learning. Now I had a decent keyword-based retriever. But I still had the problem of different wording. The user might ask: while the document says: *“To reset your password…”* A human sees the relationship immediately. BM25 doesn’t. So I needed a way to represent the **meaning** of text rather than just its words. That took me to embeddings. And that’s where things got a little weird. Because the first time I saw an embedding, all I saw was something like: ``` [0.012, -0.438, 0.721, 0.091, ...] ``` And my first question was pretty simple: ***“What am I actually looking at?”*** *That’s what I’ll try to answer in Part 3.* **And that’s a wrap! 🎯** If you’ve made it this far, I hope you found something useful in this first part. I’m building this series while going through the same learning process myself, so I’d genuinely love to hear how others are approaching RAG, AI agents, and retrieval systems. If this article helped you understand something a little better, give it a clap 👏 and share it with someone who’s trying to make sense of what’s happening under the hood. **More useful reads(worth your time):****1.** [The One Git Command Every AI Agent Architect Needs](https://ai.plainenglish.io/ai-agents-need-git-worktrees-most-developers-havent-realized-it-yet-4da51bf51140) 2. [How We Built a Robust Message Queue Using BullMQ](https://blog.stackademic.com/how-we-built-a-robust-message-queue-using-bullmq-part-1-2d5ad1016958) 3. [Docker Has Changed. Most Engineers Haven’t](https://levelup.gitconnected.com/docker-has-changed-most-engineers-havent-685edec8d26c) 4. [Beyond Microservices: Scaling and Decoupling with a Hybrid NestJS Architecture](https://blog.stackademic.com/beyond-microservices-streamlining-scaling-and-decoupling-with-a-hybrid-nestjs-architecture-f3adece72949) 5. [Building a Health Check REST API Inside a NestJS Standalone Application (Without Listening on Any Port)](https://medium.com/@karthiks05/how-we-built-a-health-check-rest-api-endpoint-inside-a-nestjs-standalone-app-that-doesnt-listen-243cf4cf5637) Until next time keep building, stay curious, and keep digging into how things actually work. [Everyone’s Obsessed With Embeddings. BM25 Is Still Doing the Heavy Lifting.](https://blog.stackademic.com/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting-1e706523ab18) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.