{"slug": "everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting", "title": "Everyone’s Obsessed With Embeddings. BM25 Is Still Doing the Heavy Lifting.", "summary": "A Medium post in the \"AI in Plain English\" series argues that BM25 keyword ranking remains essential to retrieval-augmented generation (RAG) systems even as embeddings draw widespread attention. The author walks through BM25's core mechanics, explaining that term frequency (TF) counts how often a term appears in a single chunk while document frequency (DF) counts how many chunks contain it, and that inverse document frequency (IDF) raises a term's weight as it becomes rarer across the collection — a word like \"reset\" appearing in 50 of 1,000 chunks carries more signal than \"password\" appearing in 800. The piece is the second part of a series on building a small RAG system to understand what RAG actually does.", "body_md": "In the previous [part](https://medium.com/ai-in-plain-english/i-built-a-rag-system-to-understand-what-rag-actually-does-3e8b8a8d907b), I talked about why I started building a small RAG system instead of trying to memorize RAG architecture diagrams.\n\nI started with a very simple idea:\n\n*Before asking an LLM anything, I need to find the relevant part of my documents.*\n\nThat led me to retrieval.\n\nAnd before I touched embeddings, I wanted to understand something much simpler:\n\n**Can I just search for the words in the user’s question?**\n\nThe answer is yes.\n\nBut it turns out that “search for matching words” is not quite enough.\n\nThat’s where BM25 came in.\n\nAnd honestly, BM25 was one of those things I had seen many times before actually understanding it.\n\n**But if someone had asked me:**\n\n*“Why BM25? What is it actually calculating?”*\n\nI would have struggled to explain it.\n\nSo I decided to break it down from the beginning.\n\nImagine I have these three chunks:\n\n```\nChunk 1:To reset your password, open Account Settingsand select Reset Password.\nChunk 2:Passwords are encrypted using our security system.Users must authenticate before accessing their account.\nChunk 3:To configure email notifications, open Settingsand select Notifications.\n```\n\nAnd the user asks:\n\n*How do I reset my password?*\n\nThe simplest thing I can do is look for:\n\n```\nresetpassword\n```\n\nSo intuitively:\n\n``` php\nChunk 1 ->very relevantChunk 2 ->somewhat relevantChunk 3 ->probably irrelevant\n```\n\nThat’s already the beginning of a search ranking system.\n\nBut there are some problems.\n\nSuppose I have 10,000 chunks.\n\nThe word:\n\n```\npassword\n```\n\nmight appear in 7,000 of them.\n\nIf a chunk contains password, that's not particularly surprising.\n\nNow imagine the word:\n\n```\nreset\n```\n\nappears in only 100 chunks.\n\nFinding reset tells me much more.\n\nThis was one of the first BM25 ideas that really clicked for me:\n\n***A word becomes more useful as evidence when it is relatively rare across the collection.***\n\nThink about a search for:\n\npassword is useful.\n\nBut reset might be a stronger signal.\n\nThis is where **IDF** comes in.\n\nLet’s start with something simpler.\n\nSuppose this is our chunk:\n\n```\nTo reset your password, go to Settings.You can reset your password from there.\n```\n\nThe word reset appears twice.\n\nSo:\n\n```\nTF(reset) = 2\n```\n\nTF means:\n\n***Term Frequency***\n\nBasically:\n\n*How many times does this term appear in this particular chunk?*\n\nIf password appears twice:\n\n```\nTF(password) = 2\n```\n\nIf settings appears once:\n\n```\nTF(settings) = 1\n```\n\nIf kubernetes doesn't appear:\n\n```\nTF(kubernetes) = 0\n```\n\nThis sounds straightforward.\n\nBut there’s an important distinction coming next.\n\nThis confused me initially.\n\nImagine we have 100 chunks.\n\nThe word password appears three times in Chunk 1.\n\nIt appears once in Chunk 2.\n\nAnd it appears once in Chunk 3.\n\nThen:\n\n```\nTF(password, Chunk 1) = 3\n```\n\nBut the number of chunks containing password is:\n\n```\nDF(password) = 3\n```\n\nDF means:\n\n***Document Frequency***\n\n``` php\nTF -> How many times does the word appear in this chunk?\nphp\nDF -> How many chunks contain the word?\n```\n\nThat’s an important distinction.\n\nYou can think of it like this:\n\n```\npassword\n┌─────────────────────┐          │                     │          ↓                     ↓         TF                    DF          │                     │          ↓                     ↓   Inside one chunk       Across all chunks\n```\n\nOnce I separated those two concepts, BM25 became much easier to follow.\n\nDF tells us how common a word is.\n\nIDF tells us how much that commonness should affect its importance.\n\nSuppose we have 1,000 chunks.\n\npassword appears in 800 chunks.\n\nVery common.\n\nreset appears in 50 chunks.\n\nMuch less common.\n\nSo we’d like something roughly like:\n\n``` php\npassword -> lower importancereset    -> higher importance\n```\n\nThat’s the intuition behind **Inverse Document Frequency**.\n\n*The less frequently a term appears across the collection, the more useful it can be for distinguishing relevant documents.*\n\nThis is where BM25 starts becoming more than simple keyword matching.\n\nIt isn’t just saying:\n\n*“The word exists.”*\n\nIt’s asking:\n\n*“How meaningful is the presence of this word?”*\n\nImagine searching for:\n\n*database connection timeout*\n\nIf a document contains:\n\n```\ndatabase\n```\n\nthat’s not enough.\n\nIf your entire documentation is about databases, database may appear everywhere.\n\nBut if a document contains:\n\n```\nconnection timeout\n```\n\nthat’s much more specific.\n\nThose terms give us stronger evidence.\n\nSo BM25 naturally gives more importance to terms that are useful for distinguishing one chunk from another.\n\nSuppose I have two chunks.\n\n```\nreset password\nreset password reset password reset passwordreset password reset password reset passwordreset password reset password\n```\n\nIf I only use term frequency, Chunk B wins easily.\n\nIt contains the words many more times.\n\nBut does that mean it is actually more relevant?\n\nNot necessarily.\n\nMaybe the document is simply much longer.\n\nThis is where BM25 does something clever.\n\nIt **doesn’t let term frequency increase the score forever.**\n\nThe benefit of seeing a word again starts to diminish.\n\nThis is called **TF saturation**.\n\nImagine a word appears once:\n\n```\nTF = 1\n```\n\nThat’s useful.\n\nIt appears again:\n\n```\nTF = 2\n```\n\nThat should increase confidence.\n\nAgain:\n\n```\nTF = 3\n```\n\nStill useful.\n\nBut going from:\n\n```\nTF = 100\n```\n\nto:\n\n```\nTF = 101\n```\n\nshouldn’t make a huge difference.\n\nOtherwise, a document that repeats a keyword hundreds of times would always win.\n\nBM25 prevents that.\n\n*That’s one reason BM25 is more useful than simply counting keyword occurrences.*\n\nHere’s another situation.\n\nImagine two chunks both contain the word password five times.\n\n``` php\nChunk A -> 100 charactersChunk B -> 5,000 characters\n```\n\nFive occurrences mean something different in those two chunks.\n\nIn the short chunk, the word is a much larger part of the content.\n\nIn the huge chunk, it might just be one of many words.\n\nBM25 accounts for this using **length normalization**.\n\nThe intuition is:\n\n*Don’t automatically reward long chunks just because they naturally have more opportunities to contain a word.*\n\nThis is controlled by a parameter called b.\n\nIn my implementation:\n\n``` js\nconst b = 0.75;\n```\n\nAgain, I wouldn’t start by memorizing 0.75.\n\nThe important thing is understanding what b controls:\n\n```\nb↓How strongly should document length affect the score?\n```\n\nThe implementation I used in my MVP looks like this:\n\n``` js\nconst idf = Math.log(  (chunks.length - df + 0.5) /  (df + 0.5) + 1);\njs\nconst tf = termFrequency(term, text);\njs\nconst lengthNorm =  1 - b + (b * docLength) / avgLength;\nscore +=  idf *  (    (tf * (k1 + 1)) /    (tf + k1 * lengthNorm)  );\n```\n\n**Think of k1 and b like configuration knobs**\n\nImagine your retrieval system has:\n\n``` php\nk1│├── Low -> repeated words saturate quickly│└── High ->repeated words continue contributingb│├── 0 -> document length doesn't matter│└── 1 -> document length matters strongly\n```\n\nWhen I first saw this, it looked like a wall of math.\n\nNow I can break it into three questions:\n\n```\nBM25                      │        ┌─────────────┼─────────────┐        ↓             ↓             ↓       IDF            TF        Length        │             │             │   How rare is     How often    How long is    the word?      does it      this chunk?                    appear?\n```\n\nThat’s basically the heart of it.\n\nMy function starts with:\n\n```\nexport function bm25Score(  query: string,  text: string,  chunks: Chunk[]) {}\n```\n\nThere are three things here:\n\n```\nquery ↓What did the user ask?\ntext ↓Which chunk am I scoring?\nchunks ↓What does the entire collection look like?\n```\n\nThat last parameter is important.\n\nBM25 doesn’t just need the current chunk.\n\nIt needs the entire collection to calculate things like:\n\n*How common is this word across all chunks?*\n\nBefore scoring anything, I tokenize the query:\n\n``` js\nconst terms = tokenize(query);\n```\n\nMy tokenizer is intentionally simple:\n\n```\nfunction tokenize(value: string) {  return [    ...new Set(      value        .toLowerCase()        .match(/[a-z0-9]{2,}/g) || []    )  ];}\n```\n\nFor:\n\n```\n\"How do I reset my password?\"\n```\n\nI get something roughly like:\n\n```\n[\"how\", \"do\", \"reset\", \"my\", \"password\"]\n```\n\nThe Set removes duplicates.\n\n```\n\"reset reset password\"\n```\n\nbecomes:\n\n```\n[\"reset\", \"password\"]\n```\n\nThis isn’t a production-grade tokenizer.\n\nAnd that’s okay.\n\nThis is an MVP.\n\nThe goal was to understand retrieval, not build the next Google search engine.\n\nMy term frequency function looks for the actual word inside the current chunk.\n\nConceptually:\n\n```\nQuery term:reset\nCurrent chunk:\"To reset your password...\"\n```\n\nThe function answers:\n\n```\nTF(reset) = 1\n```\n\nIf the word appears three times:\n\n```\nTF(reset) = 3\n```\n\nAnd if it doesn’t appear:\n\n```\nTF(reset) = 0\n```\n\nThat last case matters because a query can contain several terms, but a particular chunk might only match some of them.\n\nThen I calculate document frequency:\n\n```\nfunction termDocumentFrequency(  term: string,  chunks: Chunk[]) {  return chunks.filter(    (chunk) => containsWord(chunk.text, term)  ).length;}\n```\n\nThis is basically asking every chunk:\n\n*“Do you contain this word?”*\n\nIf 30 chunks say yes:\n\n```\nDF = 30\n```\n\nNotice that I’m not counting how many times the word appears.\n\nI’m counting **how many chunks contain it**.\n\nThat’s why it’s called document frequency.\n\nSuppose the query is:\n\n```\nreset password\n```\n\nBM25 doesn’t treat the entire query as one giant object.\n\nIt processes:\n\n```\nreset\n```\n\nand:\n\n```\npassword\n```\n\nseparately.\n\nFor each term:\n\n```\nterm ↓DF ↓IDF ↓TF in current chunk ↓length normalization ↓term contribution\n```\n\nThen the contributions are added together.\n\nSo conceptually:\n\n```\nBM25(query, chunk) = score(reset) + score(password)\n```\n\nIf the chunk contains both important terms, it gets a stronger score.\n\nLet’s imagine our query is:\n\n```\nreset password\n```\n\nAnd we have these chunks:\n\n```\nChunk A:To reset your password from Account Settings...\nChunk B:Our system stores password information securely...\nChunk C:Configure email notifications in Settings...\n```\n\nNow imagine:\n\n```\nreset    password\nChunk A        ✓          ✓Chunk B        ✗          ✓Chunk C        ✗          ✗\n```\n\nChunk A gets contributions from both terms.\n\nChunk B gets a contribution from only password.\n\nChunk C gets nothing.\n\nSo we’d expect:\n\n``` php\nChunk A -> highestChunk B -> lowerChunk C -> zero\n```\n\nThat’s already a useful ranking.\n\nIt doesn’t require the query and document to be converted into embeddings.\n\nThere is no model involved in this part.\n\nIt’s just:\n\n```\ntext ↓tokenize ↓count ↓calculate statistics ↓rank\n```\n\nThat makes BM25 relatively easy to reason about.\n\nAnd that’s exactly why I wanted it in my MVP.\n\nI could actually see what was happening.\n\nIf a result ranked highly, I could ask:\n\n*Which words caused this?*\n\nWith embeddings, that question becomes much harder to answer directly.\n\nLet’s go back to the example:\n\nQuery:\n\n*How do I change my login credentials?*\n\nDocument:\n\n*To reset your password, open Account Settings.*\n\nA human can see the connection.\n\nBut BM25 mostly sees:\n\n```\nchange ≠ resetlogin ≠ accountcredentials ≠ password\n```\n\nThe meaning is related.\n\nThe words aren’t.\n\nThat’s the limitation of keyword-based retrieval.\n\nBM25 is very good at answering:\n\n***“Do these important words match?”***\n\nBut it isn’t designed to answer:\n\n***“Do these two pieces of text mean roughly the same thing?”***\n\nAnd that’s exactly the problem I ran into next.\n\nI initially thought:\n\n*“If embeddings can understand meaning, why don’t I just use embeddings for everything?”*\n\nBut after working with BM25, I realized something important.\n\nBM25 and semantic search solve slightly different problems.\n\nBM25 is great when the exact word matters.\n\nFor example:\n\n```\nerror code: HTTP 429\n```\n\nIf the user searches for:\n\n```\nHTTP 429\n```\n\nI don’t necessarily want a semantic interpretation.\n\nI want the exact thing.\n\nOn the other hand, if the user asks:\n\n*“How do I change my login credentials?”*\n\nI may want a result talking about:\n\n*“resetting your password”*\n\neven though the words aren’t identical.\n\nSo instead of thinking:\n\n*BM25 vs embeddings*\n\nI started thinking:\n\n***BM25 + embeddings***\n\nAnd that became the next step in my MVP.\n\nThe biggest thing I took away wasn’t the formula.\n\nIt was this:\n\n**Search isn’t simply about finding words.**\n\nIt’s about deciding how much evidence a matching word provides.\n\nBM25 considers:\n\n```\nTF How often does the term appear here?DF How many chunks contain it?IDF How useful is that term for distinguishing chunks?Length normalization Is this chunk unusually long?TF saturation Should repeating the same word 100 times really make it 100x more relevant?\n```\n\nOnce I understood those questions, the BM25 formula stopped looking completely random.\n\nIt became a compact way of expressing those decisions.\n\nIf I had to explain BM25 to someone without showing the formula, I’d say:\n\n***BM25 is trying to answer: “How strong is the keyword-based evidence that this chunk is relevant to the query?”***\n\n```\nBM25= ∑IDF × (TF(k1+1) / TF+k1(1−b+b×dl/avgdl) )in\n```\n\nThat’s enough to make the formula worth learning.\n\nNow I had a decent keyword-based retriever.\n\nBut I still had the problem of different wording.\n\nThe user might ask:\n\nwhile the document says:\n\n*“To reset your password…”*\n\nA human sees the relationship immediately.\n\nBM25 doesn’t.\n\nSo I needed a way to represent the **meaning** of text rather than just its words.\n\nThat took me to embeddings.\n\nAnd that’s where things got a little weird.\n\nBecause the first time I saw an embedding, all I saw was something like:\n\n```\n[0.012, -0.438, 0.721, 0.091, ...]\n```\n\nAnd my first question was pretty simple:\n\n***“What am I actually looking at?”***\n\n*That’s what I’ll try to answer in Part 3.*\n\n**And that’s a wrap! 🎯**\n\nIf you’ve made it this far, I hope you found something useful in this first part.\n\nI’m building this series while going through the same learning process myself, so I’d genuinely love to hear how others are approaching RAG, AI agents, and retrieval systems.\n\nIf this article helped you understand something a little better, give it a clap 👏 and share it with someone who’s trying to make sense of what’s happening under the hood.\n\n**More useful reads(worth your time):****1.** [The One Git Command Every AI Agent Architect Needs](https://ai.plainenglish.io/ai-agents-need-git-worktrees-most-developers-havent-realized-it-yet-4da51bf51140)\n\n2. [How We Built a Robust Message Queue Using BullMQ](https://blog.stackademic.com/how-we-built-a-robust-message-queue-using-bullmq-part-1-2d5ad1016958)\n\n3. [Docker Has Changed. Most Engineers Haven’t](https://levelup.gitconnected.com/docker-has-changed-most-engineers-havent-685edec8d26c)\n\n4. [Beyond Microservices: Scaling and Decoupling with a Hybrid NestJS Architecture](https://blog.stackademic.com/beyond-microservices-streamlining-scaling-and-decoupling-with-a-hybrid-nestjs-architecture-f3adece72949)\n\n5. [Building a Health Check REST API Inside a NestJS Standalone Application (Without Listening on Any Port)](https://medium.com/@karthiks05/how-we-built-a-health-check-rest-api-endpoint-inside-a-nestjs-standalone-app-that-doesnt-listen-243cf4cf5637)\n\nUntil next time keep building, stay curious, and keep digging into how things actually work.\n\n[Everyone’s Obsessed With Embeddings. BM25 Is Still Doing the Heavy Lifting.](https://blog.stackademic.com/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting-1e706523ab18) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting", "canonical_source": "https://blog.stackademic.com/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting-1e706523ab18?source=rss----d1baaa8417a4---4", "published_at": "2026-09-15 07:59:49+00:00", "updated_at": "2026-09-15 08:44:17.891153+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "natural-language-processing", "ai-research"], "entities": ["BM25", "RAG", "Medium", "AI in Plain English", "IDF", "TF"], "alternates": {"html": "https://wpnews.pro/news/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting", "markdown": "https://wpnews.pro/news/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting.md", "text": "https://wpnews.pro/news/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting.txt", "jsonld": "https://wpnews.pro/news/everyones-obsessed-with-embeddings-bm25-is-still-doing-the-heavy-lifting.jsonld"}}