{"slug": "building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai", "title": "Building a Hybrid RAG System with FAISS, BM25, and Agentic AI", "summary": "A developer built a hybrid RAG system combining FAISS vector search and BM25 keyword search to retrieve relevant information from a knowledge base and generate grounded answers. The system uses weighted scoring to rank results and exposes the retrieval as a tool for an agent, which uses Qwen2.5-72B-Instruct for response generation.", "body_md": "As part of my AI Engineering journey, I recently worked on a project that helped me understand how Retrieval-Augmented Generation (RAG) works in practice.\n\nI built a Hybrid RAG system that combines FAISS vector search and BM25 keyword search to retrieve relevant information from a knowledge base and use it to generate grounded answers.\n\nIn this post, I’ll briefly share what I built, how the system works, and some of the things I learned along the way.\n\n**Why RAG?**\n\nLarge Language Models are great at generating natural-language responses, but they may not have access to information contained in a specific document or knowledge base.\n\nRAG addresses this by first retrieving relevant information from an external knowledge base and then providing that information to the LLM as context.\n\nThe basic workflow is:\n\nUser Query\n\n↓\n\nRetrieve Relevant Information\n\n↓\n\nProvide Context to LLM\n\n↓\n\nGenerate Answer\n\nFor my project, I wanted to take this a step further by combining semantic search and keyword search.\n\n**🔍 Hybrid Retrieval**\n\nThe system uses two retrieval methods:\n\n**Vector Search with FAISS**\n\nDocument content is divided into smaller chunks and converted into vector embeddings.\n\nThese embeddings are stored in a FAISS index, which is used to find documents that are semantically similar to the user’s query.\n\nThis is useful even when the query and the document use different wording.\n\n**Keyword Search with BM25**\n\nThe second retrieval method is BM25.\n\nBM25 focuses on the occurrence and importance of terms in the query and documents. This makes it useful for exact terminology, technical terms, names, and identifiers.\n\nInstead of depending on only one retrieval method, both approaches are combined.\n\n```\n             User Query\n                 │\n      ┌──────────┴──────────┐\n      ↓                     ↓\n  FAISS Search           BM25 Search\nSemantic Search        Keyword Search\n      │                     │\n      └──────────┬──────────┘\n                 ↓\n          Hybrid Ranking\n                 ↓\n         Relevant Context\n                 ↓\n                LLM\n                 ↓\n            Final Answer\n```\n\nThe FAISS and BM25 scores are normalized and combined using weighted scoring. The results are then ranked, and the highest-ranked chunks are used as the final context.\n\n**Document Chunking**\n\nBefore retrieval, the documents are divided into smaller chunks.\n\nChunking is an important part of a RAG pipeline because the size of a chunk can affect retrieval quality.\n\nVery small chunks may lose important context, while very large chunks may contain unnecessary information and reduce retrieval precision.\n\nSo the goal is to find a balance between context and retrieval accuracy.\n\nEach chunk is also stored with metadata such as its document ID and title, making it easier to identify the source of retrieved information.\n\n**Adding Agentic RAG**\n\nAnother part of the project was exposing the retrieval system as a tool for an agent.\n\nI created a knowledge-base search tool:\n\nknowledge_base_search(query)\n\nThe agent can use this tool to retrieve relevant information before generating the final response.\n\nThe workflow becomes:\n\nUser Query\n\n↓\n\nAgent\n\n↓\n\nknowledge_base_search()\n\n↓\n\nFAISS + BM25\n\n↓\n\nHybrid Ranking\n\n↓\n\nRetrieved Context\n\n↓\n\nLLM\n\n↓\n\nFinal Answer\n\nThe agent is instructed to use the retrieved information when answering questions and avoid generating unsupported information. If the knowledge base doesn’t contain sufficient information, the system can indicate that the information isn’t available.\n\n**LLM**\n\nFor response generation, I used Qwen2.5-72B-Instruct through InferenceClientModel.\n\nThe LLM doesn’t perform the initial retrieval. Instead, the retrieval tool provides relevant information to the agent, which is then used as context for generating the response.\n\nThis separation between retrieval and generation helps keep the knowledge base as the primary source of information.\n\n**Implementation**\n\nThe project was initially developed in Google Colab and later reorganized into a single executable Python application in VS Code.\n\nThe main pipeline includes:\n\nDocuments\n\n↓\n\nChunking\n\n↓\n\nEmbeddings\n\n↓\n\nFAISS Index\n\n+\n\nBM25 Index\n\n↓\n\nHybrid Retrieval\n\n↓\n\nRanking\n\n↓\n\nAgent Tool\n\n↓\n\nQwen2.5-72B-Instruct\n\n↓\n\nAnswer\n\nThe embedding model and retrieval indexes are initialized when the application starts, so the complete retrieval pipeline doesn’t need to be rebuilt for every question.\n\n**Testing**\n\nI tested the system with questions related to the information available in the knowledge base.\n\nFor example:\n\nWhy is document chunking important in a RAG system?\n\nThe hybrid retrieval system ranked the Document Chunking source highest because it directly addressed the question.\n\nThe retrieved context was then passed to the agent and used to generate the final response.\n\nI also considered questions where the required information isn’t available in the knowledge base. In those cases, the system is designed to avoid simply falling back to the LLM’s general knowledge.\n\n**What I Learned**\n\nWorking on this project helped me understand that building a RAG system isn’t just about connecting an LLM to a vector database.\n\nSeveral components have a direct impact on the quality of the final answer:\n\nOne of the biggest takeaways for me was understanding the importance of the retrieval stage. Even a powerful LLM can produce a poor answer if the relevant information isn’t retrieved properly.\n\n**What’s Next?**\n\nThere are several improvements I would like to explore next, including:\n\nThese are some of the areas identified for future improvement in the current system.\n\n**Conclusion**\n\nThis project gave me practical experience with Hybrid RAG, vector search, BM25, FAISS, embeddings, agentic tool use, and LLM-based generation.\n\nThe key idea I took away is:\n\nGood RAG isn’t only about the LLM — the quality of the retrieved context matters just as much.\n\nBy combining semantic and keyword-based retrieval, the system can use both the meaning of a query and its important exact terms before generating a response.\n\nThis was a great learning experience, and I’m looking forward to exploring more areas of AI Engineering and RAG systems. 🚀\n\n⸻\n\n**Tech Stack**\n\nPython · FAISS · BM25 · Sentence Embeddings · Qwen2.5-72B-Instruct · Agentic RAG · Vector Search · VS Code", "url": "https://wpnews.pro/news/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai", "canonical_source": "https://dev.to/melvin_sabu/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai-h33", "published_at": "2026-08-29 09:34:36+00:00", "updated_at": "2026-08-29 09:48:43.696004+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents", "developer-tools"], "entities": ["FAISS", "BM25", "Qwen2.5-72B-Instruct", "InferenceClientModel"], "alternates": {"html": "https://wpnews.pro/news/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai", "markdown": "https://wpnews.pro/news/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai.md", "text": "https://wpnews.pro/news/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai.txt", "jsonld": "https://wpnews.pro/news/building-a-hybrid-rag-system-with-faiss-bm25-and-agentic-ai.jsonld"}}