As part of my AI Engineering journey, I recently worked on a project that helped me understand how Retrieval-Augmented Generation (RAG) works in practice.
I built a Hybrid RAG system that combines FAISS vector search and BM25 keyword search to retrieve relevant information from a knowledge base and use it to generate grounded answers.
In this post, Iβll briefly share what I built, how the system works, and some of the things I learned along the way.
Why RAG?
Large Language Models are great at generating natural-language responses, but they may not have access to information contained in a specific document or knowledge base.
RAG addresses this by first retrieving relevant information from an external knowledge base and then providing that information to the LLM as context.
The basic workflow is:
User Query
β
Retrieve Relevant Information
β
Provide Context to LLM
β
Generate Answer
For my project, I wanted to take this a step further by combining semantic search and keyword search.
π Hybrid Retrieval
The system uses two retrieval methods:
Vector Search with FAISS
Document content is divided into smaller chunks and converted into vector embeddings.
These embeddings are stored in a FAISS index, which is used to find documents that are semantically similar to the userβs query.
This is useful even when the query and the document use different wording.
Keyword Search with BM25
The second retrieval method is BM25.
BM25 focuses on the occurrence and importance of terms in the query and documents. This makes it useful for exact terminology, technical terms, names, and identifiers.
Instead of depending on only one retrieval method, both approaches are combined.
User Query
β
ββββββββββββ΄βββββββββββ
β β
FAISS Search BM25 Search
Semantic Search Keyword Search
β β
ββββββββββββ¬βββββββββββ
β
Hybrid Ranking
β
Relevant Context
β
LLM
β
Final Answer
The FAISS and BM25 scores are normalized and combined using weighted scoring. The results are then ranked, and the highest-ranked chunks are used as the final context.
Document Chunking
Before retrieval, the documents are divided into smaller chunks.
Chunking is an important part of a RAG pipeline because the size of a chunk can affect retrieval quality.
Very small chunks may lose important context, while very large chunks may contain unnecessary information and reduce retrieval precision.
So the goal is to find a balance between context and retrieval accuracy.
Each chunk is also stored with metadata such as its document ID and title, making it easier to identify the source of retrieved information.
Adding Agentic RAG
Another part of the project was exposing the retrieval system as a tool for an agent.
I created a knowledge-base search tool:
knowledge_base_search(query)
The agent can use this tool to retrieve relevant information before generating the final response.
The workflow becomes:
User Query
β
Agent
β
knowledge_base_search()
β
FAISS + BM25
β
Hybrid Ranking
β
Retrieved Context
β
LLM
β
Final Answer
The agent is instructed to use the retrieved information when answering questions and avoid generating unsupported information. If the knowledge base doesnβt contain sufficient information, the system can indicate that the information isnβt available.
LLM
For response generation, I used Qwen2.5-72B-Instruct through InferenceClientModel.
The LLM doesnβt perform the initial retrieval. Instead, the retrieval tool provides relevant information to the agent, which is then used as context for generating the response.
This separation between retrieval and generation helps keep the knowledge base as the primary source of information.
Implementation
The project was initially developed in Google Colab and later reorganized into a single executable Python application in VS Code.
The main pipeline includes:
Documents
β
Chunking
β
Embeddings
β
FAISS Index
BM25 Index
β
Hybrid Retrieval
β
Ranking
β
Agent Tool
β
Qwen2.5-72B-Instruct
β
Answer
The embedding model and retrieval indexes are initialized when the application starts, so the complete retrieval pipeline doesnβt need to be rebuilt for every question.
Testing
I tested the system with questions related to the information available in the knowledge base.
For example:
Why is document chunking important in a RAG system?
The hybrid retrieval system ranked the Document Chunking source highest because it directly addressed the question.
The retrieved context was then passed to the agent and used to generate the final response.
I also considered questions where the required information isnβt available in the knowledge base. In those cases, the system is designed to avoid simply falling back to the LLMβs general knowledge.
What I Learned
Working on this project helped me understand that building a RAG system isnβt just about connecting an LLM to a vector database.
Several components have a direct impact on the quality of the final answer:
One of the biggest takeaways for me was understanding the importance of the retrieval stage. Even a powerful LLM can produce a poor answer if the relevant information isnβt retrieved properly.
Whatβs Next?
There are several improvements I would like to explore next, including:
These are some of the areas identified for future improvement in the current system.
Conclusion
This project gave me practical experience with Hybrid RAG, vector search, BM25, FAISS, embeddings, agentic tool use, and LLM-based generation.
The key idea I took away is:
Good RAG isnβt only about the LLM β the quality of the retrieved context matters just as much.
By combining semantic and keyword-based retrieval, the system can use both the meaning of a query and its important exact terms before generating a response.
This was a great learning experience, and Iβm looking forward to exploring more areas of AI Engineering and RAG systems. π
βΈ»
Tech Stack
Python Β· FAISS Β· BM25 Β· Sentence Embeddings Β· Qwen2.5-72B-Instruct Β· Agentic RAG Β· Vector Search Β· VS Code