Building a Closed-Domain Agentic AI Knowledge Assistant with Hybrid RAG

A developer built an 'Agentic AI Knowledge Assistant' that combines hybrid retrieval-augmented generation (RAG) with an LLM agent to answer questions strictly from a predefined knowledge base. The system uses LangChain for text splitting, sentence-transformers for embeddings, and FAISS/BM25 for hybrid search, with a smolagents CodeAgent and Qwen2.5-72B-Instruct for response generation. The assistant is designed to refuse answering when the knowledge base lacks sufficient information, ensuring closed-domain responses.

I recently built an "Agentic AI Knowledge Assistant" that combines "Retrieval-Augmented Generation RAG , hybrid search, and an LLM agent" to answer questions strictly from a predefined knowledge base. One of the main goals of this project was to address a common problem with LLM applications: the model should not answer questions using its general pretrained knowledge when the required information is not available in the knowledge base. How it works The system follows a retrieval-first approach: User Question → Hybrid Retrieval → Relevant Context → AI Agent → Final Answer: The knowledge base is divided into smaller chunks using "LangChain's RecursiveCharacterTextSplitter". Each chunk is converted into embeddings using: "sentence-transformers/all-MiniLM-L6-v2" For retrieval, I implemented two approaches: The workflow looks like this: User Query ↓ Hybrid Search ↓ FAISS || BM25 Vector || Keyword Search || Search ↓ Relevant Knowledge Base Chunks ↓ Retrieved Context ↓ Qwen Language Model ↓ Final Answer Agentic AI Layer The retrieval system is exposed to the agent through a custom: knowledge base search tool. The project uses smolagents CodeAgent along with the: "Qwen/Qwen2.5-72B-Instruct" model for response generation. The agent retrieves relevant information from the knowledge base before generating a response. Closed-Domain Knowledge Restriction One of the most important features of this project is the strict knowledge-base-only approach. The assistant is instructed not to use: If the knowledge base does not contain sufficient information, the intended response is: "The knowledge base does not contain enough content to answer this question." This makes the system more suitable for applications where responses need to remain within a controlled information domain. Technologies Used What I Learned Building this project helped me understand how different components of an AI application work together rather than treating an LLM as a standalone system. In particular, I gained practical experience with: Future Improvements Some improvements I would like to implement next include: This project was a great hands-on experience in understanding how RAG + Hybrid Search + Agentic AI + LLMs can be combined to build a more controlled AI assistant.