Building a Self-Evaluating RAG Agent with LangGraph, Qdrant Hybrid Search & Phoenix

A new open-source RAG agent combines Qdrant hybrid search (dense + BM25 with RRF fusion), LangGraph ReAct routing with safety overrides, an MCP server for tool exposure, and Arize Phoenix for observability and LLM-as-judge evaluation, as demonstrated in a tutorial by developer Ranupama Sharma. The system addresses production RAG failures by using query-aware routing to handle complex queries like year-over-year comparisons and automatically scores answers for hallucination and correctness. Full code is available on GitHub at https://github.com/sharmaranupama/hybrid-rag-mcp-agent.git.

This post walks through building a RAG agent that combines Qdrant hybrid search dense + BM25 with RRF fusion , LangGraph ReAct routing with safety overrides, an MCP server for tool exposure, and Arize Phoenix for observability and LLM-as-judge evaluation. Full code included. Everyone has seen the RAG demo. Embed some docs, retrieve top-K chunks, prompt an LLM, get an answer. It works in a notebook. It falls apart in production. A simple example: a user asks “How has parental leave changed between 2024 and 2026?” A typical RAG pulls the top 5 chunks — most end up from 2024 because there’s more data there. The user gets half an answer. This system catches that it’s a comparison query, searches each year separately, and gives a complete answer. If something goes wrong, built-in evaluation scores every answer for hallucination and correctness automatically. That’s the gap between a demo and a system. This post is about closing it. The full code is available here:https://github.com/sharmaranupama/hybrid-rag-mcp-agent.git This post assumes a basic familiarity with modern LLM-based systems. In particular, it helps to understand: If you’re new to these concepts, you can still follow along, but a quick refresher will help you get the most out of this post. Before diving into the system, let’s quickly define the core ideas used throughout. a. Retrieval-Augmented Generation RAG agent combines document retrieval with LLM reasoning. It first retrieves relevant context from a knowledge base and then generates an answer grounded in that data. Example: Query: "What is the remote work policy?"→ Retrieve relevant documents→ LLM generates answer using retrieved context b. Embeddings are numerical vector representations of text that capture its meaning. Similar texts have similar vectors, enabling semantic search. Embeddings convert text into numbers so machines can understand meaning , not just keywords. Example: Text A: "remote work policy"Text B: "work from home rules"Text C: "health insurance benefits"After embedding:A ≈ B similar meaning → vectors close together A ≠ C different meaning → vectors far apart c. BM25 is a keyword-based ranking algorithm that scores documents based on how important query terms are within them using term frequency and inverse document frequency . Example: Query: "vacation policy"Doc A: "Vacation policy allows 20 days PTO" → High score Doc B: "Health insurance details" → Low score d. Hybrid Search combines BM25 keyword search with dense embeddings semantic search to retrieve better results. Example: Query: "WFH rules"Doc: "Remote work policy allows flexible work from home"BM25 → weak match Embeddings → strong match Hybrid → retrieves correct document e i Simple search : Single-step retrieval + answer generation e ii Complex search : Multi-step reasoning e.g., comparisons, aggregation Example: Simple: "What is the vacation policy?"Complex: "Compare vacation policy between 2024 and 2026" f. Query-aware routing dynamically selects the appropriate workflow based on the query. Example: "Compare policies between 2024 and 2026"→ route to comparison workflow"What is the policy?"→ route to simple RAG search g. LLM-as-judge evaluation uses a language model to automatically evaluate responses based on criteria like correctness, relevance, and hallucination. The system is built in three cooperating parts. Part 1 : Hybrid Search & Retrieval a. It starts with data.csv as the source, runs through ingest.py which produces dense + BM25 vectors with int8 quantization, and stores them in Qdrant with named vectors, RRF fusion, and payload indexes. b. Query.py handles runtime retrieval — hybrid search, payload filtering, and multi-year balanced fetch. c. langgraph flow.py sits alongside it, handling the extract years → fetch per-year context → compare and summarise workflow. d. config.py underpins everything as the single settings file and LLM provider abstraction. Part 2: Agent & MCP takes the retrieval layer as input. a. agent.py runs the ReAct loop with routing guardrails think → act → synthesize . b. mcp server.py wraps the agent and exposes three tools via FastMCP to any MCP client. c. app.py is the Streamlit UI — five tabs covering data ingestion, agent chat, Qdrant & Phoenix dashboards, datasets & experiments, and RAG eval. Part 3 : Observability & Evaluation is handled by Arize Phoenix. a. Every query produces a full span hierarchy -latency, token counts, full audit trail captured by Phoenix tracing. b.phoenix datasets.py manages versioned query sets and experiment runs. c. The LLM-as-judge layer scores every answer for hallucination 0% , QA correctness 100% , and context relevance, using either Gemini or Ollama as the judge. All three parts share a single LLM Provider layer at the base -Ollama local llama3.2, nomic-embed-text or Gemini API gemini-2.5-flash-lite , switchable via LLM PROVIDER in .env or through the UI with no code changes needed. In the next sections, we break down ingestion, retrieval, and evaluation. Most RAG pipelines embed a document once. This system indexes every document twice — and that second representation is what makes the difference between finding “MFA” and missing it entirely. Each document gets two vector representations: a. Dense vectors 768-dim, nomic-embed-text via Ollama capture semantic meaning, so “flexible working” matches “remote policy” even without word overlap b. Sparse BM25 vectors FastEmbed, server-side IDF — capture keyword frequency, so exact terms like acronyms and codes are never missed Qdrant collection is configured with int8 scalar quantization ~4x memory reduction and payload indexes on key metadata fields enabling filtered search at query time, not after. q client.create collection vectors config={"dense": VectorParams size=768, distance=Distance.COSINE },sparse vectors config={"bm25": SparseVectorParams modifier=Modifier.IDF },quantization config=ScalarQuantization …INT8… , Dense vectors capture meaning but miss exact terms like acronyms or codes. Hybrid search fixes this by combining dense semantic and BM25 keyword signals. Qdrant runs both as sub-queries and merges the results using Reciprocal Rank Fusion RRF , which ranks documents based on position rather than raw scores. prefetch = models.Prefetch query=dense vec, using="dense", limit=20 ,models.Prefetch query=sparse vec, using="bm25", limit=20 , results = q client.query points prefetch=prefetch,query=models.FusionQuery fusion=models.Fusion.RRF , Filters are applied at query time , keeping results deterministic for specific data slices. Not every query should hit the same retrieval path. A question like “what is the remote work policy?” needs a single hybrid search. A question like “what changed between 2024 and 2026?” needs two independent searches — one per time period — then a synthesis step. Sending both through the same pipeline gives you either incomplete context or polluted results. LangGraph handles this by making the routing decision explicit in the graph structure rather than hiding it inside a prompt. MCP Tool Exposure Model Context Protocol MCP is an open standard for exposing tools to LLM clients. Using FastMCP, we expose three tools direct RAG search, the comparison workflow, and the full ReAct agent in under 50 lines of Python. The same tools are then callable from any MCP-compatible client without any additional integration code. python from mcp.server.fastmcp import FastMCPmcp = FastMCP "Company-Knowledge" @mcp.tool def search company policy query: str - str:"""Search for a specific company policy using RAG."""return ask rag query @mcp.tool def compare policies query: str - str:"""Compare policies across different years."""return graph app.invoke {"query": query, …} "final answer" @mcp.tool def ask agent query: str - str:"""ReAct agent - reasons step by step."""return agent app.invoke {…} "answer" if name == " main ":mcp.run The MCP inspector will be available at localhost:6274 The ReAct Loop The agent runs a standard ReAct loop reason, pick a tool, observe the result, repeat until it has enough context to answer. Two tools are available: a. search company policy : single hybrid search, optional metadata filter b. compare policies : runs two filtered searches independently, then synthesizes The LLM decides which tool to call based on the query. For most questions, it picks search company policy and returns in one step. For comparison queries, it invokes the dedicated workflow. VALID ACTIONS = {"search company policy", "compare policies", "finish"}REACT PROMPT = """You are a company policy assistant. You can call these tools:search company policy -- find a specific policy, rule, or benefit use for single lookups compare policies -- compare policies across different years use when the question mentions comparing, differences, changes between years, or references two or more years like "2024 and 2026" finish -- return your final answer to the userROUTING RULES — follow these strictly:- If the question contains words like "compare", "difference", "changed", "vs", "between ... and ...", or mentions TWO OR MORE years → you MUST use compare policies- If the question asks about a single policy, benefit, or rule → use search company policy- Once you receive an Observation with an answer, call finish immediately with the full answer.Your reasoning so far:{scratchpad}Question: {question}Respond with a JSON object containing exactly these three fields:- "thought": your reasoning about what to do next- "action": one of "search company policy", "compare policies", or "finish"- "action input": the query to pass to the tool, or your complete final answer if action is "finish"Example responses:{{"thought": "I need to look up the remote work policy.", "action": "search company policy", "action input": "remote work policy"}}{{"thought": "The user wants to compare parental leave between 2024 and 2026.", "action": "compare policies", "action input": "compare parental leave policy between 2024 and 2026"}}{{"thought": "The observation contains the answer. I will finish.", "action": "finish", "action input": "The remote work policy allows..."}}""" Safety Override on Routing LLMs misroute occasionally — a comparison question phrased ambiguously gets sent to search company policy , which returns a muddled single-pass result. Rather than accepting that failure silently, a lightweight regex heuristic checks the query before tool execution. If it detects comparison intent but the wrong tool was selected, it overrides the routing in-flight. php def detect comparison query question: str - bool:compare words = "compare","difference","changed","vs","between" has compare = any w in question.lower for w in compare words years = re.findall r"20\d{2}", question return has compare and len set years = 2 It’s a small addition, but it closes a real failure mode without touching the LLM. The Comparison Workflow When compare policies is called, LangGraph invokes a dedicated sub-graph: independent filtered retrievals one per year are executed inside the fetch node, followed by a merge step that builds a structured context before generation. The LLM never sees unbalanced input. A working RAG system and a debuggable RAG system are different things. When an answer is wrong, you need to know whether the problem was retrieval, routing, or generation — and how long each step took.Logging prompts and responses doesn’t tell you that. Traces do. Phoenix Arize instruments the entire pipeline via OpenTelemetry. Every LangGraph node, every Qdrant call, every LLM generation becomes a span with its own latency, inputs, and outputs. The full hierarchy is visible in one view. A typical query produces a span tree like this: a. Agent node — total end-to-end latency b. Tool selection — LLM reasoning step, which tool was picked and why c. Retrieval — Qdrant hybrid search, chunks returned and top scores d. LLM generation — prompt sent, response received, token counts For comparison queries, the retrieval branch splits into two parallel spans one per filtered search making it immediately obvious if one segment returned weak results. Latency Breakdown : Phoenix surfaces per-node latency, so you can see exactly where time is spent. Retrieval is consistently fast sub-200ms . The bottleneck is almost always LLM generation and with swappable backends, you can compare Ollama vs. Gemini latency directly in the trace view without changing any instrumentation. No custom logging. No separate dashboard. The observability is a side effect of how the system is wired. Traces tell you how the system ran. Evaluation tells you how well. Most teams handle this in a separate notebook, days after deployment, on a sample of outputs. This system does it automatically, on live traces, with scores written back to the same spans they evaluate. Three evaluators run after each query: a. Hallucination — did the answer contradict or fabricate beyond the retrieved context? b. QA Correctness — did the answer actually address the question? c. Context Relevance — did the retrieved chunks contain what was needed to answer? Phoenix’s LLM-as-judge evaluators handle the scoring. The scores are then logged back as span annotations — visible inline with the trace, alongside latency. annotations.append SpanAnnotationData name="hallucination",span id=str span id ,annotator kind="LLM",result={"label": row "label" , "score": 1.0 if row "label" == "hallucinated" else 0.0}, Results The 50% context relevance is expected as out-of-scope queries retrieve something the system doesn’t refuse , but what’s retrieved isn’t genuinely relevant. That’s not a retrieval failure; it’s the correct behavior surfaced clearly by the eval. This is the value of closing the evaluation loop inside the observability platform: you’re not just seeing that something went wrong — you’re seeing which step produced the gap. Live trace evaluation tells you how the system performs on real queries. Experiments tell you how it performs consistently — across a controlled set of inputs, repeatable across model changes or retrieval config updates. The built-in test set covers 10 queries across three categories: The experiment pipeline runs all 10 queries through the full RAG pipeline, measures retrieval score, topic hit rate, and latency per category, then uploads results to Phoenix as a versioned dataset. Every run is timestamped and tied to the model and provider used — so you can compare runs across configurations without losing history. Results In-scope and comparison queries hit perfect scores. Out-of-scope scores at 0.92, the system retrieves the closest available context rather than hallucinating, which is the right failure mode. Comparison queries are the slowest, as expected — two independent retrieval calls plus a synthesis step. Still well under 2 seconds end-to-end. a. Hybrid search over dense-only : BM25 catches exact terms that semantic embeddings miss. RRF fusion combines both without tuning weights. b. Filtered retrieval for targeted queries : Instead of hoping the global search returns balanced results, the comparison workflow queries each segment independently guaranteeing coverage. c. Safety overrides on tool routing : LLMs sometimes misroute queries. A regex-based heuristic catches comparison questions sent to the wrong tool and corrects in-flight. d. Evaluation as a first-class feature : Not a separate notebook but the eval pipeline is built into the UI, uses live traces, and logs scores back to the observability platform. e. Swappable LLM backends. A single config controls provider routing. Switch between Ollama free, local and Gemini fast, cloud at runtime — no restart, no code changes. a. Chunking for longer documents : the current model works for structured data but doesn’t scale to unstructured PDFs b. Multi-modal ingestion : PDFs, Confluence pages etc If you’re building RAG systems and wondering how to go from “it works in a notebook” to “it’s observable, evaluated, and deployable” then this is the architecture I’d start with. The tools are all open-source. The hard part isn’t any single component it’s wiring them together so they reinforce each other. Questions or feedback? Drop a comment happy to go deeper on any section. Building a Self-Evaluating RAG Agent with LangGraph, Qdrant Hybrid Search & Phoenix https://pub.towardsai.net/beyond-the-demo-building-a-rag-system-from-scratch-that-routes-retrieves-and-evaluates-itself-4bb1dc66e524 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.