{"slug": "i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know", "title": "I Built a Hybrid RAG App That Talks to My PDF — and Knows When to Say “I Don’t Know”", "summary": "A developer built a hybrid RAG (Retrieval-Augmented Generation) application that answers questions from PDF documents while refusing to answer out-of-scope queries, using a combination of semantic search and BM25 reranking with an anti-hallucination gate. The app runs fully locally with Ollama, ChromaDB, and a React frontend, demonstrating how to prevent AI hallucinations by declining to answer when no relevant document passage exists.", "body_md": "Imagine you upload an insurance policy PDF and ask:\n\n*“What is my wind/hail deductible?”*\n\nA good RAG system should find the exact clause and answer from it.\n\nNow ask:\n\n*“What is the capital of France?”*\n\nA *confident* RAG system is dangerous here. The document has nothing about France. But a chatty LLM will happily invent an answer unless you stop it.\n\nThat tension — **find the right passage** and **refuse when there is none** — is what this article is about.\n\nI built a full-stack **Talk to your PDF** app on my laptop: React UI, FastAPI backend, Ollama (llama3 + nomic-embed-text), ChromaDB, and **BM25 reranking**. The twist is hybrid retrieval plus a simple anti-hallucination gate, so out-of-scope questions get a polite decline instead of a confident guess.\n\n**RAG** means: find relevant pieces of your documents, then ask an LLM to answer using those pieces.\n\n**Hybrid RAG** means: do not rely on only one way of finding those pieces.\n\nThink of searching a library two ways at once:\n\nSemantic search is great when you ask about “storm damage” and the policy says “wind/hail.” Keyword search is great when you need the word deductible, a section number, or a precise policy term.\n\nHybrid RAG uses both strengths. In this article, that looks like:\n\nMeaning finds the neighborhood. Exact terms pick the right house.\n\n**Hallucination** is when an AI answers with confidence even though it does not actually know — or when the answer is not supported by your document.\n\n**Anti-hallucination** is the set of guardrails that reduce that behavior.\n\nIn plain English: it is how you teach the system to say *“I don’t know from this document”* instead of making something up.\n\nIn this app, anti-hallucination is not one magic trick. It is three simple layers:\n\nYou still see source excerpts in the UI. That matters. Trust comes from evidence, not from a fluent paragraph.\n\nEarlier local RAG experiments taught me the basic loop: ingest → chunk → embed → retrieve → generate.\n\nThat loop works. But two practical problems kept showing up with real policy-style documents:\n\nSo I built a demo that feels closer to a real product:\n\nStill fully local. Still no API keys.\n\nSuppose your policy PDF contains something like:\n\n```\nSection 4.2 — Wind/Hail DeductibleFor losses caused by windstorm or hail, a separate deductible of $2,500 applies.This deductible is independent of the standard all-peril deductible in Section 3.1.Mold remediation is excluded except where resulting from a covered water peril.\n```\n\n**Question A:** *“What is my wind/hail deductible?”*\n\n**Question B:** *“What is the capital of France?”*\n\nThat second case is the demo moment I care about most. A RAG demo that only answers in-document questions is incomplete. A RAG demo that **declines** out-of-scope questions is teaching the right instinct.\n\nThe big idea is not “more models.” It is **better retrieval + clearer refusal**.\n\nThe embedding model turns each chunk (and your question) into a vector. ChromaDB finds the nearest neighbors by meaning.\n\nThis is how *“storm damage”* can still find *“wind/hail”* language.\n\nBM25 is a classic keyword ranking method. It rewards chunks that contain the query terms, with sensible weighting for term rarity and document length.\n\nIn policy documents, that helps questions like:\n\nI used a simple pattern:\n\n```\nQuestion  → embed  → semantic top-20 candidates  → BM25 rerank  → top-4 chunks  → anti-hallucination gate  → llama3 answer (or polite refusal)\n```\n\nSemantic search casts a wide net. BM25 chooses the best fish. The LLM only sees a short, high-signal context window.\n\nYou can disable reranking (use_rerank: false / --no-rerank) to compare quality side by side. That comparison alone is a useful learning exercise.\n\nI kept the RAG core small and wrapped it with a web API and React frontend.\n\nEverything runs on one machine. No Docker required for the demo. No documents leave your laptop.\n\nPDF → extract text → recursive chunk (~500 chars, 50 overlap) → embed → store in ChromaDB → rebuild BM25 index\n\nFrom the UI, this is a drag-and-drop upload. Under the hood, FastAPI calls the same ingest path the CLI uses.\n\nQuestion → semantic candidates → BM25 rerank → relevance gate passes → Llama 3 answers from excerpts → UI shows answer + source snippets (semantic rank, BM25 score, distance)\n\nSame retrieval path — but the gate fails. The API returns a refusal message and marks refused: true. No invented facts. No fake confidence.\n\nEvery answer (and many refusals) can show the chunks that were considered. That is how you debug RAG honestly: look at what was retrieved before blaming the model.\n\n``` python\ndef retrieve(question, store, bm25, *, top_k=4, use_rerank=True):    candidates = store.query(question, top_k=20)    if not candidates:        return []if not use_rerank:        return candidates[:top_k]    return bm25.rerank(question, candidates, top_k=top_k)\n```\n\nWide semantic recall first. Precise keyword ordering second.\n\n``` python\ndef assess_relevance(chunks):    if not chunks:        return False, REFUSAL_MESSAGEbest_distance = min(chunk[\"distance\"] for chunk in chunks)    if best_distance <= STRONG_SEMANTIC_DISTANCE:        return True, \"\"    if best_distance > MAX_SEMANTIC_DISTANCE:        return False, REFUSAL_MESSAGE    best_bm25 = max(        (chunk.get(\"bm25_score\") or 0.0) for chunk in chunks    )    if best_bm25 < MIN_BM25_SCORE:        return False, REFUSAL_MESSAGE    return True, \"\"\n```\n\nIf retrieval looks strong, proceed. If it looks off-topic, refuse early. If it is borderline, require some keyword overlap before trusting the LLM.\n\n```\nSYSTEM_PROMPT = \"\"\"You are a document Q&A assistant.Answer questions using ONLY the provided document excerpts.If the answer is not stated in the excerpts, respond with exactly:\"I cannot answer that based on the document.\"Do not use outside knowledge. Do not guess or invent facts.\"\"\"\n```\n\nPrompting alone is not enough. Combined with the retrieval gate, it becomes a practical safety net for a local demo.\n\n```\nLLM_MODEL = \"llama3\"EMBED_MODEL = \"nomic-embed-text\"CHUNK_SIZE = 500CHUNK_OVERLAP = 50RETRIEVAL_CANDIDATES = 20TOP_K = 4STRONG_SEMANTIC_DISTANCE = 280MAX_SEMANTIC_DISTANCE = 450MIN_BM25_SCORE = 0.5\n```\n\nTighten the distance thresholds if you want fewer hallucinations and more refusals. Loosen them if you want fewer false declines on paraphrased questions.\n\n```\npython main.py ingest data/sample-policy.pdfpython main.py ask \"What is my wind/hail deductible?\" --show-sourcespython main.py search \"mold coverage\" --no-rerankpython main.py status\n```\n\nI kept the CLI because retrieval debugging is faster in the terminal. The React UI is for the product story. Both share the same core services.\n\n```\nhybrid-rag-pdf/├── api/                      # FastAPI REST layer│   ├── main.py│   └── schemas.py├── frontend/                 # React + Vite SPA├── rag/│   ├── config.py             # models, chunking, retrieval thresholds│   ├── chunker.py            # recursive splitting + overlap│   ├── document_loader.py    # PDF/TXT loading│   ├── embedder.py           # Ollama embeddings│   ├── vector_store.py       # ChromaDB│   ├── bm25_index.py         # build, persist, rerank│   ├── retriever.py          # semantic + BM25 orchestration│   ├── anti_hallucination.py # relevance gate│   ├── query.py              # prompt + generate + sources│   └── services/│       ├── ingest_service.py│       └── chat_service.py├── main.py                   # CLI├── data/sample-policy.pdf└── docs/architecture.md\n```\n\nClear layers made the project easier to explain — and easier to extend later.\n\n**BM25 is surprisingly useful on policy language.** Exact terms matter. Vector search alone often got “close enough.” Reranking made “exactly right” more common.\n\n**Refusal is a feature, not a failure.** Early versions of RAG demos feel broken when they say “I cannot answer.” For document Q&A, that is often the correct product behavior.\n\n**Sources make the demo trustworthy.** Showing excerpts, ranks, and scores turns a black-box answer into something a reader can verify in seconds.\n\n**Llama 3 raised answer quality**, but retrieval and gating still did the heavy lifting. A stronger model on weak context is still a confident storyteller.\n\n**A web UI changes how people understand RAG.** Watching upload → ask → grounded answer → refusal is more memorable than reading CLI output.\n\nThis kind of project is great if you are:\n\nYou do not need a GPU farm. You need Python, Node.js, Ollama, and one text-based PDF.\n\nNatural extensions from here:\n\nEven without those upgrades, the current demo already teaches the lesson I wanted: **good RAG is retrieval quality plus the courage to refuse.**\n\nVector search finds meaning. Keyword search finds precision. Together they make retrieval sharper.\n\nBut the quiet hero of this project is the refusal path.\n\nAnyone can build a chatbot that always answers. The more interesting system is the one that answers from your PDF when the evidence is there — and says *“I cannot answer that based on the document”* when it is not.\n\nIf you try this yourself, do one complete loop:\n\nThat five-minute exercise teaches more about trustworthy RAG than another hour of model shopping.\n\nYou can download the complete source code from GitHub: [https://github.com/parivshah/hybrid-rag-pdf](https://github.com/parivshah/hybrid-rag-pdf)\n\nThe repository includes setup instructions, architecture notes, a sample policy PDF, and both the web demo and CLI. Please let me know if you liked this article or have any questions, feedback, or suggestions. You can connect with me on [LinkedIn](https://www.linkedin.com/in/pariv-shah/).\n\n[I Built a Hybrid RAG App That Talks to My PDF — and Knows When to Say “I Don’t Know”](https://pub.towardsai.net/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know-5e4044a54921) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know", "canonical_source": "https://pub.towardsai.net/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know-5e4044a54921?source=rss----98111c9905da---4", "published_at": "2026-07-16 00:01:02+00:00", "updated_at": "2026-07-16 00:30:22.497036+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-ethics", "ai-agents"], "entities": ["Ollama", "ChromaDB", "llama3", "nomic-embed-text", "BM25", "FastAPI", "React"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know", "markdown": "https://wpnews.pro/news/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know.md", "text": "https://wpnews.pro/news/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know.txt", "jsonld": "https://wpnews.pro/news/i-built-a-hybrid-rag-app-that-talks-to-my-pdf-and-knows-when-to-say-i-dont-know.jsonld"}}