{"slug": "why-do-ai-models-hallucinate-and-how-can-we-reduce-it", "title": "Why Do AI Models Hallucinate, and How Can We Reduce It?", "summary": "A developer building a retrieval-augmented generation (RAG) document Q&A project found that connecting an LLM to a knowledge source reduces but does not eliminate hallucinations, which can arise either when the retriever returns irrelevant passages or when the model adds claims unsupported by correct passages. The writeup recommends diagnosing which stage fails and combining better retrieval (hybrid search, reranking, metadata filters), context-constrained prompting, tighter chunking, and validation checks, noting that similarity scores are not confidence percentages.", "body_md": "AI models can generate impressive answers in seconds. But sometimes, they confidently give us information that is incorrect or not supported by any reliable source. This is called an **AI hallucination.**\n\nWhile building my RAG-based document Q&A project, I explored how we can reduce hallucinations and why simply connecting an LLM to a knowledge source isn't enough.\n\nLet's understand the problem and explore some practical solutions.\n\nLarge Language Models (LLMs) generate responses by learning patterns in language and predicting what text should come next. They don't automatically verify every statement against facts.\n\nHallucinations can happen because of:\n\nIn simple terms, a fluent answer isn't necessarily a factual answer.\n\nRAG combines **information retrieval** with **answer generation**. Instead of relying only on what the model already knows, it retrieves relevant information from an external knowledge source and provides that information as context.\n\nFor example, if you ask, “What are the three main recommendations in this particular report?”, RAG retrieves relevant passages from the report and provides them to the model before it generates an answer.\n\nThe process looks like this:\n\n```\nDocument → Chunking → Embeddings → Vector Database → Retrieval → Context → LLM → Answer\n```\n\nThis gives the model evidence to work with rather than asking it to answer from memory alone.\n\nHowever, **RAG reduces the risk of hallucinations; it doesn't eliminate them.**\n\nTwo failure points in RAG: the wrong passages are retrieved, or the model adds claims unsupported by the right passages.\n\nRAG depends on both **retrieval** and **generation**. Problems in either stage can produce unreliable answers.\n\nImagine asking a question whose answer appears on page 42 of a PDF. If the retriever returns unrelated passages from pages 12, 18, and 63, the LLM may not have the evidence it needs. It might still generate a plausible answer.\n\nEven when retrieval works correctly, the LLM can misunderstand the retrieved text.\n\nThis means we need to identify **where the failure happens**: retrieval or generation before deciding how to fix it.\n\nRAG is only as useful as the evidence it retrieves and how well the model uses that evidence.\n\nThere isn't one universal fix. Several techniques work together:\n\n**Better retrieval**: Use suitable embedding models, metadata filters, hybrid search, or reranking to find relevant evidence.\n\n**Better prompting**: Explicitly instruct the model to answer from the supplied context and acknowledge missing information.\n\n```\nprompt = f\"\"\"\nAnswer the question using only the context below.\n\nIf the context does not contain enough information,\nclearly state that you don't have enough information.\nDo not invent unsupported details.\n\nContext:\n{context}\n\nQuestion:\n{question}\n\nAnswer:\n\"\"\"\n```\n\nThis prompt encourages the model to base its answer on the retrieved passages instead of filling gaps with plausible-sounding information. If the evidence is insufficient, the model is instructed to acknowledge that limitation.\n\n**Better context:** Improve chunk size and overlap, remove irrelevant passages, and avoid overwhelming the model with unnecessary text.\n\n**Confidence and relevance checks**: Use similarity thresholds or other validation methods to detect weak retrieval results. Similarity scores alone do not prove that an answer is correct.\n\n(One important detail: a similarity score is not the same as a confidence percentage. Its meaning depends on the retrieval method and score configuration. Even a highly similar passage may not contain sufficient evidence to answer the question.)\n\nFor example, if a document says, “The warranty lasts two years,” but the AI answers, “The warranty lasts two years and includes free repairs,” the second claim is unsupported.\n\nA validation step can flag claims that aren't backed by the source. This can involve rules, another model acting as an evaluator, or human verification. Each approach has limitations, so validation should not be treated as a guarantee of correctness.\n\n**Human review**: Require a person to verify high-impact or uncertain answers.\n\n**Evaluation and monitoring:**\n\nTest the system with questions whose answers are known, questions requiring multiple passages, and questions whose answers aren't present in the document.\n\nTrack retrieval quality, answer correctness, unsupported claims, and failures over time. This helps identify problems instead of relying on a few answers that happen to look good.\n\nNot every application needs every technique. The right combination depends on the documents, risks, and expected accuracy.\n\nIn my AI-Driven Document Intelligence Platform, I built a document Q&A pipeline using LangChain, ChromaDB, BGE embeddings, and Gemini.\n\nHere's what I implemented to make answers more grounded.\n\n**1. Retrieve information before generating an answer**\n\nI use BAAI/bge-small-en-v1.5 to convert text into embeddings. ChromaDB stores and searches these vectors to find passages relevant to the user's question.\n\nThis helps the system retrieve information from the uploaded document instead of relying only on the LLM's existing knowledge.\n\n**2. Retrieve the top 3 relevant chunks**\n\nMy retriever uses k=3, so it returns up to three relevant chunks for a question.\n\nThis keeps the context focused rather than passing the entire document to the model. However, retrieving three chunks doesn't guarantee that all the necessary evidence will be included.\n\n**3. Instruct Gemini to stay within the provided context**\n\nI use a prompt that tells the model to:\n\n```\nAnswer using only the supplied context.\n\nSay when the context doesn't contain enough information.\n\nAvoid inventing information.\n```\n\nThis encourages the model to acknowledge missing evidence instead of guessing.\n\n**4. Preserve source and page metadata**\n\nI load PDFs using PyMuPDFLoader and retain source/page metadata. This helps identify where retrieved information came from and makes answers easier to verify.\n\nSource tracking supports **traceability**, but it doesn't independently prove that an answer is correct.\n\nMy pipeline looks like this:\n\n```\nPDF → PyMuPDFLoader → Text Splitter → BGE Embeddings → ChromaDB → Top 3 Retrieved Chunks → Grounded Prompt → Gemini → Answer\n```\n\n**These measures improve grounding, but they don't guarantee that every answer is correct.** \n\nBased on this implementation, some practical next steps would be:\n\nThe important lesson is that we should test these changes against a set of questions with known answers and measure whether they actually improve the system.\n\nNo, not reliably in every situation.\n\nEven with RAG, a model can misunderstand evidence, retrieve incomplete information, or generate unsupported claims.\n\nThe practical goal is to reduce hallucinations, detect unreliable results, and allow the system to say, “I don't have enough information to answer that.”\n\nA system that acknowledges uncertainty is often more useful than one that always produces an answer.\n\nNot automatically.\n\nA retrieved document may contain outdated, inaccurate, sensitive, or even malicious instructions. For example, a document could contain text designed to manipulate the model into ignoring its original instructions. This is one form of indirect prompt injection.\n\nMy current project includes API-key handling through secure input rather than hard-coding the key into the notebook. However, that alone does not make the entire application secure.\n\nA production system may also need authentication, document-level access controls, input and output validation, protection against prompt injection, secure secret management, and appropriate logging.\n\n**Security** is a separate but closely related challenge when building reliable AI applications, which we'll explore in my next blog.\n\nBuilding a RAG system helped me understand that reducing hallucinations requires more than telling an LLM to be accurate.\n\nIt requires relevant evidence, careful retrieval, grounded prompts, validation, and a clear way to handle missing information.\n\nRAG isn't a switch that turns hallucinations off. It's one important part of building more reliable AI systems.\n\nAs AI engineers, our goal shouldn't be to make models sound confident. It should be to make their answers as trustworthy, traceable, and evidence-based as possible.\n\nThanks for reading! Keep learning, keep building. 🚀\n\nLink to LinkedIn Newsletter Edition 1- [https://www.linkedin.com/pulse/rag-from-scratch-what-actually-happens-inside-pipeline-anushka-gupta-m1rjf](https://www.linkedin.com/pulse/rag-from-scratch-what-actually-happens-inside-pipeline-anushka-gupta-m1rjf)\n\nLink to my RAG project- [https://docchat-bv9q6hkxq-anushkaashishgupta-5860s-projects.vercel.app/](https://docchat-bv9q6hkxq-anushkaashishgupta-5860s-projects.vercel.app/)", "url": "https://wpnews.pro/news/why-do-ai-models-hallucinate-and-how-can-we-reduce-it", "canonical_source": "https://dev.to/itsanushkaguptaa/why-do-ai-models-hallucinate-and-how-can-we-reduce-it-4n52", "published_at": "2026-10-28 16:04:27+00:00", "updated_at": "2026-10-28 16:11:53.790671+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "generative-ai", "ai-tools", "ai-agents"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-do-ai-models-hallucinate-and-how-can-we-reduce-it", "markdown": "https://wpnews.pro/news/why-do-ai-models-hallucinate-and-how-can-we-reduce-it.md", "text": "https://wpnews.pro/news/why-do-ai-models-hallucinate-and-how-can-we-reduce-it.txt", "jsonld": "https://wpnews.pro/news/why-do-ai-models-hallucinate-and-how-can-we-reduce-it.jsonld"}}