AI Chatbot Giving Wrong Answers? Why It Can't Find What's in Your Documents

A developer who builds document-based retrieval pipelines explains that wrong chatbot answers usually stem from the retrieval stages rather than the language model, walking through a refund-policy example where a heading split across pages and a 0.80 similarity cut-off kept the correct passage from ever reaching the model. The post outlines five common failure points — text extraction, chunking, embedding consistency, search cut-offs and result limits — and describes how to inspect each stage's output and verify a fix.

When an AI chatbot gives wrong answers about your own documents, the AI model is usually not the problem. The model only reads the few passages a search step hands it, and when that search hands it the wrong ones, a perfectly good model answers badly. You have probably seen both versions of this. The bot says "I couldn't find that information" when the answer is sitting on page 4 of your PDF. Or it answers confidently with something your documents never said. Same root cause, different symptom. I build these pipelines in my day job: pulling text out of documents, splitting it, indexing it, and running sets of test questions that catch wrong answers before users do. When an answer comes out wrong, the cause is usually in one of those earlier steps, not in the model. In this post I'll explain why it happens in plain English, the five places it usually breaks, how to check each one, and how to prove a fix worked. It applies to chatbots built on your own documents often called RAG, short for retrieval-augmented generation and to ChatGPT custom GPTs with uploaded files. A chatbot that answers from your documents does not read all of them every time someone asks something. That would be slow and expensive. Instead, each document is prepared once, and every question goes through the same short pipeline: Stages 1 to 5 decide what the model sees. The answer can only use what reaches it. The model at the end can only be as good as the passages it receives. If the right passage never reaches it, it either says it found nothing, which is honest, or fills the gap with something that sounds right, which is worse. Here is the failure in miniature, using a sample refund policy. The document, scores and results are illustrative, but the pattern is a very common one. A customer asks: "When can I request a refund?" The rules are on page 4. The chatbot replies: "I couldn't find that information in your documents." The heading "5. Refunds" sits at the very bottom of page 3, and the rules start at the top of page 4. The document was split page by page, so the heading landed in one chunk and the rules in another: Chunk A has the word the customer typed but no answer. Chunk B has the answer but never says refund. Chunk B holds the answer, but it talks about "money back" and a "return window". The word the customer typed lives in a different chunk. Search does find chunk B, but it scores 0.76 for similarity, just under a cut-off of 0.80 that was set to filter out weak matches. Only three results are kept anyway: The real answer scores 0.76 and falls just under the 0.80 cut-off, so the model never sees it. So the model receives a passage about shipping and one about cancellations, neither of which mentions refunds, and correctly says it found nothing. Every part of the system did what it was told. Two settings, chosen once and never tested, did the damage. Do not guess which stage is broken, and do not start by swapping the AI model. For each question that fails, look at what every stage produced. The root cause becomes something you can see, not something you argue about. Open the extracted text for the page that holds the answer. Scanned PDFs are pictures of pages, with no text at all unless they go through OCR text recognition first. Tables often come out as a jumble of numbers, and two-column layouts can be read straight across both columns, mixing sentences. If the text is wrong here, nothing later can fix it. Check the chunk that should contain the answer. Splitting by page or by a fixed number of characters cuts sections in half, exactly as in the refund example. Split along headings instead, keep each heading attached to the text under it, let neighbouring chunks overlap a little, and store the document name and page number with every chunk so answers can cite them. Documents and questions must be converted with the same embedding model. If the model was changed, or a PDF was replaced, every chunk has to be converted again. A half-updated index produces search results that look random. Look at how many passages are kept and whether there is a minimum score. Both are usually guessed when a chatbot is built; set them from test questions instead. When customers use different words from your documents "money back" versus "refund" , meaning-based search alone can miss. Combining it with plain keyword search, known as hybrid search, catches those cases. Check the final prompt. Sometimes the right passages are found and then cut to fit a length limit. And if the instructions never say "answer only from these passages, and say so when they do not cover the question", the model fills gaps with confident guesses. A made-up answer is the same failure as "I couldn't find that", with a different ending. When search returns passages that are near the topic but do not contain the answer, the model tries to be helpful and stitches together something plausible. Two changes stop most of it: an instruction to answer only from the passages provided, and a citation document and page on every answer. A citation makes a wrong answer easy to spot, for you and for your customers. Before changing anything, agree on a set of test questions and the right answer for each. Include three kinds: direct questions, reworded ones how customers actually talk , and questions your documents cannot answer, where the right reply is "that isn't covered". Run them before the fix and after. The fix only counts if this table improves: Before: not found, and a made-up answer. After: the right answer with its page, and an honest 'not covered'. | Question | Type | Before | After | |---|---|---|---| | When can I request a refund? | Direct | Not found | Correct, p. 4 | | Can I get my money back? | Reworded | Generic answer | Correct, p. 4 | | How long do I have to return something? | Reworded | Not found | Correct, p. 4 | | Can I cancel before it ships? | Direct | Correct | Correct, p. 2 | | When will my order arrive? | Direct | Correct | Correct, p. 3 | | What if my item arrives damaged? | Reworded | Generic answer | Correct, p. 5 | | Do you ship to Canada? | Not in the documents | Made-up answer | Says it's not covered | | Who is your CEO? | Not in the documents | Says it's not covered | Says it's not covered | In this sample, 3 of 8 answers were right before the fix and 8 of 8 after, each with the page cited. Look at the Canada question: before the fix the bot invented an answer, and after it says the documents do not cover it. For a business, that is the most important row in the table. Keep the questions. Every time you add documents or change a setting, run them again. The same mechanics apply. A custom GPT searches its knowledge files for relevant passages instead of reading them end to end, so the same things go wrong: If a custom GPT still misses answers after that, it has reached the limits of a tool you cannot tune. That is usually the point where a small chatbot of your own, where you control the chunking, the search settings and the prompt, starts paying for itself. Wrong answers from a document chatbot feel like an AI problem, so the instinct is to try a bigger model. Usually the fix is earlier and cheaper: text that was never extracted, a heading cut off from its section, or a search filter set by guesswork. Trace one failing question through every stage, fix the stage that breaks, and prove it with the same questions before and after. It is the same rule that got two of my fixes merged into a 10 million user app: trace the data, not the symptom https://tusharbhendarkar.com/blog/how-i-debugged-expensify-bugs-and-got-paid . More guides like this one are in AI chatbots and AI features https://tusharbhendarkar.com/blog/topics/ai-chatbots . If you would rather hand it over, I trace chatbots exactly this way. Send me the PDFs that cause trouble and a few questions that fail today, and you get the same before-and-after table for your own documents, with a short note on the root cause. Why does my chatbot say it can't find information that is in my PDF? Because the passage with the answer never reached the AI model. Usually the document was split so the answer sits in a chunk without the words people search for, or a search setting a minimum score, or too few results kept dropped it. Check the extracted text and the search results for that question before changing anything else. Why does my AI chatbot make up answers? When search returns passages that are close to the topic but do not contain the answer, the model fills the gap with something plausible. Tell it to answer only from the passages it is given and to say when they do not cover the question, and require a document and page citation on every answer. Why is my custom GPT not using its knowledge files? Custom GPTs search their knowledge files instead of reading them end to end, so scanned PDFs, very long files and vague headings all make the right passage hard to find. Use text-based PDFs, split long files into smaller ones with clear headings, and tell the GPT in its instructions to search the knowledge files before answering. Will a better AI model fix wrong answers? Rarely. A stronger model still only sees the passages it is given. If the problem is extraction, chunking or search settings, a new model gives the same wrong answers in nicer words. Can this be fixed without rebuilding the chatbot? Usually, yes. Most fixes are a change to how documents are split, a re-index, adjusted search settings and a clearer prompt. A fixed set of test questions tells you when it is done. This post first appeared on tusharbhendarkar.com https://tusharbhendarkar.com/blog/ai-chatbot-giving-wrong-answers . I build and fix AI chatbots and RAG systems for small businesses. If something here is broken for you, email me at tusharbhendarkar44@gmail.com mailto:tusharbhendarkar44@gmail.com .