Dify: Before Your Support Bot Answers a Customer, Ask It the Question Your Help Docs Don't Cover

A developer at Stepthrough built a support chatbot on Dify that answers member questions from a single Markdown help file, and documented the failure modes found in testing. Splitting the file on level-2 headings instead of newlines cut chunks from 28 to 8, and retrieval testing returned the correct meeting-room article at a 0.95 score, but with empty instructions the bot invented front-desk and service-animal policies absent from the docs. Adding instructions that restrict answers to the articles and mandate one exact fallback sentence stopped the guessing.

A small support team answers the same questions every day: prices, hours, cancellations. Each answer is quick. Together they fill the morning, and two people on the team can give two different answers to the same question. A chatbot that answers from your own help docs can take the repeat questions. Building one is the easy part. The real risk is a question your docs don't cover, because the bot sounds just as confident while it makes something up. I run Stepthrough, free hands-on AI and software training for white-collar workers, and I built a lesson on this with Dify https://learn.ynteractive.com/content/dify-support-chatbot-help-docs , a platform for building AI apps on top of your own knowledge. Disclosure: Dify's team and I were in touch about the lesson. No payment was involved, and I used my own free Sandbox workspace. The example is Lakeshore Coworking, a made-up coworking space with a small help center in one Markdown file. Every chunk count, score and answer below is what Dify actually produced in my test on October 31, 2026. That preview is the most important screen in the build. A chatbot doesn't read your whole file for every question. It looks up the pieces that match, called chunks , and answers from those. The default split on every new line gave me 28 tiny chunks. The heading "Plans and prices" sat alone in its own chunk, away from the prices it was supposed to head. A line like "Unused days do not roll over" no longer said which plan it belonged to. The help file marks each article with a level-2 heading, so I split there. Under Delimiter , I replaced the value with: That is two hash marks and a space. I clicked Preview Chunk again and got 8 chunks, one per article, each heading next to its own facts. In your own docs, use whatever marks the start of a new topic. Click Go to document , then Retrieval Testing . This shows which articles a question pulls up, with a score, before any AI writes an answer. Ask the way a member would, not in your help center's words: Can I get my meeting room hours back if I cancel at the last minute? Meeting rooms came back first at 0.95 , and that article holds the 2-hour rule. The next two scored 0.70 and 0.46. A strong top match with a clear gap is what you want. If the right article doesn't come back here, no prompt can save the answer. Fix the docs or the split first. Leave Instructions empty for now. Under Talk to Bot , ask something your docs don't answer: Do you allow dogs in the building? With no instructions, it did say it didn't know. Then it told the member to check with the front desk, and hinted that policies can differ for service animals. None of that is in the docs. The help center says to email support for anything it doesn't cover. That's the failure that matters in support: a friendly, confident answer that your company never wrote. Under Instructions , paste the rules. Here are the ones from the lesson: You answer member questions for Lakeshore Coworking. Answer only from the help center articles in your context. Keep answers to 1 to 3 short sentences and include the exact price, limit or deadline when the article gives one. If the articles do not cover the question, say exactly: "I don't have that in our help center. Please email help@lakeshorecoworking.example and we'll reply within one business day." Never guess, and never add advice or suggestions that are not in the articles. Three parts do the work: answer only from the articles, give exact numbers, and use one exact sentence when the articles don't cover the question. Changing the instructions resets the test chat, so click Restart . Then I asked a question the docs do answer: I have a Flex membership. What does it cost if I need more meeting room time, and can I get hours back if I cancel at the last minute? It used two articles: 2 hours included, $25 per extra hour, and hours back only with 2 hours' notice . All correct, with a citation. Then the dog question again. This time it gave the exact hand-off sentence and added nothing. Try a few more of your own before you trust it. My Sandbox workspace started with 200 credits and showed 136 after this build: three chat messages on gpt-5, one retrieval test and one small file. So free credits are for building and testing. Check Dify's current plans before real members use it. The free lesson has 5 parts. You click through real Dify screens with the made-up help center, and you can download the same Markdown file to try it in your own workspace. It's part of the free course Answer Every Customer in Under 5 Minutes https://learn.ynteractive.com/courses/answer-every-customer-in-5-minutes . Full lesson at learn.ynteractive.com/content/dify-support-chatbot-help-docs https://learn.ynteractive.com/content/dify-support-chatbot-help-docs