Need advice on building a chatbot

A non-technical Hugging Face forum user building a property-database RAG chatbot on Claude 4.6 via AWS Bedrock reported wrong cited numbers, hallucinations and thin answers, and was advised by forum respondent nootxlm to add a post-generation check that every number and cited fact appears in the retrieved passages, regenerating or declining to answer when it does not. The respondent also recommended freezing 20–30 real questions with hand-verified answers and scoring every change against that set, since a change that does not reduce wrong citations on it did not help.

ashalwi https://discuss.huggingface.co/u/ashalwi 1 Hello smart people of HF I’ve been wrestling to build a chatbot for a company project for months now. I’m not technical, and I am still learning. The use case for the chatbot/assistant is to help users seek answers from a property related database. I have built a RAG system from PDFs, Excel sheets/workbooks and website info. The results are often rubbish: Wrong cited numbers, model hallucinates, thin answers and many more failures. Can anyone advice on the best approach to build a chatbot, please? Is there a step by step process to this? I am using Claude 4.6 as the chatbot model via AWS bedrock nootxlm https://discuss.huggingface.co/u/nootxlm 2 Your three failures point to the same missing step: nothing checks the answer against its sources after the model writes it. Add that check first — every number and cited fact in a reply has to appear in the retrieved passages behind it, and when a number is not there, the system regenerates or says it does not know instead of guessing. Then freeze 20–30 real questions with answers you have verified by hand from the database, and score every change against them: if wrong citations do not fall on that set, the change did not help, whatever a single demo feels like.