A RAG chatbot on your company knowledge base: what it is and when it pays off

A developer's guide to retrieval-augmented generation (RAG) chatbots explains that the approach grounds language model answers in a company's own documents and cites sources, and argues it only pays off where large volumes of text meet repeated questions. Citing a sample of 903 US AI Engineer postings from Glassdoor in April 2026, the writeup notes RAG is required in 13.6% of ads, ahead of agents at 10.6% and prompt engineering at 8.9%, while vector databases appear in just 4.5%. The piece identifies four common failure modes — stale documents, out-of-scope answers, document leakage, and hallucinated citations — and recommends a two-week pilot with an evaluation set of 50 questions.

The request I hear most often this year sounds the same every time: "We want a bot that answers from our documents and does not make things up." The technical name is RAG, retrieval-augmented generation. Below, without jargon: what it is, who it pays off for, what your company needs to bring, and where these projects break. A language model answers from memory. It read the internet up to some date and knows nothing about your returns policy, September price list or warehouse manual. Ask it directly and it will either admit it does not know or confidently invent something. RAG changes the order. First the system searches your documents for the passages that relate to the question. Then it hands them to the model with the question and asks it to answer only from them, naming the source document. The model still writes the text, but the source of facts becomes your files, not its memory. To the user it looks like a chat. To the business it is a way to give people hundreds of pages of documentation without making them read those pages. The citation in every answer is not decoration. It is the main mechanism of trust. In a sample of 903 US AI Engineer postings Glassdoor, April 2026 , RAG is required in 13.6% of ads. That is more often than agents 10.6% and prompt engineering 8.9% . Vector databases, usually called the heart of RAG, appear in only 4.5%. The gap between those numbers shows where the work actually is. A database for semantic search can be stood up in a day, and there are plenty of ready options. The hard part comes before and after. In my review of the AI job market in 2026 https://www.ilinmaks.com/blog/en/ai-jobs-market-2026 I quoted the KORE1 survey: employers name building evaluation systems as the number one skill, not choosing a database. For RAG that is doubly true. Not every company needs a bot on its documents. It pays off where there is a lot of text and many repeated questions about it. The common trait: the answer already exists in the documents, but finding it costs more than asking a person. Twenty pages of documentation and five questions a day do not need a bot. One FAQ page will do. I raise this in the first meeting, because it matters more than the choice of model. The third item gets the most resistance and delivers the most value. A couple of days with support logs or email threads is enough to assemble it. Four typical failures, each with a defense. Stale documents. The bot cites last year's price list. Defense: an update owner, and the document date shown in the answer so the user sees how fresh the source is. Answering outside scope. A customer asks about the legal consequences of terminating a contract, and a bot built on delivery documents tries to answer. Defense: scoped retrieval and a hard rule: if the documents do not contain the answer, say so and hand over to an operator. Leaking documents. The bot answers a customer with a passage from an internal manual listing cost prices. Defense: permissions enforced at the retrieval level, not in the prompt. The model must never see a document the user may not see. Hallucinated citations. The model cites a section that does not exist. Defense: citations are generated by the system from the retrieved passages, not written by the model, plus an evaluation set that tracks such cases. Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027. From what I have seen, document projects die for the same reasons, and almost always nobody built an evaluation set before starting. A pilot on one set of documents takes two weeks. One source say, the support knowledge base , one scenario answers for customers or staff , an evaluation set of 50 questions, handoff of hard cases to a person. After two weeks you have a working version to show the team and measure. Other sources and permissions for several roles come in the second stage. I would not try to cover everything at once: every new source brings its own document quality problems. I build this kind of bot under the "AI inside your existing product" https://www.ilinmaks.com/ services service, from 1,200 EUR. That covers RAG, LLM features and integrations into what you already run: a website, a CRM, a support portal, an internal tool. If it is unclear whether the company has suitable documents and a process, start with an AI readiness audit from 450 EUR. If staff or customers keep asking the same questions about your knowledge base, send me a short brief: what the documents are, who asks, how many questions per week. I will tell you whether it is worth doing and how long it would take. Originally published at ilinmaks.com https://www.ilinmaks.com/blog/en/rag-chatbot-company-knowledge-base .