Building and Evaluating a RAG System: A Practical, Evidence-First Guide

A developer published a practical, evidence-first guide to building and evaluating retrieval-augmented generation (RAG) systems, arguing that RAG is a system rather than a prompt and that fluent answers can still be wrong when retrieval finds stale or irrelevant material. The guide recommends establishing a no-model retrieval baseline, building a small evaluation set with expected sources and unanswerable questions, and scoring retrieval and answer quality separately so failures are fixed at the right layer. A companion teaching project uses TF-IDF retrieval with optional local Ollama generation and lacks authentication or access control.

Retrieval-augmented generation RAG is a pattern for answering questions with information stored outside a language model. The application searches a knowledge source, selects relevant passages, and gives them to a model as evidence for an answer. The important part is that RAG is a system, not just a prompt. A fluent answer can still be wrong if retrieval found the wrong material, the source is stale, or the model claims more than the evidence supports. A useful mental model is: Question → retrieve passages → assemble evidence → generate answer → cite sources There is no universally correct chunk size. Small chunks make focused facts easier to find but can lose context. Large chunks preserve context but may add noise and use more tokens. Start with a simple, measured baseline, then compare alternatives using representative questions. For every retrieved passage, retain enough metadata to trace it back to its source. A citation should identify a document and, where practical, a page, heading, or stable chunk reference. If content changes, re-index it and track the source version. Create a small evaluation set before tuning. For each question, record the expected source documents and, if possible, the facts a correct answer should contain. Include questions that have no answer in the knowledge base. Useful retrieval measures include: Then evaluate the answer separately: Is it correct? Is each claim supported by a retrieved passage? Do citations point to the right evidence? Does the system say it does not know when the documents do not answer the question? A practical failure review asks: This separation helps you fix the right layer instead of changing the prompt to compensate for an indexing problem. Start with a no-model retrieval baseline. Verify that the correct evidence is found and show the passages to a reviewer. Add generation only after retrieval is measurable. Make failures visible: low confidence, missing evidence, invalid input, or model errors should lead to a safe fallback or human review. For a small local implementation, the companion project uses TF-IDF retrieval and can optionally call a local Ollama model. It is a teaching example, not a production service; it has no authentication or access-control layer. Disclosure: This article was prepared with AI assistance for Stack & Craft. What has been the hardest RAG failure for you to diagnose: finding the right passage, evaluating answer quality, or keeping citations trustworthy?