Retrieval-augmented generation (RAG) is a pattern for answering questions with information stored outside a language model. The application searches a knowledge source, selects relevant passages, and gives them to a model as evidence for an answer.
The important part is that RAG is a system, not just a prompt. A fluent answer can still be wrong if retrieval found the wrong material, the source is stale, or the model claims more than the evidence supports.
A useful mental model is:
Question → retrieve passages → assemble evidence → generate answer → cite sources
There is no universally correct chunk size. Small chunks make focused facts easier to find but can lose context. Large chunks preserve context but may add noise and use more tokens. Start with a simple, measured baseline, then compare alternatives using representative questions.
For every retrieved passage, retain enough metadata to trace it back to its source. A citation should identify a document and, where practical, a page, heading, or stable chunk reference. If content changes, re-index it and track the source version.
Create a small evaluation set before tuning. For each question, record the expected source documents and, if possible, the facts a correct answer should contain. Include questions that have no answer in the knowledge base.
Useful retrieval measures include:
Then evaluate the answer separately: Is it correct? Is each claim supported by a retrieved passage? Do citations point to the right evidence? Does the system say it does not know when the documents do not answer the question?
A practical failure review asks:
This separation helps you fix the right layer instead of changing the prompt to compensate for an indexing problem.
Start with a no-model retrieval baseline. Verify that the correct evidence is found and show the passages to a reviewer. Add generation only after retrieval is measurable. Make failures visible: low confidence, missing evidence, invalid input, or model errors should lead to a safe fallback or human review.
For a small local implementation, the companion project uses TF-IDF retrieval and can optionally call a local Ollama model. It is a teaching example, not a production service; it has no authentication or access-control layer.
Disclosure: This article was prepared with AI assistance for Stack & Craft.
What has been the hardest RAG failure for you to diagnose: finding the right passage, evaluating answer quality, or keeping citations trustworthy?