A RAG chatbot fails quietly. It still answers, in confident prose, even when retrieval has returned the wrong passage or the model has ignored the context. Without a way to measure that, every change to chunk size, embedding model, prompt or provider is a guess. This guide covers how to build an evaluation you can repeat, and how to describe its results without overstating them.
A wrong answer has two possible causes, and they need different fixes. Either the right passage never reached the model (a retrieval problem), or it did and the model used it badly (a generation problem). If you only score the final answer you cannot tell which, so measure both stages.
Tune the number of passages and any similarity cut-off against these numbers rather than by feel, and remember that a cut-off that is too strict produces confident refusals, while one that is too loose produces confident nonsense.
Thirty to one hundred questions is a useful starting size. Draw them from real sources: support tickets, search logs, questions the client's staff actually ask. Avoid questions you wrote by copying a sentence from a document, because they are trivially easy. For each question record the passage or passages that should be retrieved, and where it helps, the key facts a correct answer must contain. Add unanswerable questions and tricky ones: negations, numbers, questions that span two documents, and wording that does not match the document's vocabulary.
Keep the test set stable. Changing questions after a failure so the numbers recover is the quickest way to make an evaluation meaningless.
Two families of check complement each other.
Whichever you use, add periodic human review of a random sample. It is the only check that can catch what both automated methods miss.
The real value of an evaluation is comparing runs. Freeze and record the whole configuration with each run: chunk size and overlap, embedding model, number of passages, similarity cut-off, system prompt, generation model and its settings. A short fingerprint of that configuration lets you see at a glance whether two results are comparable. Then make a rule that a change to any of those settings does not ship until the suite has run and the differences are understood.
Record the time each question took and report percentiles, not just an average. For cost, use the token counts that the provider returns with each response, multiplied by the provider's current prices. A flat per-answer estimate looks tidy in a report but hides the expensive cases.
A pass rate is only meaningful with its definition beside it. A useful report states the test set size and where the questions came from, the configuration that was tested, what each check means, which cases failed and why, and the date. It also states limits. A passing score means the bot did well on those questions with those documents. It does not guarantee behavior on questions nobody wrote, and a citation check shows that an answer cites a source, not that the answer is correct.
If you are choosing a starting point for a RAG project, the RAG boilerplates category lists the options, and this comparison explains how two of them differ. Whatever you choose, build your own test set first and judge any starter by how it performs on it. A fixed test set of questions runs through the same pipeline as live traffic. For each question the run records which passages were retrieved, the answer produced and the time taken, then applies retrieval checks, answer checks and a configuration fingerprint, so two runs can be compared after any change.
Originally published at boilershelf.com.