{"slug": "how-to-evaluate-a-documentation-rag-pipeline-with-real-questions", "title": "How to evaluate a documentation RAG pipeline with real questions", "summary": "Spidra published a Python harness for evaluating a documentation RAG pipeline that grades citation coverage and answer grounding across answerable, ambiguous and unanswerable questions, then changes the pipeline one step at a time to show which layer failed: ingestion, retrieval, answering or the decision to answer at all. The worked example runs a BM25 retriever and a baseline answerer that quotes the documentation over Spidra's own documentation, capped at 30 pages selected by include and exclude path rules, and Spidra states the numbers belong to that one setup and are not a benchmark. The harness is built from 11 modules including config.py, ingest.py, questions.py, grade.py, retrieve.py, answer.py and evaluate.py, and Spidra notes it is grading its own docs, so the results should be treated as an example to re-run on other corpora.", "body_md": "A documentation assistant can sound right and be wrong, and it can be right for the wrong reason. The model may know the answer from its training, ignore the passages you gave it, or never have seen the page that holds the fact. To evaluate a documentation RAG pipeline, you need questions it can fail, and a way to tell which layer failed: ingestion, retrieval, the answer, or the decision to answer at all.\n\nThis article build builds that harness in Python. It ingests a defined documentation set through Spidra, runs retrieval and answering, and grades citation coverage and answer grounding against a reference set of answerable, ambiguous and unanswerable questions. Then it changes the pipeline one step at a time and shows what each change did, layer by layer.\n\nEvaluation results depend on the retriever and the model you put downstream, so every number here belongs to one setup: a BM25 retriever and a baseline answerer that quotes the documentation. They are not Spidra's numbers and they are not a benchmark. The harness takes any retriever and any answerer, and the point is the method.\n\nOur corpus is Spidra's own documentation, for three reasons: it's recent, it's ours to release, and a model is unlikely to have memorised it, so a correct answer is more likely to come from the passages than from memory. It also means we're grading our own docs, so treat the results as a worked example and run the harness on yours.\n\nThe code below comes from small modules: `config.py`, `ingest.py`, `corpus.py`, `questions.py`, `grade.py`, `chunking.py`, `retrieve.py`, `answer.py`, `llm_answer.py`, `evaluate.py` and `systems.py`. Put it in one file, in the order it appears, with these imports at the top, and every example runs. The calls to Spidra need your [API key](https://docs.spidra.io/authentication).\n\n``` python\nimport re\nimport sqlite3\nimport time\nimport unicodedata\nfrom collections import Counter\nfrom dataclasses import dataclass, field\nfrom urllib.parse import urlsplit\n\nfrom spidra import Spidra\n```\n\n## Define the corpus, and what each layer owes you\n\nFix the corpus before you write a question. A rule is better than a list you typed once, because it decides what counts as \"the documentation\" and can be re-run:\n\n```\nDOCS_URL = \"https://docs.spidra.io/\"\nINCLUDE = (\n    \"/quickstart\", \"/scrape\", \"/batch-scrape\", \"/crawl\", \"/map\", \"/search\", \"/parse\",\n    \"/options/\", \"/billing/\", \"/sdks/\", \"/templates\", \"/presets\", \"/monitors\", \"/guides/\",\n)\nEXCLUDE = (\"/sites/\", \"/api-reference/\")\nMAX_PAGES = 30\npython\ndef under(path, prefixes):\n    return any(path == p.rstrip(\"/\") or path.startswith(p.rstrip(\"/\") + \"/\") for p in prefixes)\npython\ndef select_pages(links, base=DOCS_URL, include=INCLUDE, exclude=EXCLUDE, limit=MAX_PAGES):\n    \"\"\"Same host, an included path, no excluded path, no duplicates, in Map's order, capped.\"\"\"\n    host, seen, chosen = urlsplit(base).netloc, set(), []\n    for link in links:\n        parts = urlsplit(link)\n        path, key = parts.path.rstrip(\"/\") or \"/\", (parts.netloc, parts.path.rstrip(\"/\"))\n        if (\n            parts.netloc == host\n            and key not in seen\n            and under(path, include)\n            and not under(path, exclude)\n        ):\n            seen.add(key)\n            chosen.append(f\"{parts.scheme}://{parts.netloc}{parts.path}\")\n    return chosen[:limit]\n```\n\nA failure can come from four places, and the fix for each is different. The table is the whole idea of the harness:\n\n| Layer | A failure looks like | How the harness sees it | \n|---|---|---|\n| Ingestion | The fact never reached the corpus | A required fact is missing from the gold page's text | \n| Retrieval | The fact is in the corpus but wasn't retrieved | No retrieved passage contains it | \n| Answering | It was retrieved, but the answer is wrong or unsupported | The answer lacks the fact, or its numbers aren't in the cited text | \n| Policy | It answered the unanswerable, or refused the answerable | The response doesn't match the behaviour the question needs | \n\nEstablished evaluation tools split their diagnostics the same way. [RAGChecker](https://github.com/amazon-science/RAGChecker) reports retriever and generator metrics separately, and [TruLens](https://github.com/truera/trulens) scores a \"**RAG triad**\" of context relevance, groundedness and answer relevance. Both use a model to judge claims. This harness doesn't need one, so every number can be checked by hand, and you can add a judge where paraphrase makes string checks too strict.\n\n## Ingest the same pages two ways\n\nA crawl returns each page's Markdown including the site's navigation and footer. A batch scrape can strip them with the Extract content only option. We fetch every page both ways, on the same URLs, so the only difference between the two corpora is that boilerplate. A raw batch is 1 credit per page, with no AI involved:\n\n``` python\ndef wait_for(client, batch_id, timeout=900):\n    deadline = time.monotonic() + timeout\n    while time.monotonic() < deadline:\n        batch = client.get_batch_scrape(batch_id)\n        if batch.completed_count + batch.failed_count >= batch.total_urls:\n            return batch\n        time.sleep(5)\n    raise TimeoutError(f\"batch {batch_id} did not finish in {timeout} s\")\nphp\ndef fetch_markdown(client, urls, content_only):\n    \"\"\"-> {url: (status, markdown or None, credits)}. A raw batch is 1 credit per page, no AI.\"\"\"\n    queued = client.start_batch_scrape(\n        list(urls), output=\"markdown\", extract_content_only=content_only\n    )\n    items = wait_for(client, queued.batch_id).items\n    return {\n        i.url: (i.status, i.result if isinstance(i.result, str) else None, i.credits_used or 0)\n        for i in items\n    }\n```\n\nEach page is stored twice, because the question set has to be checked against both:\n\n```\n@dataclass\nclass Page:\n    url: str\n    title: str\n    clean: str  # markdown with the navigation and footer stripped\n    raw: str  # the whole page as markdown, navigation included\n\ndef title_of(markdown, fallback):\n    match = re.search(r\"^#\\s+(.+)$\", markdown, re.M)\n    return match.group(1).strip() if match else fallback\n```\n\nRepeated lines give the boilerplate away. A line that appears on 80% of the pages isn't documentation:\n\n``` python\ndef repeated_lines(pages, share=0.8):\n    \"\"\"Lines on at least `share` of the pages: in practice, the navigation and footer.\"\"\"\n    counts = {}\n    for page in pages:\n        for line in {ln.strip() for ln in page.splitlines() if ln.strip()}:\n            counts[line] = counts.get(line, 0) + 1\n    need = max(1, round(share * len(pages)))\n    return sorted(\n        (line for line, n in counts.items() if n >= need), key=lambda line: (-counts[line], line)\n    )\n```\n\nThe examples from here on use a made-up site, Acme, on a reserved `.test` domain. Its pages carry a navigation line and a footer in the raw version:\n\n```\nNAV = \"Quickstart Credits Batch Errors Pricing Sign in\"\nFOOTER = \"(c) 2026 Acme. Terms. Privacy.\"\n\ndef acme(path, markdown):\n    url = f\"https://docs.acme.test/{path}\"\n    return Page(url, title_of(markdown, path), markdown, f\"{NAV}\\n\\n{markdown}\\n\\n{FOOTER}\")\nBATCH = (\n    \"# Batch fetch\\n\\nSend up to 50 URLs in one request. Each URL gets its own result.\\n\\n\"\n    \"## Limits\\nYou can submit 20 batches per minute.\\n\\n\"\n    \"## Failures\\nA batch is completed even when items failed. Failed items are not charged.\\n\"\n)\nCREDITS = (\n    \"# How credits work\\n\\nEach fetch costs 1 credit. A fetch with a schema costs 2 credits.\\n\\n\"\n    \"## Plans\\nThe Free plan includes 300 credits.\\n\"\n)\ncorpus = {p.url: p for p in (acme(\"batch\", BATCH), acme(\"credits\", CREDITS))}\nfor text in repeated_lines([p.raw for p in corpus.values()]):\n    print(text)\n```\n\nOutput:\n\n```\n(c) 2026 Acme. Terms. Privacy.\nQuickstart Credits Batch Errors Pricing Sign in\n```\n\n## Write questions that can fail\n\nA question set that can't fail tells you nothing. Ours has three classes, and each one needs a different behaviour:\n\n- **Answerable.** The documentation holds the answer. A good response states it, cites the page, and says nothing the page doesn't support.\n- **Ambiguous.** The question has more than one reading (\"How much does it cost?\"). A good response asks which one, or covers each reading with a citation.\n- **Unanswerable.** The documentation doesn't say. A good response says so, and invents nothing.\n\nEach question records what to look for, so grading is a lookup and not a judgement:\n\n```\nEXPECTED = {\"answerable\": \"answer\", \"ambiguous\": \"clarify_or_cover\", \"unanswerable\": \"abstain\"}\n@dataclass\nclass Question:\n    id: str\n    cls: str  # answerable, ambiguous or unanswerable\n    split: str  # dev or test\n    question: str\n    gold_sources: list = field(default_factory=list)  # pages that hold the answer\n    required: list = field(default_factory=list)  # groups of alternatives, each must appear\n    forbidden: list = field(default_factory=list)  # claims that would be wrong\n    absence_terms: list = field(\n        default_factory=list\n    )  # unanswerable: terms the corpus must not contain\n    source: str = \"written\"  # written by us, or a real user question\n```\n\n`required` is a list of groups. Every group must appear in a good response, and each group lists alternative wordings. `forbidden` lists claims that would be wrong. `absence_terms` is for unanswerable questions: words the corpus must not contain. Matching ignores case, spacing and thousands separators, so `1,000` and `1000` are the same fact:\n\n``` python\ndef norm(text):\n    \"\"\"Case, spacing and thousands separators should not decide whether a fact is present.\"\"\"\n    text = unicodedata.normalize(\"NFKC\", str(text)).casefold().replace(\"\\u2019\", \"'\")\n    return \" \".join(re.sub(r\"(?<=\\d),(?=\\d{3})\", \"\", text).split())\n\ndef facts_present(text, required):\n    \"\"\"True when every group has at least one of its alternatives in the text.\"\"\"\n    return all(any(norm(alt) in norm(text) for alt in group) for group in required)\n```\n\nThe questions are checked against the corpus before anyone trusts them. An answerable question whose required fact isn't on its gold page is unfair, and an unanswerable one whose key term appears anywhere, even in the navigation, isn't unanswerable:\n\n``` python\ndef check_unanswerable(q, everything):\n    found = [f\"{q.id}: the corpus contains {t!r}\" for t in q.absence_terms if norm(t) in everything]\n    return found + ([] if q.absence_terms else [f\"{q.id}: no absence_terms\"])\n\ndef check_answerable(q, corpus):\n    problems = [\n        f\"{q.id}: gold page not in the corpus: {u}\" for u in q.gold_sources if u not in corpus\n    ]\n    if not (q.gold_sources and q.required):\n        problems.append(f\"{q.id}: needs gold sources and required facts\")\n    gold_text = \" \".join(corpus[u].clean for u in q.gold_sources if u in corpus)\n    if q.required and not facts_present(gold_text, q.required):\n        problems.append(f\"{q.id}: a required fact is not on the gold pages\")\n    return problems\npython\ndef check_question(q, corpus, everything):\n    if q.cls not in EXPECTED or q.split not in (\"dev\", \"test\"):\n        return [f\"{q.id}: bad class or split\"]\n    return (\n        check_unanswerable(q, everything)\n        if q.cls == \"unanswerable\"\n        else check_answerable(q, corpus)\n    )\npython\ndef validate_questions(questions, corpus):\n    \"\"\"Problems that make a question unfair or unverifiable. Empty means the set is sound.\"\"\"\n    problems = [f\"duplicate id {i}\" for i, n in Counter(q.id for q in questions).items() if n > 1]\n    everything = norm(\" \".join(p.clean + \" \" + p.raw for p in corpus.values()))\n    for q in questions:\n        problems += check_question(q, corpus, everything)\n    return problems\n```\n\nHere are four questions for Acme, one per kind of behaviour:\n\n``` python\ndef ask(id, cls, text, **fields):\n    return Question(id, cls, \"test\", text, **fields)\n\nBATCH_URL, CREDITS_URL = \"https://docs.acme.test/batch\", \"https://docs.acme.test/credits\"\nQUESTIONS = [\n    ask(\"a1\", \"answerable\", \"How many URLs can I send in one batch?\",\n        gold_sources=[BATCH_URL], required=[[\"50 urls\"]]),\n    ask(\"a2\", \"answerable\", \"Do failed items cost credits?\",\n        gold_sources=[BATCH_URL], required=[[\"not charged\"]], forbidden=[\"1 credit\"]),\n    ask(\"m1\", \"ambiguous\", \"How much does it cost?\",\n        gold_sources=[CREDITS_URL], required=[[\"1 credit\"], [\"2 credits\"]]),\n    ask(\"u1\", \"unanswerable\", \"Can I export results to Excel?\", absence_terms=[\"excel\"]),\n]\nprint(\"problems:\", validate_questions(QUESTIONS, corpus))\nproblems: []\n```\n\nAnd three that are broken on purpose, to show what the check catches:\n\n```\nbroken = [\n    ask(\"b1\", \"answerable\", \"What is the rate limit?\",\n        gold_sources=[BATCH_URL], required=[[\"100 per second\"]]),\n    ask(\"b2\", \"unanswerable\", \"Is there a Pricing page?\", absence_terms=[\"pricing\"]),\n    ask(\"b3\", \"answerable\", \"Where is the changelog?\",\n        gold_sources=[\"https://docs.acme.test/changelog\"], required=[[\"v2\"]]),\n]\nfor problem in validate_questions(broken, corpus):\n    print(problem)\n```\n\nOutput:\n\n```\nb1: a required fact is not on the gold pages\nb2: the corpus contains 'pricing'\nb3: gold page not in the corpus: https://docs.acme.test/changelog\nb3: a required fact is not on the gold pages\n```\n\nThe second one is the interesting case: \"pricing\" appears only in the navigation menu, and the check still finds it, because it searches the raw pages too.\n\n## Cut the pages into chunks\n\nRetrieval works on chunks, and how you cut decides what can be found. The simple way is fixed windows of text. The other starts a chunk at each heading and begins every chunk with the page title and its heading path, so the words in the headings are searchable:\n\n```\nCHUNK_CHARS = 1000\nSECTION_CHARS = 1500\nFENCE = \"`\" * 3  # the marker that opens and closes a code block in Markdown\n@dataclass\nclass Chunk:\n    id: str\n    url: str\n    title: str\n    heading: str  # the section's path without the page title, empty for the opening section\n    text: str\n    label: str = \"\"  # title and path, the first line of the text. For retrieval, never quoted.\npython\ndef fixed_chunks(page, text, size=CHUNK_CHARS):\n    \"\"\"Windows of text that ignore the page's structure.\"\"\"\n    return [\n        Chunk(f\"{page.url}#{n}\", page.url, page.title, \"\", text[start : start + size])\n        for n, start in enumerate(range(0, len(text), size))\n    ]\nHEADING = re.compile(r\"(#{1,4})\\s+(.+)\")\npython\ndef sections(markdown):\n    \"\"\"[(heading path, body)]. Headings count only outside code fences. Empty sections drop.\"\"\"\n    found, path, body, fenced = [], [], [], False\n    for line in markdown.splitlines():\n        fenced ^= line.startswith(FENCE)\n        heading = None if fenced else HEADING.fullmatch(line)\n        if heading:\n            found.append((path, \"\\n\".join(body).strip()))\n            path, body = path[: len(heading.group(1)) - 1] + [heading.group(2).strip()], []\n        else:\n            body.append(line)\n    found.append((path, \"\\n\".join(body).strip()))\n    return [(\" > \".join(p), b) for p, b in found if b]\npython\ndef paragraphs(body):\n    \"\"\"Blocks separated by blank lines. A code fence is never split.\"\"\"\n    blocks, current, fenced = [], [], False\n    for line in body.splitlines():\n        fenced ^= line.startswith(FENCE)\n        if not fenced and not line.strip():\n            if current:\n                blocks.append(\"\\n\".join(current))\n                current = []\n        else:\n            current.append(line)\n    return blocks + ([\"\\n\".join(current)] if current else [])\npython\ndef split_long(body, max_chars):\n    \"\"\"Pack whole paragraphs into pieces of about max_chars. One long paragraph stays whole.\"\"\"\n    pieces, piece = [], \"\"\n    for block in paragraphs(body):\n        if piece and len(piece) + len(block) > max_chars:\n            pieces.append(piece)\n            piece = \"\"\n        piece = f\"{piece}\\n\\n{block}\".strip()\n    return pieces + ([piece] if piece else [])\n\ndef split_heading(title, path):\n    \"\"\"The section path without the page title, and the label every chunk starts with.\"\"\"\n    heading = path[len(title) :].removeprefix(\" > \") if path.startswith(title) else path\n    return heading, f\"{title} > {heading}\" if heading else title\npython\ndef heading_chunks(page, text, max_chars=SECTION_CHARS):\n    \"\"\"One chunk per section, each starting with the page title and its heading path.\"\"\"\n    chunks = []\n    for path, body in sections(text):\n        heading, label = split_heading(page.title, path)\n        for piece in split_long(body, max_chars):\n            chunk_id = f\"{page.url}#{len(chunks)}\"\n            chunks.append(\n                Chunk(chunk_id, page.url, page.title, heading, f\"{label}\\n{piece}\", label)\n            )\n    return chunks\n```\n\nHeadings are only recognised outside code fences, because a `# comment` in a code block isn't a section:\n\n```\npage = corpus[BATCH_URL]\nfor chunk in heading_chunks(page, page.clean):\n    print(repr(chunk.label), \"|\", chunk.text.splitlines()[1][:46])\nprint(len(fixed_chunks(page, page.clean, size=100)), \"fixed chunks of 100 characters\")\n```\n\nOutput:\n\n```\n'Batch fetch' | Send up to 50 URLs in one request. Each URL ge\n'Batch fetch > Limits' | You can submit 20 batches per minute.\n'Batch fetch > Failures' | A batch is completed even when items failed. F\n3 fixed chunks of 100 characters\n```\n\n## Retrieve with BM25\n\nWe use BM25 through SQLite's full-text search, so there's nothing to install and every score can be reproduced. It's a lexical retriever. If you use embeddings, the harness only needs an object with the same `search` method:\n\n```\nSTOP = set(\n    \"a an and are as at be by can do does for from how i in is it its of on or that the this to \"\n    \"what when where which who why will with you your\".split()\n)\n\ndef terms(text):\n    \"\"\"Lower-case words without stop words. An identifier such as batch_scrape stays whole.\"\"\"\n    return [w for w in re.findall(r\"[a-z0-9_]+\", text.casefold()) if w not in STOP]\n\ndef match_query(question):\n    \"\"\"An OR query of quoted word pieces. Quoting keeps stray punctuation out of FTS5 syntax.\"\"\"\n    pieces = [p for term in terms(question) for p in re.findall(r\"[^\\W_]+\", term)]\n    return \" OR \".join(f'\"{p}\"' for p in dict.fromkeys(pieces))\npython\ndef open_db(chunks):\n    \"\"\"An in-memory FTS5 table with one row per chunk, ranked by BM25.\"\"\"\n    db = sqlite3.connect(\":memory:\")\n    db.execute(\n        \"CREATE VIRTUAL TABLE c USING fts5(text, id UNINDEXED, \"\n        \"tokenize='porter unicode61 remove_diacritics 2')\"\n    )\n    db.executemany(\"INSERT INTO c VALUES (?, ?)\", [(c.text, c.id) for c in chunks])\n    return db\npython\ndef cap_per_page(hits, per_page):\n    \"\"\"Best first, with at most per_page chunks from any one page.\"\"\"\n    seen, kept = Counter(), []\n    for chunk, score in hits:\n        if not per_page or seen[chunk.url] < per_page:\n            seen[chunk.url] += 1\n            kept.append((chunk, score))\n    return kept\npython\nclass Index:\n    def __init__(self, chunks):\n        self.chunks, self.db = {c.id: c for c in chunks}, open_db(chunks)\n\n    def search(self, question, k=5, per_page=None):\n        \"\"\"-> [(chunk, score)], best first. per_page caps how many chunks one page may supply.\"\"\"\n        query = match_query(question)\n        sql = \"SELECT id, -bm25(c) FROM c WHERE c MATCH ? ORDER BY bm25(c) LIMIT ?\"\n        rows = self.db.execute(sql, (query, k * 20)) if query else []\n        return cap_per_page([(self.chunks[i], score) for i, score in rows], per_page)[:k]\n```\n\n`per_page` caps how many chunks one page can supply, so a single long page can't fill every slot:\n\n```\nchunks = [c for p in corpus.values() for c in heading_chunks(p, p.clean)]\nindex = Index(chunks)\nfor chunk, score in index.search(\"How many URLs can I send in one batch?\", k=3):\n    print(f\"{score:6.3f}\", chunk.label)\n```\n\nOutput:\n\n```\n3.488 Batch fetch\n0.000 Batch fetch > Limits\n0.000 Batch fetch > Failures\n```\n\n## Answer with something you can inspect\n\nThe baseline answerer quotes. It scores every sentence in the retrieved chunks by how many of the question's words it contains, and returns the best two with a citation number. There's no model to blame, and you can read every answer it gives:\n\n```\nABSTAIN_TEXT = \"I can't find this in the documentation.\"\nSPLIT = re.compile(r\"(?<=[.!?])\\s+\")\n\n@dataclass\nclass Answer:\n    text: str\n    abstained: bool = False\n\ndef stem(word):\n    \"\"\"Strip a plural s, so credits matches credit. Short words and ss, us, is endings stay.\"\"\"\n    plural = len(word) > 3 and word.endswith(\"s\") and not word.endswith((\"ss\", \"us\", \"is\"))\n    return word[:-1] if plural else word\npython\ndef plain(text):\n    \"\"\"Markdown as words: list marks, links, backticks and table bars removed.\"\"\"\n    text = re.sub(r\"\\[([^\\]]*)\\]\\([^)]*\\)\", r\"\\1\", text).replace(\"`\", \"\").replace(\"|\", \" \")\n    return \" \".join(re.sub(r\"^[#>*\\-\\s]+\", \"\", text).split())\npython\ndef candidate_sentences(chunk):\n    \"\"\"Sentences worth quoting: not code, headings or the label, and three words or more.\"\"\"\n    lines, fenced = chunk.text.splitlines(), False\n    if chunk.label and lines and lines[0] == chunk.label:\n        lines = lines[1:]\n    found = []\n    for line in lines:\n        fenced ^= line.startswith(FENCE)\n        if not fenced and not line.startswith((FENCE, \"#\")):\n            found += [plain(s) for s in SPLIT.split(line)]\n    return [s for s in found if len(s.split()) >= 3]\npython\ndef overlap(sentence, wanted):\n    \"\"\"The share of the wanted words (stemmed) that the sentence contains.\"\"\"\n    have = {stem(t) for t in terms(sentence)}\n    return len({stem(w) for w in wanted} & have) / len(wanted) if wanted else 0.0\n\ndef best_sentences(question, hits, take=2):\n    \"\"\"[(overlap, chunk number, sentence)] for the best-matching sentences, best first.\"\"\"\n    wanted = set(terms(question))\n    scored = [\n        (overlap(s, wanted), -n, -i, s)\n        for n, (chunk, _) in enumerate(hits, 1)\n        for i, s in enumerate(candidate_sentences(chunk))\n    ]\n    return [(o, -n, s) for o, n, _, s in sorted(scored, reverse=True)[:take]]\nclass ExtractiveAnswerer:\n    \"\"\"Quotes the best-matching sentences, each with its chunk number. No model involved.\"\"\"\n\n    def __init__(self, min_score=0.0, min_overlap=0.0, take=2):\n        self.min_score, self.min_overlap, self.take = min_score, min_overlap, take\n\n    def __call__(self, question, hits):\n        best = best_sentences(question, hits, self.take)\n        if not best or best[0][0] < self.min_overlap or hits[0][1] < self.min_score:\n            return Answer(ABSTAIN_TEXT, True)\n        return Answer(\" \".join(f\"{s.rstrip('.!?')} [{n}].\" for _, n, s in best))\n```\n\nTwo thresholds let it abstain: `min_score` for how well the top passage matched, and `min_overlap` for how much of the question the best sentence covers. At zero it always answers:\n\n``` python\ndef respond(question, answerer, k=3):\n    return answerer(question, index.search(question, k))\n\nfor question in (\"How many URLs can I send in one batch?\", \"Can I export results to Excel?\"):\n    print(respond(question, ExtractiveAnswerer()).text)\n```\n\nOutput:\n\n```\nSend up to 50 URLs in one request [1]. Each URL gets its own result [1].\nEach URL gets its own result [1]. Send up to 50 URLs in one request [1].\n```\n\nThe second answer is the problem a system that always answers has with a question the documentation doesn't cover: it quotes the closest sentence it can find. To use a model instead, give `LLMAnswerer` one function that turns a prompt into text. Nothing else in the harness changes, so a difference in the results is the model's:\n\n```\nPROMPT = \"\"\"Answer the question using only the numbered passages.\nCite every sentence with its number, like [1].\nIf the passages do not contain the answer, reply exactly: {abstain}\nIf the question is ambiguous, ask one clarifying question instead.\n\n{passages}\n\nQuestion: {question}\nAnswer:\"\"\"\npython\nclass LLMAnswerer:\n    def __init__(self, complete):\n        self.complete = complete\n\n    def __call__(self, question, hits):\n        passages = \"\\n\\n\".join(f\"[{n}] {c.title}: {c.text}\" for n, (c, _) in enumerate(hits, 1))\n        text = self.complete(\n            PROMPT.format(abstain=ABSTAIN_TEXT, passages=passages, question=question)\n        ).strip()\n        return Answer(text, abstained=ABSTAIN_TEXT.lower() in text.lower())\n```\n\n## Grade citations and grounding\n\nA response can be correct and still untrustworthy. We grade three things apart from correctness. **Coverage** is the share of sentences that carry a citation. **Precision** is the share of cited chunks that come from a gold page. **Recall** is the share of gold pages that were cited.\n\n``` python\nCITE = re.compile(r\"\\[(\\d+)\\]\")\n\ndef split_sentences(text):\n    return [s.strip() for s in re.split(r\"(?<=[.!?\\]])\\s+(?=[A-Z0-9])|\\n+\", text) if s.strip()]\n\ndef cited_chunks(sentence, hits):\n    return [hits[int(n) - 1][0] for n in CITE.findall(sentence) if 0 < int(n) <= len(hits)]\npython\ndef grade_citations(text, hits, q):\n    \"\"\"Citation coverage, precision and recall, measured against the gold pages.\"\"\"\n    cited = [cited_chunks(s, hits) for s in split_sentences(text)]\n    used = {c.id: c for group in cited for c in group}\n    urls = {c.url for c in used.values()}\n    gold = set(q.gold_sources)\n    return {\n        \"coverage\": sum(bool(g) for g in cited) / len(cited) if cited else 0.0,\n        \"precision\": (\n            sum(c.url in gold for c in used.values()) / len(used) if used and gold else None\n        ),\n        \"recall\": len(gold & urls) / len(gold) if gold else None,\n    }\n```\n\n**Grounding** asks whether a cited sentence is supported by the chunks it cites. A sentence is supported when every number in it appears in the cited text, and 80% of its words do:\n\n``` python\ndef numbers(text):\n    return set(re.findall(r\"\\d+(?:\\.\\d+)?\", norm(text)))\n\ndef supported(sentence, chunks):\n    \"\"\"A cited sentence is supported when its numbers and 80% of its words are in what it cites.\"\"\"\n    source, claim = \" \".join(c.text for c in chunks), CITE.sub(\"\", sentence)\n    words, have = {stem(t) for t in terms(claim)}, {stem(t) for t in terms(source)}\n    return numbers(claim) <= numbers(source) and (\n        not words or len(words & have) / len(words) >= 0.8\n    )\npython\ndef grade_grounding(text, hits):\n    cited = [(s, cited_chunks(s, hits)) for s in split_sentences(text)]\n    cited = [(s, chunks) for s, chunks in cited if chunks]\n    unsupported = [s for s, chunks in cited if not supported(s, chunks)]\n    return bool(cited) and not unsupported, unsupported\n```\n\nTwo small checks decide whether a response abstained or asked a question instead:\n\n```\nABSTAIN_PHRASES = (\n    \"can't find\", \"cannot find\", \"not covered\", \"doesn't say\",\n    \"does not say\", \"no information\", \"not documented\",\n)\nCLARIFY_WORDS = (\"which\", \"do you mean\", \"could you\", \"are you asking\", \"please specify\")\n\ndef is_abstention(answer):\n    return answer.abstained or any(p in norm(answer.text) for p in ABSTAIN_PHRASES)\n\ndef is_clarifying(text):\n    return \"?\" in text and any(w in norm(text) for w in CLARIFY_WORDS)\npython\ndef grade_answer(q, answer, hits):\n    text = answer.text\n    grounded, unsupported = grade_grounding(text, hits)\n    forbidden = any(norm(f) in norm(text) for f in q.forbidden)\n    return {\n        \"abstained\": is_abstention(answer),\n        \"clarifying\": is_clarifying(text),\n        \"correct\": facts_present(text, q.required) and not forbidden,\n        \"forbidden\": forbidden,\n        \"grounded\": grounded,\n        \"unsupported\": unsupported,\n        **grade_citations(text, hits, q),\n    }\n```\n\nHere's an answer with the wrong number and an uncited sentence. The grader sees both:\n\n```\nhits = index.search(QUESTIONS[0].question, 3)\ntext = \"You can send up to 80 URLs in one request [1]. It is very fast.\"\ng = grade_answer(QUESTIONS[0], Answer(text), hits)\nprint({key: g[key] for key in (\"correct\", \"coverage\", \"grounded\", \"precision\", \"recall\")})\nprint(\"unsupported:\", g[\"unsupported\"])\n```\n\nOutput:\n\n```\n{'correct': False, 'coverage': 0.5, 'grounded': False, 'precision': 1.0, 'recall': 1.0}\nunsupported: ['You can send up to 80 URLs in one request [1].']\n```\n\n## Grade the behaviour, then name the layer\n\nA pass depends on the class of question. An answerable question needs a correct answer that is cited and grounded. An unanswerable one needs an abstention. An ambiguous one needs a clarifying question, or a correct, cited answer:\n\n``` python\ndef passes(q, g):\n    \"\"\"Did the response do what this class of question needs?\"\"\"\n    if EXPECTED[q.cls] == \"abstain\":\n        return g[\"abstained\"] and not g[\"forbidden\"]\n    answered = not g[\"abstained\"] and g[\"correct\"] and g[\"coverage\"] == 1 and g[\"grounded\"]\n    return g[\"clarifying\"] or answered if EXPECTED[q.cls] == \"clarify_or_cover\" else answered\n```\n\nWhen a question fails, `layer_of` names the first layer that explains it, checked in pipeline order. If the fact wasn't in the page text, nothing downstream could have found it. If it was, but no retrieved passage had it, retrieval failed. If a passage had it, the answer or the decision to answer failed:\n\n``` python\ndef layer_of(q, hits, g, gold_text):\n    \"\"\"The first layer that explains a failure, checked in pipeline order. None when it passed.\"\"\"\n    if passes(q, g):\n        return None\n    if EXPECTED[q.cls] == \"abstain\":\n        return \"policy\"\n    if not facts_present(gold_text, q.required):\n        return \"ingestion\"\n    if not facts_present(\" \".join(c.text for c, _ in hits), q.required):\n        return \"retrieval\"\n    return \"policy\" if g[\"abstained\"] else \"answering\"\n```\n\nFour failures on the same question, one per layer:\n\n```\nq, gold = QUESTIONS[0], corpus[BATCH_URL].clean\ngood_hits, other_hits = index.search(q.question, 3), index.search(\"Is the Free plan large?\", 3)\ncases = [\n    (\"the page text lost the fact\", good_hits, Answer(\"Many [1].\"), NAV),\n    (\"the fact was not retrieved\", other_hits, Answer(\"Many [1].\"), gold),\n    (\"retrieved, answered wrongly\", good_hits, Answer(\"Many URLs [1].\"), gold),\n    (\"retrieved, refused to answer\", good_hits, Answer(ABSTAIN_TEXT, True), gold),\n]\nfor label, hits, answer, text in cases:\n    print(f\"{label:30s} -> {layer_of(q, hits, grade_answer(q, answer, hits), text)}\")\n```\n\nOutput:\n\n``` php\nthe page text lost the fact    -> ingestion\nthe fact was not retrieved     -> retrieval\nretrieved, answered wrongly    -> answering\nretrieved, refused to answer   -> policy\n```\n\n## Run five systems, one change at a time\n\nA system is an ingestion variant, a chunker, a retrieval depth and an answerer. Each system differs from the one before it by a single change, so each change gets measured on its own:\n\n```\nCLASSES = (\"answerable\", \"ambiguous\", \"unanswerable\")\n\n@dataclass\nclass System:\n    name: str\n    variant: str  # \"raw\" (navigation included) or \"clean\"\n    chunker: str  # \"fixed\" or \"heading\"\n    k: int\n    per_page: \"int | None\"\n    answerer: object\npython\ndef build_index(system, corpus):\n    cut = fixed_chunks if system.chunker == \"fixed\" else heading_chunks\n    return Index([c for page in corpus.values() for c in cut(page, getattr(page, system.variant))])\npython\ndef record(q, answer, g, hits, layer):\n    return {\n        \"id\": q.id,\n        \"cls\": q.cls,\n        \"split\": q.split,\n        \"question\": q.question,\n        \"answer\": answer.text,\n        \"grades\": g,\n        \"retrieved\": [c.id for c, _ in hits],\n        \"urls\": [c.url for c, _ in hits],\n        \"passed\": layer is None,\n        \"layer\": layer,\n    }\npython\ndef evaluate_question(q, system, index, corpus):\n    hits = index.search(q.question, system.k, system.per_page)\n    answer = system.answerer(q.question, hits)\n    g = grade_answer(q, answer, hits)\n    gold_text = \" \".join(getattr(corpus[u], system.variant) for u in q.gold_sources if u in corpus)\n    return record(q, answer, g, hits, layer_of(q, hits, g, gold_text))\n\ndef run_system(system, questions, corpus):\n    index = build_index(system, corpus)\n    return [evaluate_question(q, system, index, corpus) for q in questions]\npython\ndef mean(values):\n    values = [v for v in values if v is not None]\n    return round(sum(values) / len(values), 3) if values else None\n\ndef class_rates(rows):\n    return {f\"pass_{c}\": mean(r[\"passed\"] for r in rows if r[\"cls\"] == c) for c in CLASSES}\n\ndef citation_means(answered):\n    keys = {\"cite_coverage\": \"coverage\", \"cite_precision\": \"precision\", \"cite_recall\": \"recall\"}\n    means = {name: mean(r[\"grades\"][key] for r in answered) for name, key in keys.items()}\n    return means | {\"grounded\": mean(r[\"grades\"][\"grounded\"] for r in answered)}\npython\ndef gold_page_rate(answerable, questions):\n    gold = {q.id: set(q.gold_sources) for q in questions}\n    return mean(bool(gold[r[\"id\"]] & set(r[\"urls\"])) for r in answerable)\npython\ndef summarize(results, questions, split=None):\n    \"\"\"Rates by class, the gold page in the top k, citation means, and failures by layer.\"\"\"\n    rows = [r for r in results if split in (None, r[\"split\"])]\n    answerable = [r for r in rows if r[\"cls\"] == \"answerable\"]\n    answered = [r for r in answerable if not r[\"grades\"][\"abstained\"]]\n    return {\n        \"n\": len(rows),\n        \"pass\": mean(r[\"passed\"] for r in rows),\n        **class_rates(rows),\n        \"gold_page_in_top_k\": gold_page_rate(answerable, questions),\n        **citation_means(answered),\n        \"layers\": dict(Counter(r[\"layer\"] for r in rows if r[\"layer\"])),\n    }\npython\ndef standard_systems(min_score=0.0, min_overlap=0.0):\n    always = ExtractiveAnswerer()\n    careful = ExtractiveAnswerer(min_score, min_overlap)\n    return [\n        System(\"S0 raw pages, fixed chunks, top 3\", \"raw\", \"fixed\", 3, None, always),\n        System(\"S1 clean pages\", \"clean\", \"fixed\", 3, None, always),\n        System(\"S2 heading-aware chunks\", \"clean\", \"heading\", 3, None, always),\n        System(\"S3 top 5, at most 2 per page\", \"clean\", \"heading\", 5, 2, always),\n        System(\"S4 abstain below a threshold\", \"clean\", \"heading\", 5, 2, careful),\n    ]\n```\n\nThe first system reads the raw pages, cuts fixed chunks, retrieves three and always answers. Then the navigation is stripped, chunks follow the headings, retrieval goes to five with at most two per page, and finally the answerer may abstain. The last change has thresholds to choose, and they're chosen on the dev questions only, so the test questions never influence them:\n\n``` python\ndef tune_abstention(dev, corpus, scores=(0, 2, 4, 6, 8), overlaps=(0, 0.25, 0.34, 0.5, 0.67)):\n    \"\"\"Pick the thresholds with the best pass rate on the dev questions. Ties keep the lowest.\"\"\"\n    best = (-1.0, 0.0, 0.0)\n    for min_score in scores:\n        for min_overlap in overlaps:\n            system = standard_systems(min_score, min_overlap)[4]\n            rate = summarize(run_system(system, dev, corpus), dev)[\"pass\"]\n            best = max(best, (rate, -min_score, -min_overlap))\n    return -best[1], -best[2]\n```\n\nOn four questions the numbers mean little, but the mechanics are visible:\n\n```\nfor system in standard_systems():\n    s = summarize(run_system(system, QUESTIONS, corpus), QUESTIONS)\n    print(f\"{system.name:36s} pass {s['pass']:.2f}  layers {s['layers']}\")\n```\n\nOutput:\n\n```\nS0 raw pages, fixed chunks, top 3    pass 0.75  layers {'policy': 1}\nS1 clean pages                       pass 0.50  layers {'answering': 1, 'policy': 1}\nS2 heading-aware chunks              pass 0.75  layers {'policy': 1}\nS3 top 5, at most 2 per page         pass 0.75  layers {'policy': 1}\nS4 abstain below a threshold         pass 0.75  layers {'policy': 1}\n```\n\nTwo things in that output are artefacts of having four questions, not findings. S1 scores below S0 because a single answer changed, which is a 25-point swing, and S4 matches S3 because its thresholds are left at zero. The real run uses dozens of questions and tunes the thresholds on dev.\n\n## Test the grader before you trust it\n\nDeterministic graders can be wrong. A required fact written one way can be rendered another, and a correct paraphrase can fail a string match. So the harness samples 20 answers from the final system into a sheet, with the grader's verdict and its layer, and a person labels each one without looking at the verdict.\n\n## What this can't tell you\n\nThe retriever is lexical, so these numbers say nothing about an embedding retriever. The answerer is extractive, so they say nothing about how a model behaves, including whether it answers from memory.\n\nThe questions were written by one author for one corpus, and a test split of this size means one question moves a rate by several points.\n\nOur results are for our documentation, and some of the failures are gaps in that documentation, not in the pipeline. A different docs set, retriever or model will fail in different places, and that's what the layers are for.\n\n## Try it on your documentation\n\nCreate a free account at [app.spidra.io](https://app.spidra.io/), point `DOCS_URL` at your docs, and ingest them: [Map](https://docs.spidra.io/map) lists the pages for 1 credit, and [batch scrape](https://docs.spidra.io/batch-scrape) fetches them at 1 credit per page. Then write questions that can fail, add every wrong answer you see in production, and run the harness before each change to your pipeline.\n\nIf you want the ingestion side in more detail, the docs [guide on turning a docs site into RAG chunks](https://docs.spidra.io/guides/overview) covers it, and [how credits work](https://docs.spidra.io/billing/how-credits-work) has the pricing rule.", "url": "https://wpnews.pro/news/how-to-evaluate-a-documentation-rag-pipeline-with-real-questions", "canonical_source": "https://spidra.io/blog/evaluate-documentation-rag-pipeline", "published_at": "2026-11-16 00:00:00+00:00", "updated_at": "2026-11-16 16:22:15.098747+00:00", "lang": "en", "topics": ["ai-search", "artificial-intelligence", "ai-tools"], "entities": ["Spidra", "BM25"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-evaluate-a-documentation-rag-pipeline-with-real-questions", "markdown": "https://wpnews.pro/news/how-to-evaluate-a-documentation-rag-pipeline-with-real-questions.md", "text": "https://wpnews.pro/news/how-to-evaluate-a-documentation-rag-pipeline-with-real-questions.txt", "jsonld": "https://wpnews.pro/news/how-to-evaluate-a-documentation-rag-pipeline-with-real-questions.jsonld"}}