cd /news/ai-search/how-to-evaluate-a-documentation-rag-… · home › topics › ai-search › article
[ARTICLE · art-170698] src=spidra.io ↗ pub= topic=ai-search verified=true sentiment=· neutral

How to evaluate a documentation RAG pipeline with real questions

Spidra published a Python harness for evaluating a documentation RAG pipeline that grades citation coverage and answer grounding across answerable, ambiguous and unanswerable questions, then changes the pipeline one step at a time to show which layer failed: ingestion, retrieval, answering or the decision to answer at all. The worked example runs a BM25 retriever and a baseline answerer that quotes the documentation over Spidra's own documentation, capped at 30 pages selected by include and exclude path rules, and Spidra states the numbers belong to that one setup and are not a benchmark. The harness is built from 11 modules including config.py, ingest.py, questions.py, grade.py, retrieve.py, answer.py and evaluate.py, and Spidra notes it is grading its own docs, so the results should be treated as an example to re-run on other corpora.

by read26 min views1 publishedNov 16, 2026
How to evaluate a documentation RAG pipeline with real questions
Image: Spidra (auto-discovered)

A documentation assistant can sound right and be wrong, and it can be right for the wrong reason. The model may know the answer from its training, ignore the passages you gave it, or never have seen the page that holds the fact. To evaluate a documentation RAG pipeline, you need questions it can fail, and a way to tell which layer failed: ingestion, retrieval, the answer, or the decision to answer at all.

This article build builds that harness in Python. It ingests a defined documentation set through Spidra, runs retrieval and answering, and grades citation coverage and answer grounding against a reference set of answerable, ambiguous and unanswerable questions. Then it changes the pipeline one step at a time and shows what each change did, layer by layer.

Evaluation results depend on the retriever and the model you put downstream, so every number here belongs to one setup: a BM25 retriever and a baseline answerer that quotes the documentation. They are not Spidra's numbers and they are not a benchmark. The harness takes any retriever and any answerer, and the point is the method.

Our corpus is Spidra's own documentation, for three reasons: it's recent, it's ours to release, and a model is unlikely to have memorised it, so a correct answer is more likely to come from the passages than from memory. It also means we're grading our own docs, so treat the results as a worked example and run the harness on yours.

The code below comes from small modules: config.py, ingest.py, corpus.py, questions.py, grade.py, chunking.py, retrieve.py, answer.py, llm_answer.py, evaluate.py and systems.py. Put it in one file, in the order it appears, with these imports at the top, and every example runs. The calls to Spidra need your API key.

import re
import sqlite3
import time
import unicodedata
from collections import Counter
from dataclasses import dataclass, field
from urllib.parse import urlsplit

from spidra import Spidra

Define the corpus, and what each layer owes you #

Fix the corpus before you write a question. A rule is better than a list you typed once, because it decides what counts as "the documentation" and can be re-run:

DOCS_URL = "https://docs.spidra.io/"
INCLUDE = (
    "/quickstart", "/scrape", "/batch-scrape", "/crawl", "/map", "/search", "/parse",
    "/options/", "/billing/", "/sdks/", "/templates", "/presets", "/monitors", "/guides/",
)
EXCLUDE = ("/sites/", "/api-reference/")
MAX_PAGES = 30
python
def under(path, prefixes):
    return any(path == p.rstrip("/") or path.startswith(p.rstrip("/") + "/") for p in prefixes)
python
def select_pages(links, base=DOCS_URL, include=INCLUDE, exclude=EXCLUDE, limit=MAX_PAGES):
    """Same host, an included path, no excluded path, no duplicates, in Map's order, capped."""
    host, seen, chosen = urlsplit(base).netloc, set(), []
    for link in links:
        parts = urlsplit(link)
        path, key = parts.path.rstrip("/") or "/", (parts.netloc, parts.path.rstrip("/"))
        if (
            parts.netloc == host
            and key not in seen
            and under(path, include)
            and not under(path, exclude)
        ):
            seen.add(key)
            chosen.append(f"{parts.scheme}://{parts.netloc}{parts.path}")
    return chosen[:limit]

A failure can come from four places, and the fix for each is different. The table is the whole idea of the harness:

Layer A failure looks like How the harness sees it
Ingestion The fact never reached the corpus A required fact is missing from the gold page's text
Retrieval The fact is in the corpus but wasn't retrieved No retrieved passage contains it
Answering It was retrieved, but the answer is wrong or unsupported The answer lacks the fact, or its numbers aren't in the cited text
Policy It answered the unanswerable, or refused the answerable The response doesn't match the behaviour the question needs

Established evaluation tools split their diagnostics the same way. RAGChecker reports retriever and generator metrics separately, and TruLens scores a "RAG triad" of context relevance, groundedness and answer relevance. Both use a model to judge claims. This harness doesn't need one, so every number can be checked by hand, and you can add a judge where paraphrase makes string checks too strict.

Ingest the same pages two ways #

A crawl returns each page's Markdown including the site's navigation and footer. A batch scrape can strip them with the Extract content only option. We fetch every page both ways, on the same URLs, so the only difference between the two corpora is that boilerplate. A raw batch is 1 credit per page, with no AI involved:

def wait_for(client, batch_id, timeout=900):
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        batch = client.get_batch_scrape(batch_id)
        if batch.completed_count + batch.failed_count >= batch.total_urls:
            return batch
        time.sleep(5)
    raise TimeoutError(f"batch {batch_id} did not finish in {timeout} s")
php
def fetch_markdown(client, urls, content_only):
    """-> {url: (status, markdown or None, credits)}. A raw batch is 1 credit per page, no AI."""
    queued = client.start_batch_scrape(
        list(urls), output="markdown", extract_content_only=content_only
    )
    items = wait_for(client, queued.batch_id).items
    return {
        i.url: (i.status, i.result if isinstance(i.result, str) else None, i.credits_used or 0)
        for i in items
    }

Each page is stored twice, because the question set has to be checked against both:

@dataclass
class Page:
    url: str
    title: str
    clean: str  # markdown with the navigation and footer stripped
    raw: str  # the whole page as markdown, navigation included

def title_of(markdown, fallback):
    match = re.search(r"^#\s+(.+)$", markdown, re.M)
    return match.group(1).strip() if match else fallback

Repeated lines give the boilerplate away. A line that appears on 80% of the pages isn't documentation:

def repeated_lines(pages, share=0.8):
    """Lines on at least `share` of the pages: in practice, the navigation and footer."""
    counts = {}
    for page in pages:
        for line in {ln.strip() for ln in page.splitlines() if ln.strip()}:
            counts[line] = counts.get(line, 0) + 1
    need = max(1, round(share * len(pages)))
    return sorted(
        (line for line, n in counts.items() if n >= need), key=lambda line: (-counts[line], line)
    )

The examples from here on use a made-up site, Acme, on a reserved .test domain. Its pages carry a navigation line and a footer in the raw version:

NAV = "Quickstart Credits Batch Errors Pricing Sign in"
FOOTER = "(c) 2026 Acme. Terms. Privacy."

def acme(path, markdown):
    url = f"https://docs.acme.test/{path}"
    return Page(url, title_of(markdown, path), markdown, f"{NAV}\n\n{markdown}\n\n{FOOTER}")
BATCH = (
    "# Batch fetch\n\nSend up to 50 URLs in one request. Each URL gets its own result.\n\n"
    "## Limits\nYou can submit 20 batches per minute.\n\n"
    "## Failures\nA batch is completed even when items failed. Failed items are not charged.\n"
)
CREDITS = (
    "# How credits work\n\nEach fetch costs 1 credit. A fetch with a schema costs 2 credits.\n\n"
    "## Plans\nThe Free plan includes 300 credits.\n"
)
corpus = {p.url: p for p in (acme("batch", BATCH), acme("credits", CREDITS))}
for text in repeated_lines([p.raw for p in corpus.values()]):
    print(text)

Output:

(c) 2026 Acme. Terms. Privacy.
Quickstart Credits Batch Errors Pricing Sign in

Write questions that can fail #

A question set that can't fail tells you nothing. Ours has three classes, and each one needs a different behaviour:

  • Answerable. The documentation holds the answer. A good response states it, cites the page, and says nothing the page doesn't support.
  • Ambiguous. The question has more than one reading ("How much does it cost?"). A good response asks which one, or covers each reading with a citation.
  • Unanswerable. The documentation doesn't say. A good response says so, and invents nothing.

Each question records what to look for, so grading is a lookup and not a judgement:

EXPECTED = {"answerable": "answer", "ambiguous": "clarify_or_cover", "unanswerable": "abstain"}
@dataclass
class Question:
    id: str
    cls: str  # answerable, ambiguous or unanswerable
    split: str  # dev or test
    question: str
    gold_sources: list = field(default_factory=list)  # pages that hold the answer
    required: list = field(default_factory=list)  # groups of alternatives, each must appear
    forbidden: list = field(default_factory=list)  # claims that would be wrong
    absence_terms: list = field(
        default_factory=list
    )  # unanswerable: terms the corpus must not contain
    source: str = "written"  # written by us, or a real user question

required is a list of groups. Every group must appear in a good response, and each group lists alternative wordings. forbidden lists claims that would be wrong. absence_terms is for unanswerable questions: words the corpus must not contain. Matching ignores case, spacing and thousands separators, so 1,000 and 1000 are the same fact:

def norm(text):
    """Case, spacing and thousands separators should not decide whether a fact is present."""
    text = unicodedata.normalize("NFKC", str(text)).casefold().replace("\u2019", "'")
    return " ".join(re.sub(r"(?<=\d),(?=\d{3})", "", text).split())

def facts_present(text, required):
    """True when every group has at least one of its alternatives in the text."""
    return all(any(norm(alt) in norm(text) for alt in group) for group in required)

The questions are checked against the corpus before anyone trusts them. An answerable question whose required fact isn't on its gold page is unfair, and an unanswerable one whose key term appears anywhere, even in the navigation, isn't unanswerable:

def check_unanswerable(q, everything):
    found = [f"{q.id}: the corpus contains {t!r}" for t in q.absence_terms if norm(t) in everything]
    return found + ([] if q.absence_terms else [f"{q.id}: no absence_terms"])

def check_answerable(q, corpus):
    problems = [
        f"{q.id}: gold page not in the corpus: {u}" for u in q.gold_sources if u not in corpus
    ]
    if not (q.gold_sources and q.required):
        problems.append(f"{q.id}: needs gold sources and required facts")
    gold_text = " ".join(corpus[u].clean for u in q.gold_sources if u in corpus)
    if q.required and not facts_present(gold_text, q.required):
        problems.append(f"{q.id}: a required fact is not on the gold pages")
    return problems
python
def check_question(q, corpus, everything):
    if q.cls not in EXPECTED or q.split not in ("dev", "test"):
        return [f"{q.id}: bad class or split"]
    return (
        check_unanswerable(q, everything)
        if q.cls == "unanswerable"
        else check_answerable(q, corpus)
    )
python
def validate_questions(questions, corpus):
    """Problems that make a question unfair or unverifiable. Empty means the set is sound."""
    problems = [f"duplicate id {i}" for i, n in Counter(q.id for q in questions).items() if n > 1]
    everything = norm(" ".join(p.clean + " " + p.raw for p in corpus.values()))
    for q in questions:
        problems += check_question(q, corpus, everything)
    return problems

Here are four questions for Acme, one per kind of behaviour:

def ask(id, cls, text, **fields):
    return Question(id, cls, "test", text, **fields)

BATCH_URL, CREDITS_URL = "https://docs.acme.test/batch", "https://docs.acme.test/credits"
QUESTIONS = [
    ask("a1", "answerable", "How many URLs can I send in one batch?",
        gold_sources=[BATCH_URL], required=[["50 urls"]]),
    ask("a2", "answerable", "Do failed items cost credits?",
        gold_sources=[BATCH_URL], required=[["not charged"]], forbidden=["1 credit"]),
    ask("m1", "ambiguous", "How much does it cost?",
        gold_sources=[CREDITS_URL], required=[["1 credit"], ["2 credits"]]),
    ask("u1", "unanswerable", "Can I export results to Excel?", absence_terms=["excel"]),
]
print("problems:", validate_questions(QUESTIONS, corpus))
problems: []

And three that are broken on purpose, to show what the check catches:

broken = [
    ask("b1", "answerable", "What is the rate limit?",
        gold_sources=[BATCH_URL], required=[["100 per second"]]),
    ask("b2", "unanswerable", "Is there a Pricing page?", absence_terms=["pricing"]),
    ask("b3", "answerable", "Where is the changelog?",
        gold_sources=["https://docs.acme.test/changelog"], required=[["v2"]]),
]
for problem in validate_questions(broken, corpus):
    print(problem)

Output:

b1: a required fact is not on the gold pages
b2: the corpus contains 'pricing'
b3: gold page not in the corpus: https://docs.acme.test/changelog
b3: a required fact is not on the gold pages

The second one is the interesting case: "pricing" appears only in the navigation menu, and the check still finds it, because it searches the raw pages too.

Cut the pages into chunks #

Retrieval works on chunks, and how you cut decides what can be found. The simple way is fixed windows of text. The other starts a chunk at each heading and begins every chunk with the page title and its heading path, so the words in the headings are searchable:

CHUNK_CHARS = 1000
SECTION_CHARS = 1500
FENCE = "`" * 3  # the marker that opens and closes a code block in Markdown
@dataclass
class Chunk:
    id: str
    url: str
    title: str
    heading: str  # the section's path without the page title, empty for the opening section
    text: str
    label: str = ""  # title and path, the first line of the text. For retrieval, never quoted.
python
def fixed_chunks(page, text, size=CHUNK_CHARS):
    """Windows of text that ignore the page's structure."""
    return [
        Chunk(f"{page.url}#{n}", page.url, page.title, "", text[start : start + size])
        for n, start in enumerate(range(0, len(text), size))
    ]
HEADING = re.compile(r"(#{1,4})\s+(.+)")
python
def sections(markdown):
    """[(heading path, body)]. Headings count only outside code fences. Empty sections drop."""
    found, path, body, fenced = [], [], [], False
    for line in markdown.splitlines():
        fenced ^= line.startswith(FENCE)
        heading = None if fenced else HEADING.fullmatch(line)
        if heading:
            found.append((path, "\n".join(body).strip()))
            path, body = path[: len(heading.group(1)) - 1] + [heading.group(2).strip()], []
        else:
            body.append(line)
    found.append((path, "\n".join(body).strip()))
    return [(" > ".join(p), b) for p, b in found if b]
python
def paragraphs(body):
    """Blocks separated by blank lines. A code fence is never split."""
    blocks, current, fenced = [], [], False
    for line in body.splitlines():
        fenced ^= line.startswith(FENCE)
        if not fenced and not line.strip():
            if current:
                blocks.append("\n".join(current))
                current = []
        else:
            current.append(line)
    return blocks + (["\n".join(current)] if current else [])
python
def split_long(body, max_chars):
    """Pack whole paragraphs into pieces of about max_chars. One long paragraph stays whole."""
    pieces, piece = [], ""
    for block in paragraphs(body):
        if piece and len(piece) + len(block) > max_chars:
            pieces.append(piece)
            piece = ""
        piece = f"{piece}\n\n{block}".strip()
    return pieces + ([piece] if piece else [])

def split_heading(title, path):
    """The section path without the page title, and the label every chunk starts with."""
    heading = path[len(title) :].removeprefix(" > ") if path.startswith(title) else path
    return heading, f"{title} > {heading}" if heading else title
python
def heading_chunks(page, text, max_chars=SECTION_CHARS):
    """One chunk per section, each starting with the page title and its heading path."""
    chunks = []
    for path, body in sections(text):
        heading, label = split_heading(page.title, path)
        for piece in split_long(body, max_chars):
            chunk_id = f"{page.url}#{len(chunks)}"
            chunks.append(
                Chunk(chunk_id, page.url, page.title, heading, f"{label}\n{piece}", label)
            )
    return chunks

Headings are only recognised outside code fences, because a # comment in a code block isn't a section:

page = corpus[BATCH_URL]
for chunk in heading_chunks(page, page.clean):
    print(repr(chunk.label), "|", chunk.text.splitlines()[1][:46])
print(len(fixed_chunks(page, page.clean, size=100)), "fixed chunks of 100 characters")

Output:

'Batch fetch' | Send up to 50 URLs in one request. Each URL ge
'Batch fetch > Limits' | You can submit 20 batches per minute.
'Batch fetch > Failures' | A batch is completed even when items failed. F
3 fixed chunks of 100 characters

Retrieve with BM25 #

We use BM25 through SQLite's full-text search, so there's nothing to install and every score can be reproduced. It's a lexical retriever. If you use embeddings, the harness only needs an object with the same search method:

STOP = set(
    "a an and are as at be by can do does for from how i in is it its of on or that the this to "
    "what when where which who why will with you your".split()
)

def terms(text):
    """Lower-case words without stop words. An identifier such as batch_scrape stays whole."""
    return [w for w in re.findall(r"[a-z0-9_]+", text.casefold()) if w not in STOP]

def match_query(question):
    """An OR query of quoted word pieces. Quoting keeps stray punctuation out of FTS5 syntax."""
    pieces = [p for term in terms(question) for p in re.findall(r"[^\W_]+", term)]
    return " OR ".join(f'"{p}"' for p in dict.fromkeys(pieces))
python
def open_db(chunks):
    """An in-memory FTS5 table with one row per chunk, ranked by BM25."""
    db = sqlite3.connect(":memory:")
    db.execute(
        "CREATE VIRTUAL TABLE c USING fts5(text, id UNINDEXED, "
        "tokenize='porter unicode61 remove_diacritics 2')"
    )
    db.executemany("INSERT INTO c VALUES (?, ?)", [(c.text, c.id) for c in chunks])
    return db
python
def cap_per_page(hits, per_page):
    """Best first, with at most per_page chunks from any one page."""
    seen, kept = Counter(), []
    for chunk, score in hits:
        if not per_page or seen[chunk.url] < per_page:
            seen[chunk.url] += 1
            kept.append((chunk, score))
    return kept
python
class Index:
    def __init__(self, chunks):
        self.chunks, self.db = {c.id: c for c in chunks}, open_db(chunks)

    def search(self, question, k=5, per_page=None):
        """-> [(chunk, score)], best first. per_page caps how many chunks one page may supply."""
        query = match_query(question)
        sql = "SELECT id, -bm25(c) FROM c WHERE c MATCH ? ORDER BY bm25(c) LIMIT ?"
        rows = self.db.execute(sql, (query, k * 20)) if query else []
        return cap_per_page([(self.chunks[i], score) for i, score in rows], per_page)[:k]

per_page caps how many chunks one page can supply, so a single long page can't fill every slot:

chunks = [c for p in corpus.values() for c in heading_chunks(p, p.clean)]
index = Index(chunks)
for chunk, score in index.search("How many URLs can I send in one batch?", k=3):
    print(f"{score:6.3f}", chunk.label)

Output:

3.488 Batch fetch
0.000 Batch fetch > Limits
0.000 Batch fetch > Failures

Answer with something you can inspect #

The baseline answerer quotes. It scores every sentence in the retrieved chunks by how many of the question's words it contains, and returns the best two with a citation number. There's no model to blame, and you can read every answer it gives:

ABSTAIN_TEXT = "I can't find this in the documentation."
SPLIT = re.compile(r"(?<=[.!?])\s+")

@dataclass
class Answer:
    text: str
    abstained: bool = False

def stem(word):
    """Strip a plural s, so credits matches credit. Short words and ss, us, is endings stay."""
    plural = len(word) > 3 and word.endswith("s") and not word.endswith(("ss", "us", "is"))
    return word[:-1] if plural else word
python
def plain(text):
    """Markdown as words: list marks, links, backticks and table bars removed."""
    text = re.sub(r"\[([^\]]*)\]\([^)]*\)", r"\1", text).replace("`", "").replace("|", " ")
    return " ".join(re.sub(r"^[#>*\-\s]+", "", text).split())
python
def candidate_sentences(chunk):
    """Sentences worth quoting: not code, headings or the label, and three words or more."""
    lines, fenced = chunk.text.splitlines(), False
    if chunk.label and lines and lines[0] == chunk.label:
        lines = lines[1:]
    found = []
    for line in lines:
        fenced ^= line.startswith(FENCE)
        if not fenced and not line.startswith((FENCE, "#")):
            found += [plain(s) for s in SPLIT.split(line)]
    return [s for s in found if len(s.split()) >= 3]
python
def overlap(sentence, wanted):
    """The share of the wanted words (stemmed) that the sentence contains."""
    have = {stem(t) for t in terms(sentence)}
    return len({stem(w) for w in wanted} & have) / len(wanted) if wanted else 0.0

def best_sentences(question, hits, take=2):
    """[(overlap, chunk number, sentence)] for the best-matching sentences, best first."""
    wanted = set(terms(question))
    scored = [
        (overlap(s, wanted), -n, -i, s)
        for n, (chunk, _) in enumerate(hits, 1)
        for i, s in enumerate(candidate_sentences(chunk))
    ]
    return [(o, -n, s) for o, n, _, s in sorted(scored, reverse=True)[:take]]
class ExtractiveAnswerer:
    """Quotes the best-matching sentences, each with its chunk number. No model involved."""

    def __init__(self, min_score=0.0, min_overlap=0.0, take=2):
        self.min_score, self.min_overlap, self.take = min_score, min_overlap, take

    def __call__(self, question, hits):
        best = best_sentences(question, hits, self.take)
        if not best or best[0][0] < self.min_overlap or hits[0][1] < self.min_score:
            return Answer(ABSTAIN_TEXT, True)
        return Answer(" ".join(f"{s.rstrip('.!?')} [{n}]." for _, n, s in best))

Two thresholds let it abstain: min_score for how well the top passage matched, and min_overlap for how much of the question the best sentence covers. At zero it always answers:

def respond(question, answerer, k=3):
    return answerer(question, index.search(question, k))

for question in ("How many URLs can I send in one batch?", "Can I export results to Excel?"):
    print(respond(question, ExtractiveAnswerer()).text)

Output:

Send up to 50 URLs in one request [1]. Each URL gets its own result [1].
Each URL gets its own result [1]. Send up to 50 URLs in one request [1].

The second answer is the problem a system that always answers has with a question the documentation doesn't cover: it quotes the closest sentence it can find. To use a model instead, give LLMAnswerer one function that turns a prompt into text. Nothing else in the harness changes, so a difference in the results is the model's:

PROMPT = """Answer the question using only the numbered passages.
Cite every sentence with its number, like [1].
If the passages do not contain the answer, reply exactly: {abstain}
If the question is ambiguous, ask one clarifying question instead.

{passages}

Question: {question}
Answer:"""
python
class LLMAnswerer:
    def __init__(self, complete):
        self.complete = complete

    def __call__(self, question, hits):
        passages = "\n\n".join(f"[{n}] {c.title}: {c.text}" for n, (c, _) in enumerate(hits, 1))
        text = self.complete(
            PROMPT.format(abstain=ABSTAIN_TEXT, passages=passages, question=question)
        ).strip()
        return Answer(text, abstained=ABSTAIN_TEXT.lower() in text.lower())

Grade citations and grounding #

A response can be correct and still untrustworthy. We grade three things apart from correctness. Coverage is the share of sentences that carry a citation. Precision is the share of cited chunks that come from a gold page. Recall is the share of gold pages that were cited.

CITE = re.compile(r"\[(\d+)\]")

def split_sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?\]])\s+(?=[A-Z0-9])|\n+", text) if s.strip()]

def cited_chunks(sentence, hits):
    return [hits[int(n) - 1][0] for n in CITE.findall(sentence) if 0 < int(n) <= len(hits)]
python
def grade_citations(text, hits, q):
    """Citation coverage, precision and recall, measured against the gold pages."""
    cited = [cited_chunks(s, hits) for s in split_sentences(text)]
    used = {c.id: c for group in cited for c in group}
    urls = {c.url for c in used.values()}
    gold = set(q.gold_sources)
    return {
        "coverage": sum(bool(g) for g in cited) / len(cited) if cited else 0.0,
        "precision": (
            sum(c.url in gold for c in used.values()) / len(used) if used and gold else None
        ),
        "recall": len(gold & urls) / len(gold) if gold else None,
    }

Grounding asks whether a cited sentence is supported by the chunks it cites. A sentence is supported when every number in it appears in the cited text, and 80% of its words do:

def numbers(text):
    return set(re.findall(r"\d+(?:\.\d+)?", norm(text)))

def supported(sentence, chunks):
    """A cited sentence is supported when its numbers and 80% of its words are in what it cites."""
    source, claim = " ".join(c.text for c in chunks), CITE.sub("", sentence)
    words, have = {stem(t) for t in terms(claim)}, {stem(t) for t in terms(source)}
    return numbers(claim) <= numbers(source) and (
        not words or len(words & have) / len(words) >= 0.8
    )
python
def grade_grounding(text, hits):
    cited = [(s, cited_chunks(s, hits)) for s in split_sentences(text)]
    cited = [(s, chunks) for s, chunks in cited if chunks]
    unsupported = [s for s, chunks in cited if not supported(s, chunks)]
    return bool(cited) and not unsupported, unsupported

Two small checks decide whether a response abstained or asked a question instead:

ABSTAIN_PHRASES = (
    "can't find", "cannot find", "not covered", "doesn't say",
    "does not say", "no information", "not documented",
)
CLARIFY_WORDS = ("which", "do you mean", "could you", "are you asking", "please specify")

def is_abstention(answer):
    return answer.abstained or any(p in norm(answer.text) for p in ABSTAIN_PHRASES)

def is_clarifying(text):
    return "?" in text and any(w in norm(text) for w in CLARIFY_WORDS)
python
def grade_answer(q, answer, hits):
    text = answer.text
    grounded, unsupported = grade_grounding(text, hits)
    forbidden = any(norm(f) in norm(text) for f in q.forbidden)
    return {
        "abstained": is_abstention(answer),
        "clarifying": is_clarifying(text),
        "correct": facts_present(text, q.required) and not forbidden,
        "forbidden": forbidden,
        "grounded": grounded,
        "unsupported": unsupported,
        **grade_citations(text, hits, q),
    }

Here's an answer with the wrong number and an uncited sentence. The grader sees both:

hits = index.search(QUESTIONS[0].question, 3)
text = "You can send up to 80 URLs in one request [1]. It is very fast."
g = grade_answer(QUESTIONS[0], Answer(text), hits)
print({key: g[key] for key in ("correct", "coverage", "grounded", "precision", "recall")})
print("unsupported:", g["unsupported"])

Output:

{'correct': False, 'coverage': 0.5, 'grounded': False, 'precision': 1.0, 'recall': 1.0}
unsupported: ['You can send up to 80 URLs in one request [1].']

Grade the behaviour, then name the layer #

A pass depends on the class of question. An answerable question needs a correct answer that is cited and grounded. An unanswerable one needs an abstention. An ambiguous one needs a clarifying question, or a correct, cited answer:

def passes(q, g):
    """Did the response do what this class of question needs?"""
    if EXPECTED[q.cls] == "abstain":
        return g["abstained"] and not g["forbidden"]
    answered = not g["abstained"] and g["correct"] and g["coverage"] == 1 and g["grounded"]
    return g["clarifying"] or answered if EXPECTED[q.cls] == "clarify_or_cover" else answered

When a question fails, layer_of names the first layer that explains it, checked in pipeline order. If the fact wasn't in the page text, nothing downstream could have found it. If it was, but no retrieved passage had it, retrieval failed. If a passage had it, the answer or the decision to answer failed:

def layer_of(q, hits, g, gold_text):
    """The first layer that explains a failure, checked in pipeline order. None when it passed."""
    if passes(q, g):
        return None
    if EXPECTED[q.cls] == "abstain":
        return "policy"
    if not facts_present(gold_text, q.required):
        return "ingestion"
    if not facts_present(" ".join(c.text for c, _ in hits), q.required):
        return "retrieval"
    return "policy" if g["abstained"] else "answering"

Four failures on the same question, one per layer:

q, gold = QUESTIONS[0], corpus[BATCH_URL].clean
good_hits, other_hits = index.search(q.question, 3), index.search("Is the Free plan large?", 3)
cases = [
    ("the page text lost the fact", good_hits, Answer("Many [1]."), NAV),
    ("the fact was not retrieved", other_hits, Answer("Many [1]."), gold),
    ("retrieved, answered wrongly", good_hits, Answer("Many URLs [1]."), gold),
    ("retrieved, refused to answer", good_hits, Answer(ABSTAIN_TEXT, True), gold),
]
for label, hits, answer, text in cases:
    print(f"{label:30s} -> {layer_of(q, hits, grade_answer(q, answer, hits), text)}")

Output:

the page text lost the fact    -> ingestion
the fact was not retrieved     -> retrieval
retrieved, answered wrongly    -> answering
retrieved, refused to answer   -> policy

Run five systems, one change at a time #

A system is an ingestion variant, a chunker, a retrieval depth and an answerer. Each system differs from the one before it by a single change, so each change gets measured on its own:

CLASSES = ("answerable", "ambiguous", "unanswerable")

@dataclass
class System:
    name: str
    variant: str  # "raw" (navigation included) or "clean"
    chunker: str  # "fixed" or "heading"
    k: int
    per_page: "int | None"
    answerer: object
python
def build_index(system, corpus):
    cut = fixed_chunks if system.chunker == "fixed" else heading_chunks
    return Index([c for page in corpus.values() for c in cut(page, getattr(page, system.variant))])
python
def record(q, answer, g, hits, layer):
    return {
        "id": q.id,
        "cls": q.cls,
        "split": q.split,
        "question": q.question,
        "answer": answer.text,
        "grades": g,
        "retrieved": [c.id for c, _ in hits],
        "urls": [c.url for c, _ in hits],
        "passed": layer is None,
        "layer": layer,
    }
python
def evaluate_question(q, system, index, corpus):
    hits = index.search(q.question, system.k, system.per_page)
    answer = system.answerer(q.question, hits)
    g = grade_answer(q, answer, hits)
    gold_text = " ".join(getattr(corpus[u], system.variant) for u in q.gold_sources if u in corpus)
    return record(q, answer, g, hits, layer_of(q, hits, g, gold_text))

def run_system(system, questions, corpus):
    index = build_index(system, corpus)
    return [evaluate_question(q, system, index, corpus) for q in questions]
python
def mean(values):
    values = [v for v in values if v is not None]
    return round(sum(values) / len(values), 3) if values else None

def class_rates(rows):
    return {f"pass_{c}": mean(r["passed"] for r in rows if r["cls"] == c) for c in CLASSES}

def citation_means(answered):
    keys = {"cite_coverage": "coverage", "cite_precision": "precision", "cite_recall": "recall"}
    means = {name: mean(r["grades"][key] for r in answered) for name, key in keys.items()}
    return means | {"grounded": mean(r["grades"]["grounded"] for r in answered)}
python
def gold_page_rate(answerable, questions):
    gold = {q.id: set(q.gold_sources) for q in questions}
    return mean(bool(gold[r["id"]] & set(r["urls"])) for r in answerable)
python
def summarize(results, questions, split=None):
    """Rates by class, the gold page in the top k, citation means, and failures by layer."""
    rows = [r for r in results if split in (None, r["split"])]
    answerable = [r for r in rows if r["cls"] == "answerable"]
    answered = [r for r in answerable if not r["grades"]["abstained"]]
    return {
        "n": len(rows),
        "pass": mean(r["passed"] for r in rows),
        **class_rates(rows),
        "gold_page_in_top_k": gold_page_rate(answerable, questions),
        **citation_means(answered),
        "layers": dict(Counter(r["layer"] for r in rows if r["layer"])),
    }
python
def standard_systems(min_score=0.0, min_overlap=0.0):
    always = ExtractiveAnswerer()
    careful = ExtractiveAnswerer(min_score, min_overlap)
    return [
        System("S0 raw pages, fixed chunks, top 3", "raw", "fixed", 3, None, always),
        System("S1 clean pages", "clean", "fixed", 3, None, always),
        System("S2 heading-aware chunks", "clean", "heading", 3, None, always),
        System("S3 top 5, at most 2 per page", "clean", "heading", 5, 2, always),
        System("S4 abstain below a threshold", "clean", "heading", 5, 2, careful),
    ]

The first system reads the raw pages, cuts fixed chunks, retrieves three and always answers. Then the navigation is stripped, chunks follow the headings, retrieval goes to five with at most two per page, and finally the answerer may abstain. The last change has thresholds to choose, and they're chosen on the dev questions only, so the test questions never influence them:

def tune_abstention(dev, corpus, scores=(0, 2, 4, 6, 8), overlaps=(0, 0.25, 0.34, 0.5, 0.67)):
    """Pick the thresholds with the best pass rate on the dev questions. Ties keep the lowest."""
    best = (-1.0, 0.0, 0.0)
    for min_score in scores:
        for min_overlap in overlaps:
            system = standard_systems(min_score, min_overlap)[4]
            rate = summarize(run_system(system, dev, corpus), dev)["pass"]
            best = max(best, (rate, -min_score, -min_overlap))
    return -best[1], -best[2]

On four questions the numbers mean little, but the mechanics are visible:

for system in standard_systems():
    s = summarize(run_system(system, QUESTIONS, corpus), QUESTIONS)
    print(f"{system.name:36s} pass {s['pass']:.2f}  layers {s['layers']}")

Output:

S0 raw pages, fixed chunks, top 3    pass 0.75  layers {'policy': 1}
S1 clean pages                       pass 0.50  layers {'answering': 1, 'policy': 1}
S2 heading-aware chunks              pass 0.75  layers {'policy': 1}
S3 top 5, at most 2 per page         pass 0.75  layers {'policy': 1}
S4 abstain below a threshold         pass 0.75  layers {'policy': 1}

Two things in that output are artefacts of having four questions, not findings. S1 scores below S0 because a single answer changed, which is a 25-point swing, and S4 matches S3 because its thresholds are left at zero. The real run uses dozens of questions and tunes the thresholds on dev.

Test the grader before you trust it #

Deterministic graders can be wrong. A required fact written one way can be rendered another, and a correct paraphrase can fail a string match. So the harness samples 20 answers from the final system into a sheet, with the grader's verdict and its layer, and a person labels each one without looking at the verdict.

What this can't tell you #

The retriever is lexical, so these numbers say nothing about an embedding retriever. The answerer is extractive, so they say nothing about how a model behaves, including whether it answers from memory.

The questions were written by one author for one corpus, and a test split of this size means one question moves a rate by several points.

Our results are for our documentation, and some of the failures are gaps in that documentation, not in the pipeline. A different docs set, retriever or model will fail in different places, and that's what the layers are for.

Try it on your documentation #

Create a free account at app.spidra.io, point DOCS_URL at your docs, and ingest them: Map lists the pages for 1 credit, and batch scrape fetches them at 1 credit per page. Then write questions that can fail, add every wrong answer you see in production, and run the harness before each change to your pipeline.

If you want the ingestion side in more detail, the docs guide on turning a docs site into RAG chunks covers it, and how credits work has the pricing rule.

── more in #ai-search 4 stories · sorted by recency
── more on @spidra 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-evaluate-a-do…] indexed:0 read:26min 2026-11-16 · —