How to evaluate a documentation RAG pipeline with real questions Spidra published a Python harness for evaluating a documentation RAG pipeline that grades citation coverage and answer grounding across answerable, ambiguous and unanswerable questions, then changes the pipeline one step at a time to show which layer failed: ingestion, retrieval, answering or the decision to answer at all. The worked example runs a BM25 retriever and a baseline answerer that quotes the documentation over Spidra's own documentation, capped at 30 pages selected by include and exclude path rules, and Spidra states the numbers belong to that one setup and are not a benchmark. The harness is built from 11 modules including config.py, ingest.py, questions.py, grade.py, retrieve.py, answer.py and evaluate.py, and Spidra notes it is grading its own docs, so the results should be treated as an example to re-run on other corpora. A documentation assistant can sound right and be wrong, and it can be right for the wrong reason. The model may know the answer from its training, ignore the passages you gave it, or never have seen the page that holds the fact. To evaluate a documentation RAG pipeline, you need questions it can fail, and a way to tell which layer failed: ingestion, retrieval, the answer, or the decision to answer at all. This article build builds that harness in Python. It ingests a defined documentation set through Spidra, runs retrieval and answering, and grades citation coverage and answer grounding against a reference set of answerable, ambiguous and unanswerable questions. Then it changes the pipeline one step at a time and shows what each change did, layer by layer. Evaluation results depend on the retriever and the model you put downstream, so every number here belongs to one setup: a BM25 retriever and a baseline answerer that quotes the documentation. They are not Spidra's numbers and they are not a benchmark. The harness takes any retriever and any answerer, and the point is the method. Our corpus is Spidra's own documentation, for three reasons: it's recent, it's ours to release, and a model is unlikely to have memorised it, so a correct answer is more likely to come from the passages than from memory. It also means we're grading our own docs, so treat the results as a worked example and run the harness on yours. The code below comes from small modules: config.py , ingest.py , corpus.py , questions.py , grade.py , chunking.py , retrieve.py , answer.py , llm answer.py , evaluate.py and systems.py . Put it in one file, in the order it appears, with these imports at the top, and every example runs. The calls to Spidra need your API key https://docs.spidra.io/authentication . python import re import sqlite3 import time import unicodedata from collections import Counter from dataclasses import dataclass, field from urllib.parse import urlsplit from spidra import Spidra Define the corpus, and what each layer owes you Fix the corpus before you write a question. A rule is better than a list you typed once, because it decides what counts as "the documentation" and can be re-run: DOCS URL = "https://docs.spidra.io/" INCLUDE = "/quickstart", "/scrape", "/batch-scrape", "/crawl", "/map", "/search", "/parse", "/options/", "/billing/", "/sdks/", "/templates", "/presets", "/monitors", "/guides/", EXCLUDE = "/sites/", "/api-reference/" MAX PAGES = 30 python def under path, prefixes : return any path == p.rstrip "/" or path.startswith p.rstrip "/" + "/" for p in prefixes python def select pages links, base=DOCS URL, include=INCLUDE, exclude=EXCLUDE, limit=MAX PAGES : """Same host, an included path, no excluded path, no duplicates, in Map's order, capped.""" host, seen, chosen = urlsplit base .netloc, set , for link in links: parts = urlsplit link path, key = parts.path.rstrip "/" or "/", parts.netloc, parts.path.rstrip "/" if parts.netloc == host and key not in seen and under path, include and not under path, exclude : seen.add key chosen.append f"{parts.scheme}://{parts.netloc}{parts.path}" return chosen :limit A failure can come from four places, and the fix for each is different. The table is the whole idea of the harness: | Layer | A failure looks like | How the harness sees it | |---|---|---| | Ingestion | The fact never reached the corpus | A required fact is missing from the gold page's text | | Retrieval | The fact is in the corpus but wasn't retrieved | No retrieved passage contains it | | Answering | It was retrieved, but the answer is wrong or unsupported | The answer lacks the fact, or its numbers aren't in the cited text | | Policy | It answered the unanswerable, or refused the answerable | The response doesn't match the behaviour the question needs | Established evaluation tools split their diagnostics the same way. RAGChecker https://github.com/amazon-science/RAGChecker reports retriever and generator metrics separately, and TruLens https://github.com/truera/trulens scores a " RAG triad " of context relevance, groundedness and answer relevance. Both use a model to judge claims. This harness doesn't need one, so every number can be checked by hand, and you can add a judge where paraphrase makes string checks too strict. Ingest the same pages two ways A crawl returns each page's Markdown including the site's navigation and footer. A batch scrape can strip them with the Extract content only option. We fetch every page both ways, on the same URLs, so the only difference between the two corpora is that boilerplate. A raw batch is 1 credit per page, with no AI involved: python def wait for client, batch id, timeout=900 : deadline = time.monotonic + timeout while time.monotonic < deadline: batch = client.get batch scrape batch id if batch.completed count + batch.failed count = batch.total urls: return batch time.sleep 5 raise TimeoutError f"batch {batch id} did not finish in {timeout} s" php def fetch markdown client, urls, content only : """- {url: status, markdown or None, credits }. A raw batch is 1 credit per page, no AI.""" queued = client.start batch scrape list urls , output="markdown", extract content only=content only items = wait for client, queued.batch id .items return { i.url: i.status, i.result if isinstance i.result, str else None, i.credits used or 0 for i in items } Each page is stored twice, because the question set has to be checked against both: @dataclass class Page: url: str title: str clean: str markdown with the navigation and footer stripped raw: str the whole page as markdown, navigation included def title of markdown, fallback : match = re.search r"^ \s+ .+ $", markdown, re.M return match.group 1 .strip if match else fallback Repeated lines give the boilerplate away. A line that appears on 80% of the pages isn't documentation: python def repeated lines pages, share=0.8 : """Lines on at least share of the pages: in practice, the navigation and footer.""" counts = {} for page in pages: for line in {ln.strip for ln in page.splitlines if ln.strip }: counts line = counts.get line, 0 + 1 need = max 1, round share len pages return sorted line for line, n in counts.items if n = need , key=lambda line: -counts line , line The examples from here on use a made-up site, Acme, on a reserved .test domain. Its pages carry a navigation line and a footer in the raw version: NAV = "Quickstart Credits Batch Errors Pricing Sign in" FOOTER = " c 2026 Acme. Terms. Privacy." def acme path, markdown : url = f"https://docs.acme.test/{path}" return Page url, title of markdown, path , markdown, f"{NAV}\n\n{markdown}\n\n{FOOTER}" BATCH = " Batch fetch\n\nSend up to 50 URLs in one request. Each URL gets its own result.\n\n" " Limits\nYou can submit 20 batches per minute.\n\n" " Failures\nA batch is completed even when items failed. Failed items are not charged.\n" CREDITS = " How credits work\n\nEach fetch costs 1 credit. A fetch with a schema costs 2 credits.\n\n" " Plans\nThe Free plan includes 300 credits.\n" corpus = {p.url: p for p in acme "batch", BATCH , acme "credits", CREDITS } for text in repeated lines p.raw for p in corpus.values : print text Output: c 2026 Acme. Terms. Privacy. Quickstart Credits Batch Errors Pricing Sign in Write questions that can fail A question set that can't fail tells you nothing. Ours has three classes, and each one needs a different behaviour: - Answerable. The documentation holds the answer. A good response states it, cites the page, and says nothing the page doesn't support. - Ambiguous. The question has more than one reading "How much does it cost?" . A good response asks which one, or covers each reading with a citation. - Unanswerable. The documentation doesn't say. A good response says so, and invents nothing. Each question records what to look for, so grading is a lookup and not a judgement: EXPECTED = {"answerable": "answer", "ambiguous": "clarify or cover", "unanswerable": "abstain"} @dataclass class Question: id: str cls: str answerable, ambiguous or unanswerable split: str dev or test question: str gold sources: list = field default factory=list pages that hold the answer required: list = field default factory=list groups of alternatives, each must appear forbidden: list = field default factory=list claims that would be wrong absence terms: list = field default factory=list unanswerable: terms the corpus must not contain source: str = "written" written by us, or a real user question required is a list of groups. Every group must appear in a good response, and each group lists alternative wordings. forbidden lists claims that would be wrong. absence terms is for unanswerable questions: words the corpus must not contain. Matching ignores case, spacing and thousands separators, so 1,000 and 1000 are the same fact: python def norm text : """Case, spacing and thousands separators should not decide whether a fact is present.""" text = unicodedata.normalize "NFKC", str text .casefold .replace "\u2019", "'" return " ".join re.sub r" ?<=\d , ?=\d{3} ", "", text .split def facts present text, required : """True when every group has at least one of its alternatives in the text.""" return all any norm alt in norm text for alt in group for group in required The questions are checked against the corpus before anyone trusts them. An answerable question whose required fact isn't on its gold page is unfair, and an unanswerable one whose key term appears anywhere, even in the navigation, isn't unanswerable: python def check unanswerable q, everything : found = f"{q.id}: the corpus contains {t r}" for t in q.absence terms if norm t in everything return found + if q.absence terms else f"{q.id}: no absence terms" def check answerable q, corpus : problems = f"{q.id}: gold page not in the corpus: {u}" for u in q.gold sources if u not in corpus if not q.gold sources and q.required : problems.append f"{q.id}: needs gold sources and required facts" gold text = " ".join corpus u .clean for u in q.gold sources if u in corpus if q.required and not facts present gold text, q.required : problems.append f"{q.id}: a required fact is not on the gold pages" return problems python def check question q, corpus, everything : if q.cls not in EXPECTED or q.split not in "dev", "test" : return f"{q.id}: bad class or split" return check unanswerable q, everything if q.cls == "unanswerable" else check answerable q, corpus python def validate questions questions, corpus : """Problems that make a question unfair or unverifiable. Empty means the set is sound.""" problems = f"duplicate id {i}" for i, n in Counter q.id for q in questions .items if n 1 everything = norm " ".join p.clean + " " + p.raw for p in corpus.values for q in questions: problems += check question q, corpus, everything return problems Here are four questions for Acme, one per kind of behaviour: python def ask id, cls, text, fields : return Question id, cls, "test", text, fields BATCH URL, CREDITS URL = "https://docs.acme.test/batch", "https://docs.acme.test/credits" QUESTIONS = ask "a1", "answerable", "How many URLs can I send in one batch?", gold sources= BATCH URL , required= "50 urls" , ask "a2", "answerable", "Do failed items cost credits?", gold sources= BATCH URL , required= "not charged" , forbidden= "1 credit" , ask "m1", "ambiguous", "How much does it cost?", gold sources= CREDITS URL , required= "1 credit" , "2 credits" , ask "u1", "unanswerable", "Can I export results to Excel?", absence terms= "excel" , print "problems:", validate questions QUESTIONS, corpus problems: And three that are broken on purpose, to show what the check catches: broken = ask "b1", "answerable", "What is the rate limit?", gold sources= BATCH URL , required= "100 per second" , ask "b2", "unanswerable", "Is there a Pricing page?", absence terms= "pricing" , ask "b3", "answerable", "Where is the changelog?", gold sources= "https://docs.acme.test/changelog" , required= "v2" , for problem in validate questions broken, corpus : print problem Output: b1: a required fact is not on the gold pages b2: the corpus contains 'pricing' b3: gold page not in the corpus: https://docs.acme.test/changelog b3: a required fact is not on the gold pages The second one is the interesting case: "pricing" appears only in the navigation menu, and the check still finds it, because it searches the raw pages too. Cut the pages into chunks Retrieval works on chunks, and how you cut decides what can be found. The simple way is fixed windows of text. The other starts a chunk at each heading and begins every chunk with the page title and its heading path, so the words in the headings are searchable: CHUNK CHARS = 1000 SECTION CHARS = 1500 FENCE = " " 3 the marker that opens and closes a code block in Markdown @dataclass class Chunk: id: str url: str title: str heading: str the section's path without the page title, empty for the opening section text: str label: str = "" title and path, the first line of the text. For retrieval, never quoted. python def fixed chunks page, text, size=CHUNK CHARS : """Windows of text that ignore the page's structure.""" return Chunk f"{page.url} {n}", page.url, page.title, "", text start : start + size for n, start in enumerate range 0, len text , size HEADING = re.compile r" {1,4} \s+ .+ " python def sections markdown : """ heading path, body . Headings count only outside code fences. Empty sections drop.""" found, path, body, fenced = , , , False for line in markdown.splitlines : fenced ^= line.startswith FENCE heading = None if fenced else HEADING.fullmatch line if heading: found.append path, "\n".join body .strip path, body = path : len heading.group 1 - 1 + heading.group 2 .strip , else: body.append line found.append path, "\n".join body .strip return " ".join p , b for p, b in found if b python def paragraphs body : """Blocks separated by blank lines. A code fence is never split.""" blocks, current, fenced = , , False for line in body.splitlines : fenced ^= line.startswith FENCE if not fenced and not line.strip : if current: blocks.append "\n".join current current = else: current.append line return blocks + "\n".join current if current else python def split long body, max chars : """Pack whole paragraphs into pieces of about max chars. One long paragraph stays whole.""" pieces, piece = , "" for block in paragraphs body : if piece and len piece + len block max chars: pieces.append piece piece = "" piece = f"{piece}\n\n{block}".strip return pieces + piece if piece else def split heading title, path : """The section path without the page title, and the label every chunk starts with.""" heading = path len title : .removeprefix " " if path.startswith title else path return heading, f"{title} {heading}" if heading else title python def heading chunks page, text, max chars=SECTION CHARS : """One chunk per section, each starting with the page title and its heading path.""" chunks = for path, body in sections text : heading, label = split heading page.title, path for piece in split long body, max chars : chunk id = f"{page.url} {len chunks }" chunks.append Chunk chunk id, page.url, page.title, heading, f"{label}\n{piece}", label return chunks Headings are only recognised outside code fences, because a comment in a code block isn't a section: page = corpus BATCH URL for chunk in heading chunks page, page.clean : print repr chunk.label , "|", chunk.text.splitlines 1 :46 print len fixed chunks page, page.clean, size=100 , "fixed chunks of 100 characters" Output: 'Batch fetch' | Send up to 50 URLs in one request. Each URL ge 'Batch fetch Limits' | You can submit 20 batches per minute. 'Batch fetch Failures' | A batch is completed even when items failed. F 3 fixed chunks of 100 characters Retrieve with BM25 We use BM25 through SQLite's full-text search, so there's nothing to install and every score can be reproduced. It's a lexical retriever. If you use embeddings, the harness only needs an object with the same search method: STOP = set "a an and are as at be by can do does for from how i in is it its of on or that the this to " "what when where which who why will with you your".split def terms text : """Lower-case words without stop words. An identifier such as batch scrape stays whole.""" return w for w in re.findall r" a-z0-9 +", text.casefold if w not in STOP def match query question : """An OR query of quoted word pieces. Quoting keeps stray punctuation out of FTS5 syntax.""" pieces = p for term in terms question for p in re.findall r" ^\W +", term return " OR ".join f'"{p}"' for p in dict.fromkeys pieces python def open db chunks : """An in-memory FTS5 table with one row per chunk, ranked by BM25.""" db = sqlite3.connect ":memory:" db.execute "CREATE VIRTUAL TABLE c USING fts5 text, id UNINDEXED, " "tokenize='porter unicode61 remove diacritics 2' " db.executemany "INSERT INTO c VALUES ?, ? ", c.text, c.id for c in chunks return db python def cap per page hits, per page : """Best first, with at most per page chunks from any one page.""" seen, kept = Counter , for chunk, score in hits: if not per page or seen chunk.url < per page: seen chunk.url += 1 kept.append chunk, score return kept python class Index: def init self, chunks : self.chunks, self.db = {c.id: c for c in chunks}, open db chunks def search self, question, k=5, per page=None : """- chunk, score , best first. per page caps how many chunks one page may supply.""" query = match query question sql = "SELECT id, -bm25 c FROM c WHERE c MATCH ? ORDER BY bm25 c LIMIT ?" rows = self.db.execute sql, query, k 20 if query else return cap per page self.chunks i , score for i, score in rows , per page :k per page caps how many chunks one page can supply, so a single long page can't fill every slot: chunks = c for p in corpus.values for c in heading chunks p, p.clean index = Index chunks for chunk, score in index.search "How many URLs can I send in one batch?", k=3 : print f"{score:6.3f}", chunk.label Output: 3.488 Batch fetch 0.000 Batch fetch Limits 0.000 Batch fetch Failures Answer with something you can inspect The baseline answerer quotes. It scores every sentence in the retrieved chunks by how many of the question's words it contains, and returns the best two with a citation number. There's no model to blame, and you can read every answer it gives: ABSTAIN TEXT = "I can't find this in the documentation." SPLIT = re.compile r" ?<= . ? \s+" @dataclass class Answer: text: str abstained: bool = False def stem word : """Strip a plural s, so credits matches credit. Short words and ss, us, is endings stay.""" plural = len word 3 and word.endswith "s" and not word.endswith "ss", "us", "is" return word :-1 if plural else word python def plain text : """Markdown as words: list marks, links, backticks and table bars removed.""" text = re.sub r"\ ^\ \ \ ^ \ ", r"\1", text .replace " ", "" .replace "|", " " return " ".join re.sub r"^ \-\s +", "", text .split python def candidate sentences chunk : """Sentences worth quoting: not code, headings or the label, and three words or more.""" lines, fenced = chunk.text.splitlines , False if chunk.label and lines and lines 0 == chunk.label: lines = lines 1: found = for line in lines: fenced ^= line.startswith FENCE if not fenced and not line.startswith FENCE, " " : found += plain s for s in SPLIT.split line return s for s in found if len s.split = 3 python def overlap sentence, wanted : """The share of the wanted words stemmed that the sentence contains.""" have = {stem t for t in terms sentence } return len {stem w for w in wanted} & have / len wanted if wanted else 0.0 def best sentences question, hits, take=2 : """ overlap, chunk number, sentence for the best-matching sentences, best first.""" wanted = set terms question scored = overlap s, wanted , -n, -i, s for n, chunk, in enumerate hits, 1 for i, s in enumerate candidate sentences chunk return o, -n, s for o, n, , s in sorted scored, reverse=True :take class ExtractiveAnswerer: """Quotes the best-matching sentences, each with its chunk number. No model involved.""" def init self, min score=0.0, min overlap=0.0, take=2 : self.min score, self.min overlap, self.take = min score, min overlap, take def call self, question, hits : best = best sentences question, hits, self.take if not best or best 0 0 < self.min overlap or hits 0 1 < self.min score: return Answer ABSTAIN TEXT, True return Answer " ".join f"{s.rstrip '. ?' } {n} ." for , n, s in best Two thresholds let it abstain: min score for how well the top passage matched, and min overlap for how much of the question the best sentence covers. At zero it always answers: python def respond question, answerer, k=3 : return answerer question, index.search question, k for question in "How many URLs can I send in one batch?", "Can I export results to Excel?" : print respond question, ExtractiveAnswerer .text Output: Send up to 50 URLs in one request 1 . Each URL gets its own result 1 . Each URL gets its own result 1 . Send up to 50 URLs in one request 1 . The second answer is the problem a system that always answers has with a question the documentation doesn't cover: it quotes the closest sentence it can find. To use a model instead, give LLMAnswerer one function that turns a prompt into text. Nothing else in the harness changes, so a difference in the results is the model's: PROMPT = """Answer the question using only the numbered passages. Cite every sentence with its number, like 1 . If the passages do not contain the answer, reply exactly: {abstain} If the question is ambiguous, ask one clarifying question instead. {passages} Question: {question} Answer:""" python class LLMAnswerer: def init self, complete : self.complete = complete def call self, question, hits : passages = "\n\n".join f" {n} {c.title}: {c.text}" for n, c, in enumerate hits, 1 text = self.complete PROMPT.format abstain=ABSTAIN TEXT, passages=passages, question=question .strip return Answer text, abstained=ABSTAIN TEXT.lower in text.lower Grade citations and grounding A response can be correct and still untrustworthy. We grade three things apart from correctness. Coverage is the share of sentences that carry a citation. Precision is the share of cited chunks that come from a gold page. Recall is the share of gold pages that were cited. python CITE = re.compile r"\ \d+ \ " def split sentences text : return s.strip for s in re.split r" ?<= . ?\ \s+ ?= A-Z0-9 |\n+", text if s.strip def cited chunks sentence, hits : return hits int n - 1 0 for n in CITE.findall sentence if 0 < int n <= len hits python def grade citations text, hits, q : """Citation coverage, precision and recall, measured against the gold pages.""" cited = cited chunks s, hits for s in split sentences text used = {c.id: c for group in cited for c in group} urls = {c.url for c in used.values } gold = set q.gold sources return { "coverage": sum bool g for g in cited / len cited if cited else 0.0, "precision": sum c.url in gold for c in used.values / len used if used and gold else None , "recall": len gold & urls / len gold if gold else None, } Grounding asks whether a cited sentence is supported by the chunks it cites. A sentence is supported when every number in it appears in the cited text, and 80% of its words do: python def numbers text : return set re.findall r"\d+ ?:\.\d+ ?", norm text def supported sentence, chunks : """A cited sentence is supported when its numbers and 80% of its words are in what it cites.""" source, claim = " ".join c.text for c in chunks , CITE.sub "", sentence words, have = {stem t for t in terms claim }, {stem t for t in terms source } return numbers claim <= numbers source and not words or len words & have / len words = 0.8 python def grade grounding text, hits : cited = s, cited chunks s, hits for s in split sentences text cited = s, chunks for s, chunks in cited if chunks unsupported = s for s, chunks in cited if not supported s, chunks return bool cited and not unsupported, unsupported Two small checks decide whether a response abstained or asked a question instead: ABSTAIN PHRASES = "can't find", "cannot find", "not covered", "doesn't say", "does not say", "no information", "not documented", CLARIFY WORDS = "which", "do you mean", "could you", "are you asking", "please specify" def is abstention answer : return answer.abstained or any p in norm answer.text for p in ABSTAIN PHRASES def is clarifying text : return "?" in text and any w in norm text for w in CLARIFY WORDS python def grade answer q, answer, hits : text = answer.text grounded, unsupported = grade grounding text, hits forbidden = any norm f in norm text for f in q.forbidden return { "abstained": is abstention answer , "clarifying": is clarifying text , "correct": facts present text, q.required and not forbidden, "forbidden": forbidden, "grounded": grounded, "unsupported": unsupported, grade citations text, hits, q , } Here's an answer with the wrong number and an uncited sentence. The grader sees both: hits = index.search QUESTIONS 0 .question, 3 text = "You can send up to 80 URLs in one request 1 . It is very fast." g = grade answer QUESTIONS 0 , Answer text , hits print {key: g key for key in "correct", "coverage", "grounded", "precision", "recall" } print "unsupported:", g "unsupported" Output: {'correct': False, 'coverage': 0.5, 'grounded': False, 'precision': 1.0, 'recall': 1.0} unsupported: 'You can send up to 80 URLs in one request 1 .' Grade the behaviour, then name the layer A pass depends on the class of question. An answerable question needs a correct answer that is cited and grounded. An unanswerable one needs an abstention. An ambiguous one needs a clarifying question, or a correct, cited answer: python def passes q, g : """Did the response do what this class of question needs?""" if EXPECTED q.cls == "abstain": return g "abstained" and not g "forbidden" answered = not g "abstained" and g "correct" and g "coverage" == 1 and g "grounded" return g "clarifying" or answered if EXPECTED q.cls == "clarify or cover" else answered When a question fails, layer of names the first layer that explains it, checked in pipeline order. If the fact wasn't in the page text, nothing downstream could have found it. If it was, but no retrieved passage had it, retrieval failed. If a passage had it, the answer or the decision to answer failed: python def layer of q, hits, g, gold text : """The first layer that explains a failure, checked in pipeline order. None when it passed.""" if passes q, g : return None if EXPECTED q.cls == "abstain": return "policy" if not facts present gold text, q.required : return "ingestion" if not facts present " ".join c.text for c, in hits , q.required : return "retrieval" return "policy" if g "abstained" else "answering" Four failures on the same question, one per layer: q, gold = QUESTIONS 0 , corpus BATCH URL .clean good hits, other hits = index.search q.question, 3 , index.search "Is the Free plan large?", 3 cases = "the page text lost the fact", good hits, Answer "Many 1 ." , NAV , "the fact was not retrieved", other hits, Answer "Many 1 ." , gold , "retrieved, answered wrongly", good hits, Answer "Many URLs 1 ." , gold , "retrieved, refused to answer", good hits, Answer ABSTAIN TEXT, True , gold , for label, hits, answer, text in cases: print f"{label:30s} - {layer of q, hits, grade answer q, answer, hits , text }" Output: php the page text lost the fact - ingestion the fact was not retrieved - retrieval retrieved, answered wrongly - answering retrieved, refused to answer - policy Run five systems, one change at a time A system is an ingestion variant, a chunker, a retrieval depth and an answerer. Each system differs from the one before it by a single change, so each change gets measured on its own: CLASSES = "answerable", "ambiguous", "unanswerable" @dataclass class System: name: str variant: str "raw" navigation included or "clean" chunker: str "fixed" or "heading" k: int per page: "int | None" answerer: object python def build index system, corpus : cut = fixed chunks if system.chunker == "fixed" else heading chunks return Index c for page in corpus.values for c in cut page, getattr page, system.variant python def record q, answer, g, hits, layer : return { "id": q.id, "cls": q.cls, "split": q.split, "question": q.question, "answer": answer.text, "grades": g, "retrieved": c.id for c, in hits , "urls": c.url for c, in hits , "passed": layer is None, "layer": layer, } python def evaluate question q, system, index, corpus : hits = index.search q.question, system.k, system.per page answer = system.answerer q.question, hits g = grade answer q, answer, hits gold text = " ".join getattr corpus u , system.variant for u in q.gold sources if u in corpus return record q, answer, g, hits, layer of q, hits, g, gold text def run system system, questions, corpus : index = build index system, corpus return evaluate question q, system, index, corpus for q in questions python def mean values : values = v for v in values if v is not None return round sum values / len values , 3 if values else None def class rates rows : return {f"pass {c}": mean r "passed" for r in rows if r "cls" == c for c in CLASSES} def citation means answered : keys = {"cite coverage": "coverage", "cite precision": "precision", "cite recall": "recall"} means = {name: mean r "grades" key for r in answered for name, key in keys.items } return means | {"grounded": mean r "grades" "grounded" for r in answered } python def gold page rate answerable, questions : gold = {q.id: set q.gold sources for q in questions} return mean bool gold r "id" & set r "urls" for r in answerable python def summarize results, questions, split=None : """Rates by class, the gold page in the top k, citation means, and failures by layer.""" rows = r for r in results if split in None, r "split" answerable = r for r in rows if r "cls" == "answerable" answered = r for r in answerable if not r "grades" "abstained" return { "n": len rows , "pass": mean r "passed" for r in rows , class rates rows , "gold page in top k": gold page rate answerable, questions , citation means answered , "layers": dict Counter r "layer" for r in rows if r "layer" , } python def standard systems min score=0.0, min overlap=0.0 : always = ExtractiveAnswerer careful = ExtractiveAnswerer min score, min overlap return System "S0 raw pages, fixed chunks, top 3", "raw", "fixed", 3, None, always , System "S1 clean pages", "clean", "fixed", 3, None, always , System "S2 heading-aware chunks", "clean", "heading", 3, None, always , System "S3 top 5, at most 2 per page", "clean", "heading", 5, 2, always , System "S4 abstain below a threshold", "clean", "heading", 5, 2, careful , The first system reads the raw pages, cuts fixed chunks, retrieves three and always answers. Then the navigation is stripped, chunks follow the headings, retrieval goes to five with at most two per page, and finally the answerer may abstain. The last change has thresholds to choose, and they're chosen on the dev questions only, so the test questions never influence them: python def tune abstention dev, corpus, scores= 0, 2, 4, 6, 8 , overlaps= 0, 0.25, 0.34, 0.5, 0.67 : """Pick the thresholds with the best pass rate on the dev questions. Ties keep the lowest.""" best = -1.0, 0.0, 0.0 for min score in scores: for min overlap in overlaps: system = standard systems min score, min overlap 4 rate = summarize run system system, dev, corpus , dev "pass" best = max best, rate, -min score, -min overlap return -best 1 , -best 2 On four questions the numbers mean little, but the mechanics are visible: for system in standard systems : s = summarize run system system, QUESTIONS, corpus , QUESTIONS print f"{system.name:36s} pass {s 'pass' :.2f} layers {s 'layers' }" Output: S0 raw pages, fixed chunks, top 3 pass 0.75 layers {'policy': 1} S1 clean pages pass 0.50 layers {'answering': 1, 'policy': 1} S2 heading-aware chunks pass 0.75 layers {'policy': 1} S3 top 5, at most 2 per page pass 0.75 layers {'policy': 1} S4 abstain below a threshold pass 0.75 layers {'policy': 1} Two things in that output are artefacts of having four questions, not findings. S1 scores below S0 because a single answer changed, which is a 25-point swing, and S4 matches S3 because its thresholds are left at zero. The real run uses dozens of questions and tunes the thresholds on dev. Test the grader before you trust it Deterministic graders can be wrong. A required fact written one way can be rendered another, and a correct paraphrase can fail a string match. So the harness samples 20 answers from the final system into a sheet, with the grader's verdict and its layer, and a person labels each one without looking at the verdict. What this can't tell you The retriever is lexical, so these numbers say nothing about an embedding retriever. The answerer is extractive, so they say nothing about how a model behaves, including whether it answers from memory. The questions were written by one author for one corpus, and a test split of this size means one question moves a rate by several points. Our results are for our documentation, and some of the failures are gaps in that documentation, not in the pipeline. A different docs set, retriever or model will fail in different places, and that's what the layers are for. Try it on your documentation Create a free account at app.spidra.io https://app.spidra.io/ , point DOCS URL at your docs, and ingest them: Map https://docs.spidra.io/map lists the pages for 1 credit, and batch scrape https://docs.spidra.io/batch-scrape fetches them at 1 credit per page. Then write questions that can fail, add every wrong answer you see in production, and run the harness before each change to your pipeline. If you want the ingestion side in more detail, the docs guide on turning a docs site into RAG chunks https://docs.spidra.io/guides/overview covers it, and how credits work https://docs.spidra.io/billing/how-credits-work has the pricing rule.