{"slug": "rag-refresher-notebook", "title": "RAG Refresher Notebook", "summary": "A new technical notebook on retrieval-augmented generation (RAG) reports that fusing keyword (BM25) and dense-embedding search lifts recall@5 to 0.94 on 18 hand-labelled questions, and adding a cross-encoder reranker raises MRR to 0.77. The notebook, which runs offline on a laptop with two small models (176 MB total) and no API key, shows that keyword search finds more answers (recall@5 0.89) while dense embeddings rank them better (MRR 0.61), and that the two methods miss different questions.", "body_md": "A notebook about retrieval-augmented generation: getting a language model to answer questions from documents it was never trained on.\n\nSearch does the work: find the few paragraphs most likely to hold the answer, and paste only those into the prompt. Nearly all the engineering lives in that finding step, and most of it predates the models. This is written as a refresher, and section 1 places each piece against the ML you already know.\n\nHere is the problem in one example. Ask *\"how do I wipe my local database and start\nover?\"* about a project whose README says `make reset`\n\n. Keyword search misses it, because\nyou did not guess the author's words. Search by meaning finds it, because \"wipe\" and\n\"reset\" sit near each other once words are vectors. Now ask which port local Postgres\nlistens on. The answer is the bare token `5433`\n\n, which keyword search puts second\nand search by meaning never surfaces at all. A real system runs both, and section 4 shows\neach one failing where the other works.\n\nAbout a twenty minute read. It is written to be *read* as much as run: outputs and figures\nare committed, so it makes sense start to finish without executing a cell.\n\nEverything runs on a laptop with no account and no API key. Two small models download once (176 MB) and after that it works offline. A full run takes under a minute.\n\nFour methods, scored on 18 hand-labelled questions:\n\n| method | recall@5 | MRR |\n|---|---|---|\n| by meaning (dense embeddings) | 0.78 | 0.61 |\n| by keyword (BM25) | 0.89 | 0.52 |\n| both, fused (RRF) | 0.94 | 0.60 |\n| fused + rerank (cross-encoder) | 0.94 |\n0.77 |\n\n**recall@5** is how often the right passage appeared in the top five. **MRR** is how high\nit landed. Grading is at passage level: a labelled question names the exact string a\ncorrect passage has to contain.\n\nKeyword search finds more answers, search by meaning ranks them better, and they miss different questions, which is the reason to run both. Fusing them lifts recall to 0.94. Only the reranker fixes the order.\n\nWith 18 questions each one is worth 0.06 recall, so this can say \"that change was a bad idea\" and cannot separate 0.89 from 0.94. Section 7 says so rather than rounding up, and spends a cell watching the eval catch a plausible chunking change that quietly breaks four of the eighteen questions.\n\nOne question has no single right answer. Two of these projects run a dev server on port\n5173, so *\"start the frontend dev server\"* is ambiguous rather than hard, and no retriever\nfixes it. A metadata filter does. The ceiling is more instructive than the score.\n\nThe problem |\nwhy a language model cannot answer from your documents, and what to do about it |\n1. Where this sits |\nsupervised, unsupervised and reinforcement; traditional ML against deep learning; and where RAG lands, which is none of them |\n2. Cutting documents into chunks |\nwhy retrieval returns passages rather than files, and what the overlap is for, shown at a real seam |\n3. Text as vectors |\nwhat an embedding is and is not, why each row is one chunk, and why cosine similarity is the idea you already know with the coordinates learned rather than chosen |\n4. Two ways to search |\nby meaning and by keyword, shown failing on different questions |\n5. Measuring retrieval |\nthe 18 labelled questions and the two metrics, before anything is improved, plus which questions each method alone gets right |\n6. Fusing and reranking |\nfusing two ranked lists, then a slower model fixing the order, measuring after each |\n7. What moves the numbers |\nthe eval catching a change that looks like an improvement, where you cut, questions with no single answer, and keeping embeddings fresh |\n8. Generation, in one prompt |\nassembling the prompt, and what each instruction prevents. It prints the prompt rather than sending it, which is what keeps the notebook keyless |\n\nWhere a piece has an obvious production counterpart, an *In production* note names it on\nthe spot: LangChain, `rank_bm25`\n\n, Elasticsearch, Cohere Rerank and the rest.\n\n```\npython -m venv .venv && .venv/bin/pip install -r requirements.txt\n.venv/bin/jupyter lab rag.ipynb\n```\n\nRun it from the repo root, since all paths are relative.\n\n13 markdown files, about 12,000 words, mostly READMEs and CLAUDE.md files from 8 public repos of mine, plus a SECURITY.md and one guide page. Far too small to need retrieval, which is the point. It is a test bed rather than a use case.\n\nEvaluating retrieval means knowing whether a result is *right*, and on an unfamiliar\ncorpus that labelling is the expensive part. It is why a lot of RAG demos stop before the\nevaluation. Here it is cheap, because I wrote every file in the corpus and can check an\nanswer by reading it.\n\n`rag.md`\n\nis the source and `rag.ipynb`\n\nis generated from it. The two are paired with\n[jupytext](https://jupytext.readthedocs.io/), so editing either updates the other:\n\n```\n.venv/bin/jupytext --sync rag.md                   # after editing either file\n.venv/bin/jupyter nbconvert --to notebook --execute --inplace \\\n    --ExecutePreprocessor.timeout=1800 rag.ipynb   # refresh the committed outputs\n```\n\n`rag.md` |\nthe source. Plain markdown, code in fences. Edit this |\n`rag.ipynb` |\ngenerated, with outputs committed |\n`figures.py` |\nmatplotlib drawing, kept out of the notebook so cells stay about retrieval |\n`corpus/` |\nthe 13 markdown files |\n`queries.json` |\n18 hand-labelled questions, each with the exact string a correct passage must contain |\n`check.py` |\nverifies every marker still appears in the corpus. Run it after changing either |\n\nI got help from Claude building this. Agents can be useful not just for coding but for creating and sharing materials that help people understand key concepts.\n\nSean Helvey. [github.com/seanhelvey](https://github.com/seanhelvey)", "url": "https://wpnews.pro/news/rag-refresher-notebook", "canonical_source": "https://github.com/seanhelvey/rag-in-context", "published_at": "2026-08-27 23:35:02+00:00", "updated_at": "2026-08-27 23:47:55.458244+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["BM25", "Cohere Rerank", "Elasticsearch", "LangChain", "Postgres"], "alternates": {"html": "https://wpnews.pro/news/rag-refresher-notebook", "markdown": "https://wpnews.pro/news/rag-refresher-notebook.md", "text": "https://wpnews.pro/news/rag-refresher-notebook.txt", "jsonld": "https://wpnews.pro/news/rag-refresher-notebook.jsonld"}}