Tool Use and RAG in Production — A Complete Beginner's Study Guide

A developer published a beginner's study guide explaining tool use and retrieval-augmented generation (RAG) for large language models in production. The guide uses real-world analogies and runnable code to demonstrate how models delegate external tasks like weather lookups to deterministic functions. It emphasizes that the model only reasons and constructs arguments, while the application executes the actual function calls.

This guide assumes you know nothing about LLMs beyond "I can chat with ChatGPT/Claude." Every concept is introduced with a real-world analogy first, then explained technically, then backed by runnable code. Imagine you ask a friend, "Hey, what's the weather in Tokyo right now?" Your friend doesn't know this off the top of their head — nobody just knows live weather. So your friend pulls out their phone, opens a weather app, types "Tokyo," reads the result, and tells you: "It's 24°C and cloudy." Notice what happened: your friend the "brain" doing the reasoning never became a weather sensor. They recognized they needed external, live information, used a tool the weather app to fetch it, and then used that fetched information to answer you in natural language. This is exactly what "tool use" also called "function calling" means for an LLM like Claude or GPT. The model is the friend. It's great at reasoning and language, but it has no live connection to today's weather, your company's database, or a calculator that never makes arithmetic mistakes. So we give it a list of "apps" tools it's allowed to ask for, and something else — your application code — actually goes and uses those apps on its behalf. Tool use extends a model's capabilities to call "external systems" — anything outside the model's own weights and training data. Examples from the notes: These are all things a language model is fundamentally bad at doing internally it can't refresh live weather data, and it's notoriously unreliable at exact arithmetic , so instead of asking the model to guess, we let it delegate to a real, deterministic function. How it's wired up: get weather with {"city": "Tokyo"} ." The single most important sentence in this whole section, straight from the notes: "The model never calls a function itself " The model's role is reasoning and argument construction only . It cannot reach out to the internet, touch your database, or execute code. Everything it does is emit text/JSON. Your application is the one with hands. The full lifecycle the loop , step by step: get weather "Tokyo" for real — hits a weather API, database, calculator, whatever the tool represents.Along the way your application also has to handle practical realities like rate limits on the tools you're calling e.g., a weather API only allows so many requests per minute — that's your application's job too, not the model's. Below is a minimal, close-to-runnable example using the Anthropic SDK style referenced in the notes. It defines a get weather tool, sends a prompt, receives a tool use block, executes a mock function standing in for a real weather API , and sends the result back so the model can finish its answer. python pip install anthropic --break-system-packages import anthropic import json Step 0: create a client assumes ANTHROPIC API KEY is set as an environment variable client = anthropic.Anthropic --------------------------------------------------------------------------- Step 1: Define the tool as "a function + metadata" This is NOT the real function. It's a JSON description that tells the model "this function exists, here's its name, what it does, and what arguments it needs." --------------------------------------------------------------------------- tools = { "name": "get weather", "description": "Get the current weather for a given city.", "input schema": { "type": "object", "properties": { "city": {"type": "string", "description": "City name, e.g. Tokyo"}, "unit": { "type": "string", "enum": "celsius", "fahrenheit" , "description": "Temperature unit to return", }, }, "required": "city" , unit is optional }, } --------------------------------------------------------------------------- Step 2: This is the REAL function that will actually run. In production this would call a weather API e.g. OpenWeatherMap . Here we mock it so the example is runnable without external dependencies. --------------------------------------------------------------------------- def get weather city: str, unit: str = "celsius" - dict: """Pretend weather lookup — replace with a real HTTP call in production.""" fake database = {"Tokyo": 24, "London": 14, "Delhi": 33} temp c = fake database.get city, 20 temp = temp c if unit == "celsius" else temp c 9 / 5 + 32 return {"city": city, "temperature": temp, "unit": unit} --------------------------------------------------------------------------- Step 3: Send the user's question + the tool list to the model. The model will read the question and DECIDE if it needs the tool. --------------------------------------------------------------------------- messages = {"role": "user", "content": "What is the weather in Tokyo?"} response = client.messages.create model="claude-sonnet-4-6", max tokens=1024, tools=tools, messages=messages, --------------------------------------------------------------------------- Step 4: "Application intercepts the response." We look through the model's response blocks. If it contains a "tool use" block, that means the model wants us to run a function on its behalf. The model itself did NOT run anything — it only asked. --------------------------------------------------------------------------- tool use block = None for block in response.content: if block.type == "tool use": tool use block = block break if tool use block: print f"Model wants to call: {tool use block.name} with {tool use block.input}" Step 5: "Application invokes the function." We actually run the real code. if tool use block.name == "get weather": result = get weather tool use block.input else: result = {"error": "unknown tool"} Step 6: "Application sends the result back to model" as a tool result message. We must echo back the ORIGINAL assistant message containing the tool use plus a new "user" message containing the tool result, so the model has full context of what it asked for and what it got back. messages.append {"role": "assistant", "content": response.content} messages.append { "role": "user", "content": { "type": "tool result", "tool use id": tool use block.id, "content": json.dumps result , } , } Step 7: "Model proceeds..." — call the model again so it can turn the raw tool result into a natural-language final answer. final response = client.messages.create model="claude-sonnet-4-6", max tokens=1024, tools=tools, messages=messages, for block in final response.content: if block.type == "text": print "Final answer:", block.text else: The model answered directly without needing a tool. for block in response.content: if block.type == "text": print "Final answer:", block.text Notice the loop shape: model decides → app intercepts → app invokes → app returns result → model proceeds. That five-step handoff is the entire mental model for tool use, and it never changes no matter how complex the tool is. Imagine you're hiring a new employee and you hand them a one-line job description: "Search for products." That's it — no other instructions. On day one, a customer asks them "how much does this cost?" and the new employee, trying to be helpful, uses the "search for products" tool because it's the only tool they were told about, even though there's actually a dedicated pricing tool sitting right next to it that they didn't know they should prefer. They also aren't told whether "search" means searching by name, by category, or by a product code — so they guess, and sometimes guess wrong. Now imagine instead you hand them a proper SOP: "Search for products by name, category, or SKU. Use this only when the customer is explicitly trying to find or filter catalog items. Do NOT use this for pricing questions or stock checks — those have their own dedicated tools." Now the employee has almost no room to misinterpret their job. This is the difference between a bad tool schema and a good one. The LLM is that new employee, and the schema is the only "training" it gets before being thrown into the job. From the notes: "JSON schema is the contract between your intent and the model's behaviour." And critically: "Ambiguous schema is the most common root cause of tool failure and is 'invisible' until production." It's invisible because in your own testing you tend to ask clean, obvious questions — production users ask messy, ambiguous, unexpected ones, and that's where a vague schema quietly breaks. There are three rules for designing a good tool schema: Rule 1 — Specify semantics, not just a description. Don't just say what the function is named or roughly does. Say when to use it , when not to use it , and what it should never be confused with . The notes' worked example: "search for products" "Search for products by name, category, or SKU. Use only when user is explicitly looking to find or filter catalog items. Do not use for pricing questions or stock checks — those have other dedicated tools." Rule 2 — Be deliberate about required vs. optional parameters. If a parameter is optional and you don't clearly explain what happens when it's omitted, the model may hallucinate a value for it just to fill the slot marked with a red ✗ in the notes . Conversely, if a parameter is marked mandatory but the model doesn't have a value for it, it may pass nothing at all or an invalid placeholder also marked ✗ . The fix from the notes: provide default values for the model to use so there's no ambiguity about what an "empty" optional parameter should look like. Rule 3 — Enums are the cheapest reliability primitive you have. Whenever a parameter has a fixed, known set of valid values a category, a unit, a status , constrain it with an enum instead of leaving it as a free-form string. This "keeps things readable and reduces hallucination" — the model can only pick from the list you gave it, it can't invent "CategoryXYZ" out of thin air. The "too many tools" problem, and its fix — dynamic tool loading. The notes flag: "large number of tools: model gets confused" — when a model is handed 50+ tools at once, two hard questions creep in: "What category tool?" and "Which specific tool?" The model starts guessing between similar-looking tools. The fix is dynamic tool loading : instead of always sending the model your entire tool catalog, your application first figures out from context, intent classification, or the conversation's domain which subset of tools is actually relevant, and only sends that subset to the model for this particular turn. Fewer, more relevant choices means fewer wrong guesses. ============================================================================= BAD SCHEMA Vague description, no enum for category, ambiguous optional parameter. ============================================================================= bad tool = { "name": "search products", "description": "Search for products", <-- too vague: search how? for what purpose? "input schema": { "type": "object", "properties": { "query": {"type": "string"}, "category": {"type": "string"}, <-- free text: model may invent categories "include out of stock": {"type": "boolean"}, <-- optional, no default explained }, "required": "query" , }, } Problems this causes in production: 1. Model may call this tool for pricing questions, since nothing tells it not to. 2. Model may pass category="Electronics & Gadgets" when your real categories are "electronics", "apparel", "home" - zero results, silent failure. 3. Model may omit include out of stock, or guess True/False randomly, since there's no guidance on what happens if it's left out. ============================================================================= GOOD SCHEMA Specifies semantics when to use / not use , enum for category, explicit default behaviour for optional params. ============================================================================= good tool = { "name": "search products", "description": "Search for products by name, category, or SKU. " "Use only when the user is explicitly looking to find or filter catalog " "items. Do NOT use this for pricing questions or stock-level checks — " "those have their own dedicated tools get price, check stock ." , "input schema": { "type": "object", "properties": { "query": { "type": "string", "description": "Free-text search term, e.g. a product name or SKU.", }, "category": { "type": "string", Enum = the model can ONLY pick from these exact values. "enum": "electronics", "apparel", "home", "books", "toys" , "description": "Optional. Restrict results to one catalog category.", }, "include out of stock": { "type": "boolean", "description": "Whether to include out-of-stock items. " "If omitted, DEFAULTS to false only show in-stock items ." , }, }, "required": "query" , }, } The good schema tells the model: what it's for, what it's NOT for, exactly which category strings are valid, and exactly what "not specifying" means. That triple clarity is what the notes call the "contract" between your intent and the model's behaviour. A tiny illustration of picking a relevant SUBSET of tools before calling the model, instead of always sending your entire tool catalog. ALL TOOLS = { "search products": {...}, schema dicts omitted for brevity "get price": {...}, "check stock": {...}, "get weather": {...}, "run bash command": {...}, "search hr docs": {...}, } A simple keyword-based intent router. In production this is often a small classifier model or an embedding-similarity lookup, not just keywords — but the principle narrow the toolset BEFORE calling the main model is identical. def select relevant tools user message: str - list str : msg = user message.lower if any word in msg for word in "price", "cost", "how much" : return "get price" if any word in msg for word in "stock", "available", "in stock" : return "check stock" if any word in msg for word in "find", "search", "looking for" : return "search products" if "weather" in msg: return "get weather" Fallback: give a small default set rather than everything return "search products", "get price" def build tools for request user message: str - list dict : relevant names = select relevant tools user message return ALL TOOLS name for name in relevant names if name in ALL TOOLS Usage: only the relevant tools go into the API call, not all 6+. tools for this call = build tools for request "How much does the red mug cost?" - only sends the get price tool, so the model can't confuse it with search products. Suppose you ask a travel-planning friend: "Can you check the weather in Paris, Tokyo, and New York for me?" A slow friend would check Paris, wait, tell you, then check Tokyo, wait, tell you, then check New York. A smart friend instead opens three browser tabs at once, checks all three simultaneously, and gives you all three answers together. Same total work, far less waiting. Now imagine one of those three lookups fails — say the New York weather site is down. Does your friend refuse to tell you about Paris and Tokyo just because New York failed? Or do they hand you what they do have, and mention New York didn't work? That's a judgment call — and it's the same judgment call your application has to make about tool calls. Parallel tool calls: the notes describe a model that can emit multiple tool-call requests in a single turn — e.g., "compare weather across N cities" or a "trip plan" that needs both search flight and search hotel at once. Your orchestration layer the application code, not the model is responsible for "fanning out" these calls concurrently , collecting all the results, and sending them back together. The direct payoff: it improves overall time to completion — you pay for the slowest of the N calls, not the sum of all N. The notes' guidance: use this whenever the operations are independent of one another one call's result doesn't depend on another's . Failure handling — "it depends". The notes are refreshingly honest here: "Answer to any question could well be IT DEPENDS." There isn't one universal right answer; it's a judgment call based on your product's tolerance for incompleteness vs. correctness. Three named strategies: The notes give a concrete example of when parallel fan-out matters: "4 websearch calls, lookup across multiple knowledge bases." If a question needs facts from 4 different knowledge bases, firing all 4 lookups concurrently instead of one after another is the entire difference between a fast, responsive system and a sluggish one. python import asyncio import random --------------------------------------------------------------------------- Mock tool: fetch weather for one city. Sometimes fails, to demonstrate failure handling simulating a flaky downstream API . --------------------------------------------------------------------------- async def fetch weather city: str - dict: await asyncio.sleep random.uniform 0.1, 0.5 simulate network latency if city == "New York" and random.random < 0.5: raise ConnectionError f"Weather API timed out for {city}" return {"city": city, "temp c": random.randint 10, 35 } ============================================================================= STRATEGY 1: PREVENT Validate/guard BEFORE attempting the call, so we never even try a call we already know is unsafe e.g. an unsupported city, a rate-limited tool . ============================================================================= SUPPORTED CITIES = {"Paris", "Tokyo", "New York", "London"} async def call with prevention city: str - dict: if city not in SUPPORTED CITIES: We "prevent" the failure by never calling the API for a city we know isn't supported, instead of letting it fail downstream. return {"city": city, "error": "unsupported city", "skipped call": True} return await fetch weather city ============================================================================= STRATEGY 2: ABSORB Run all calls concurrently; if one fails, catch the exception, keep going, and return only the successful results plus a note about what failed . ============================================================================= async def call with absorption cities: list str - dict: tasks = fetch weather city for city in cities return exceptions=True means a failed task returns its Exception object instead of crashing the whole asyncio.gather call. results = await asyncio.gather tasks, return exceptions=True successes, failures = , for city, result in zip cities, results : if isinstance result, Exception : failures.append {"city": city, "error": str result } else: successes.append result return {"successful results": successes, "failed cities": failures} ============================================================================= STRATEGY 3: FAIL GRACEFULLY If ANY call fails and a partial answer would be misleading e.g. this tool call was mandatory context for the user's question , abort entirely and surface a clear, honest message instead of a silently incomplete answer. ============================================================================= async def call with graceful failure cities: list str - dict: tasks = fetch weather city for city in cities try: results = await asyncio.gather tasks no return exceptions: first failure raises return {"ok": True, "results": results} except Exception as exc: return { "ok": False, "refused": True, "reason": f"Could not complete weather comparison: {exc}", } async def main : cities = "Paris", "Tokyo", "New York" print "PREVENT strategy one unsupported city :" print await asyncio.gather call with prevention c for c in "Paris", "Atlantis" print "\nABSORB strategy best-effort, partial results ok :" print await call with absorption cities print "\nFAIL GRACEFULLY strategy all-or-nothing :" print await call with graceful failure cities if name == " main ": asyncio.run main Which strategy to pick is a product decision, not a technical one — that's the "it depends" in the notes. A weather-comparison feature can tolerate "absorb" 2 out of 3 cities is still useful . A financial transaction pipeline probably needs "fail gracefully" a half-completed transfer is dangerous, not just incomplete . Say you're searching your email for something. If you search "birthday party invite," a smart search that understands meaning semantic search will find an email titled "Come celebrate turning 30 " even though none of your exact words appear in it — because it understands the concept of a birthday party. But now say you search for an exact error code, like 1099-MISC . A meaning-based search might get confused and return generic tax documents about "how do I reset my password" style topics — it doesn't realize you need an exact string match , not a conceptual one. A classic old-school keyword search like Ctrl+F, but smarter would nail this instantly because it's built for exact-term matching. Neither approach alone is good at everything. The fix: run both kinds of search at the same time , and combine merge and rerank their results — a bit like asking two friends with different strengths for restaurant recommendations, and combining their two rankings into one better, more trustworthy ranking. Dense retrieval semantic search : your query and your documents are both converted into vectors "embeddings" that capture meaning . Two pieces of text that mean similar things end up as vectors that are close together in space, even if they don't share any exact words. This is great when there's genuine semantic similarity — the notes' example: "How do I reset my password?" naturally matches a document titled "Account Recovery Steps," even with zero overlapping words. Sparse retrieval / BM25 : this is the notes' name for traditional keyword-based search "BM25" = "Best Matching 25," a well-known statistical formula scoring how well a document's exact terms match a query's exact terms, weighted by term rarity and frequency . BM25 is the opposite strength of dense search: it's excellent at exact terms, codes, IDs, acronyms — things like "error code 1099-MISC" — but it has no concept of meaning, so it's poor at "How do I reset my password?" style natural-language questions where the right document doesn't share your exact wording. Hybrid search = dense + sparse, run together. The notes' rule of thumb: "20|50 documents certainly having the right answer" — hybrid retrieval improves recall the fraction of genuinely relevant documents that get retrieved at all , because you're covering both failure modes semantic gap AND exact-term gap simultaneously. The fix line from the notes: "Run both and merge and rerank " Reciprocal Rank Fusion RRF is the merging technique. Analogy: imagine two friends each independently rank the same 10 restaurants from 1 best to 10 worst . To get one combined ranking that reflects both opinions, you don't just average their scores their internal scoring "scales" may be totally different: one friend rates out of 5 stars, another out of 100 points . Instead, RRF says: only the rank position matters, not the internal score — a restaurant ranked 1 by both friends should shoot to the top of the combined list, even if their raw scores were on incompatible scales. This "brings reciprocal ranks together" as the notes put it. The formula , straight from the notes: RRF d = Σ 1 / k + rank r d r∈R Read this piece by piece: d is a particular document you're scoring. R is the set of ranked result lists you're fusing in our case: the dense-search ranking and the BM25 ranking . rank r d is the position of document d in ranking r 1st place, 2nd place, etc. . k is a small constant, 1 / k + rank for The notes' visual intuition: dense list gives fractions like 1/1, 1/2, ... and sparse list gives 1/61, 1/62, ... i.e., 1/ 60+1 , 1/ 60+2 , ... — " Fusion is better " because a document doesn't need to win outright in either individual list; it just needs to consistently rank reasonably well across both to accumulate a high combined score. One more practical note from the notes: Weaviate and Elasticsearch ES support hybrid search natively — meaning they've built RRF-style fusion directly into the database. Otherwise, you have to do it yourself with a scatter-gather pattern fire off both queries yourself, gather both result sets back in your application, then fuse them . ============================================================================= Reciprocal Rank Fusion RRF — implemented from the raw formula. RRF d = sum over each ranked list r containing d of 1 / k + rank r d ============================================================================= def reciprocal rank fusion ranked lists: list list str , k: int = 60 - list tuple str, float : """ ranked lists: a list of ranked result lists. Each inner list is already sorted best-to-worst, e.g. "doc3", "doc1", "doc9" . k: the RRF damping constant 60 is the standard default from the notes . Returns a list of doc id, fused score sorted from best to worst. """ scores: dict str, float = {} for ranked list in ranked lists: for position, doc id in enumerate ranked list : rank = position + 1 ranks start at 1, not 0 This is the exact formula: 1 / k + rank scores doc id = scores.get doc id, 0.0 + 1.0 / k + rank Sort documents by their fused score, highest first fused ranking = sorted scores.items , key=lambda pair: pair 1 , reverse=True return fused ranking --- Example usage --------------------------------------------------------- Dense semantic search results for "how do I reset my password?" dense results = "account recovery steps", "login faq", "security settings", "billing faq" BM25 keyword search results for the SAME query — note it ranks things differently because it only sees exact word overlap sparse results = "login faq", "account recovery steps", "terms of service", "billing faq" fused = reciprocal rank fusion dense results, sparse results , k=60 for doc id, score in fused: print f"{doc id}: {score:.5f}" Expected behaviour: "account recovery steps" and "login faq" both rank highly in BOTH lists, so their fused scores rise above documents that only appeared in one list or ranked poorly in both. This shows the SHAPE of a real hybrid search call: one sparse BM25 query, one dense vector query, run independently, then fused with RRF. rank bm25 and sentence-transformers are common lightweight libraries. pip install rank bm25 sentence-transformers --break-system-packages from rank bm25 import BM25Okapi from sentence transformers import SentenceTransformer import numpy as np documents = "To reset your password, go to Account Recovery and click 'Forgot Password'.", "Error code 1099-MISC indicates a mismatch in the tax reporting form.", "Our billing FAQ covers refunds, invoices, and subscription changes.", doc ids = "account recovery", "error 1099 misc", "billing faq" query = "how do I reset my password" --- Sparse BM25 leg: pure keyword/token overlap scoring ----------------- tokenized corpus = doc.lower .split for doc in documents bm25 = BM25Okapi tokenized corpus bm25 scores = bm25.get scores query.lower .split sparse ranking = doc ids i for i in np.argsort bm25 scores ::-1 --- Dense semantic leg: embed query + docs, rank by cosine similarity --- embedder = SentenceTransformer "all-MiniLM-L6-v2" small, fast embedding model doc embeddings = embedder.encode documents, normalize embeddings=True query embedding = embedder.encode query, normalize embeddings=True cosine scores = doc embeddings @ query embedding dot product of normalized vectors = cosine similarity dense ranking = doc ids i for i in np.argsort cosine scores ::-1 print "Sparse BM25 ranking:", sparse ranking print "Dense semantic ranking:", dense ranking --- Fuse both rankings with RRF reusing the function from section 4.3 --- final ranking = reciprocal rank fusion dense ranking, sparse ranking , k=60 print "Final hybrid ranking:", final ranking Imagine you're searching your email for a specific message and you know it arrived last week. You could a tell your email client "only show me emails from last week," and then search the text — that's fast and narrow. Or you could b search the text across your entire mailbox history, get back thousands of results, and manually throw away everything not from last week — that's slow and wasteful, since your search engine did a ton of unnecessary work scanning old emails it was always going to discard. Now a second, more serious example: at a company, an HR employee should be able to see every department's HR documents, but a regular software engineer should only see documents relevant to their own team — they should never even see, let alone search inside, another department's confidential HR files. This isn't just a nice-to-have; it's a hard security requirement, often called RBAC Role-Based Access Control — a plain-English one-liner: "restricting what a user can see or do based on their assigned role" . Metadata filtering applies hard constraints like date ranges, document owner, department, tags either before or after the vector search runs, on top of whatever relevance ranking your retrieval produces. The notes' example query: "What was our Q1 2026 revenue?" — you'd want to filter to date range = Q1 2026 before even letting semantic search loose on the whole corpus. Beyond just retrieval quality, the notes stress this is especially important for Enterprise RAG and RBAC : an employee simply must not be able to retrieve another department's HR docs, no matter how semantically relevant those docs might look to their query. This is a security boundary, not just a relevance tweak. There are two places you can apply the filter: ============================================================================= PRE-FILTERING: the filter is passed INTO the vector database query itself. The DB never even considers documents outside the RBAC/date boundary, so ranking + filtering happen together, efficiently, in one pass. Qdrant-style filter syntax, as referenced in the notes' prototype. ============================================================================= pip install qdrant-client --break-system-packages from qdrant client import QdrantClient from qdrant client.models import Filter, FieldCondition, MatchValue, Range client = QdrantClient url="http://localhost:6333" assumes a running Qdrant instance query vector = 0.12, 0.98, 0.31 a real embedding would be much longer, e.g. 1536-dim results = client.search collection name="hr documents", query vector=query vector, query filter=Filter must= RBAC constraint: only documents this user's department may see FieldCondition key="department", match=MatchValue value="engineering" , Date constraint: only Q1 2026 documents FieldCondition key="date", range=Range gte="2026-01-01", lte="2026-03-31" , , limit=10, Qdrant applies the filter WHILE searching, so all 10 results returned are guaranteed to satisfy both constraints — no wasted compute, no leakage risk. ============================================================================= POST-FILTERING: search everything first, THEN discard invalid results in plain Python. Shown for contrast — this is what the notes call "wasteful." ============================================================================= def post filter search all search results: list dict , user department: str - list dict : """ all search results: e.g. the top-K matches from an UNFILTERED vector search across the entire corpus, regardless of department. """ filtered = doc for doc in all search results if doc "department" == user department and "2026-01-01" <= doc "date" <= "2026-03-31" return filtered Danger illustrated: if the unfiltered top-10 search returns mostly documents from OTHER departments because they happened to be semantically similar , post-filtering might leave you with just 1-2 usable results, or worse — if the filtering logic has a bug, a forbidden document could slip through into the LLM's context before being filtered, which is a real security risk with RBAC-sensitive data. Pre-filtering avoids this entirely. Bi-encoder analogy: imagine two strangers each write their own short bio independently — "I like hiking, sci-fi movies, and cooking Italian food" and "I enjoy the outdoors, watching space films, and making pasta." You then compare these two bios after the fact to guess how compatible these two people might be. It's fast you can pre-write and store millions of bios ahead of time and just compare later , but it's a bit shallow — the two people never actually talked to each other, so subtle nuances get missed. Cross-encoder analogy: now imagine instead you sit those same two strangers down together for an actual conversation. They can react to each other in real time, clarify ambiguous statements, and pick up on nuance that never would have shown up in two separately-written bios. You get a much better read on true compatibility — but it only works one pair at a time, and a live conversation takes real time and effort per pair. You couldn't have every person in a stadium have a real conversation with every other person — it simply wouldn't scale. Bi-encoders used for the first stage of dense retrieval, described back in Section 4 : the query is turned into an embedding, and each document is turned into an embedding, completely independently of each other. Relevance is then just vector proximity — how close the two embeddings are measured via cosine similarity . Because documents can be embedded once, offline, ahead of time, bi-encoders are fast at query time and scalable — this is why they're "good enough for most" use cases, per the notes. Their key weakness, from the notes: "No cross attention, i.e. query and doc never see each other." Because the two embeddings were produced in isolation, the model can't attend to specific interactions between query terms and document terms. This makes bi-encoders weaker on negation, exact terms, etc. — e.g., a bi-encoder might not clearly distinguish "the flight is NOT delayed" from "the flight IS delayed," because both sentences produce very similar embeddings overall they share almost all the same words . Cross-encoders , by contrast, take the <query, document pair as a single input — literally concatenating the query and the candidate document together — and output The notes draw a small attention diagram illustrating the "lost in the middle" problem : attention/relevance tends to concentrate most heavily near the query token and decays as you move deeper into a long document — meaning information buried in the middle of a long document or a long context window can get comparatively under-weighted relative to information at the start or end. This is a known failure mode of transformer attention over long inputs, and it's part of why reranking with a small, focused candidate set rather than dumping a giant unranked context at the LLM genuinely helps. Cross-encoders: much higher accuracy — they can compare every token in the query to every token in the document, so they handle negation, exact matches, etc. far better than bi-encoders. Example model named in the notes: cross-encoder/ms-marco-MiniLM-L-6-v2 . The catch — cross-encoders are slow and don't scale. From the notes: they work well on "50 doc s v s million doc s per query" — meaning cross-encoders are only practical on a small shortlist, never the entire corpus. There's no indexing possible because the score depends on the specific query, you can't precompute it ahead of time the way you can with bi-encoder embeddings , and each pair costs roughly ~10ms of latency . The latency math, worked out step by step from the notes : <query, doc pairThat's obviously unusable in a real product. The fix, stated directly in the notes: "could be done on candidate docs only." This is exactly why the two techniques are used together in a pipeline, not as alternatives: use the cheap, scalable bi-encoder plus BM25/hybrid search to narrow millions of documents down to a small shortlist say, the top 40 , and then run the expensive, high-accuracy cross-encoder reranker only on that small shortlist. You get bi-encoder speed at scale, and cross-encoder accuracy where it matters most — the final ranking of the few documents that actually make it into the LLM's context. pip install sentence-transformers --break-system-packages from sentence transformers import SentenceTransformer, CrossEncoder import numpy as np query = "Why is my API returning a 500 error under high load?" candidate docs = "Throughput degradation under load can trigger unhandled exceptions, " "which surface to callers as HTTP 500 errors.", "To reset your password, click 'Forgot Password' on the login screen.", "500 errors are NOT related to client-side input validation issues.", "Our billing FAQ covers refunds, invoices, and subscription cancellations.", ============================================================================= STAGE 1 fast, scalable : BI-ENCODER — embed query and docs SEPARATELY, then compare with cosine similarity. This is what you'd run across millions of documents to get an initial shortlist. ============================================================================= bi encoder = SentenceTransformer "all-MiniLM-L6-v2" query vec = bi encoder.encode query, normalize embeddings=True doc vecs = bi encoder.encode candidate docs, normalize embeddings=True cosine similarity = dot product of normalized vectors bi encoder scores = doc vecs @ query vec print "Bi-encoder ranking fast, first-pass :" for idx in np.argsort bi encoder scores ::-1 : print f" {bi encoder scores idx :.3f} {candidate docs idx :60 }..." ============================================================================= STAGE 2 slow, accurate : CROSS-ENCODER — feed the query+doc TOGETHER as a single pair, get one relevance score per pair. Only run this on the small shortlist that stage 1 already narrowed down here, all 4, since it's tiny . ============================================================================= cross encoder = CrossEncoder "cross-encoder/ms-marco-MiniLM-L-6-v2" pairs = query, doc for doc in candidate docs literal <query, doc pairing cross encoder scores = cross encoder.predict pairs print "\nCross-encoder ranking slow, high-precision rerank :" for idx in np.argsort cross encoder scores ::-1 : print f" {cross encoder scores idx :.3f} {candidate docs idx :60 }..." You'll typically notice the cross-encoder more confidently separates the TRUE match the throughput/500-error doc from the negation trap doc 3, which mentions "500 errors" but explicitly says they are NOT related to the topic — a distinction bi-encoders are known to struggle with. Imagine you ask a librarian, "Why is my code slow?" The librarian's catalog, though, doesn't use casual words like "slow" — the technical manuals on the shelf are indexed under formal terms like "throughput degradation," "latency regression," or "bottleneck analysis." If the librarian searches the catalog using your exact casual phrase, they might get nothing useful back, even though the perfect manual for your problem exists on the shelf. The words just don't line up. A clever librarian's trick: before searching, they first jot down in their head roughly what the ideal, technical-sounding manual would probably say about your problem — then they search the catalog using that imagined, more formal description instead of your casual phrasing. That imagined "ideal answer" acts as a bridge between your everyday words and the library's formal vocabulary. That's exactly what HyDE Hypothetical Document Embedding does for a RAG system. The notes call the "Query-Document Gap" the most persistent failure mode in RAG . It shows up because user queries tend to be short and conversational , while the underlying documents/corpus are often semantically distant in their vocabulary — the classic example: "why is my pipeline slow?" user's words vs. "Throughput degradation is..." the document's actual wording . Even though these mean almost the same thing conceptually, their raw embeddings may not be close enough for a naive semantic search to reliably connect them. There are two named fixes in the notes: 1. Expand or rephrase the user query. Simply ask an LLM: "generate 3 alternate phrasings..." of the user's question, and search using all of them or a combination , increasing the odds that at least one phrasing's embedding lands close to the real document. 2. HyDE — Hypothetical Document Embedding. Instead of embedding the user's raw, short question, you first ask an LLM to write a short document that would answer this question — the exact prompt style from the notes: "Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like." You then take that generated hypothetical document — not the original question — and embed it . Because this hypothetical answer is written in a style and vocabulary much closer to how real documents in your corpus are likely written, its embedding tends to land much closer to the real relevant document's embedding than the original short, casual question would have. Technically, HyDE reframes retrieval as a similarity search between d and d ∈ D — that is, comparing a hypothetical document to the real documents in your corpus D , rather than comparing a short question to documents which is a fundamentally harder, more mismatched comparison . When does this actually help? The notes are precise about this: HyDE "works when corpus vocab is very different than user query." Good candidates: scientific papers, legal texts, internal jargon-heavy documentation — domains where the "real" documents are written in dense, formal, or highly technical language that a typical user's question would never naturally use. python pip install anthropic sentence-transformers --break-system-packages import anthropic from sentence transformers import SentenceTransformer import numpy as np client = anthropic.Anthropic embedder = SentenceTransformer "all-MiniLM-L6-v2" def generate hypothetical answer user query: str - str: """ Ask an LLM to write a short, FACTUAL-SOUNDING document that would answer the user's question. This is the exact prompt shape from the notes: "Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like." We use this generated text purely to bridge the vocabulary gap — we do NOT show this hypothetical text to the end user; it's an internal retrieval-only artifact. """ response = client.messages.create model="claude-sonnet-4-6", max tokens=300, messages= { "role": "user", "content": "Write a short document that will answer this question. " "Be factual. If you don't know the answer, write what the " f"answer would typically look like.\n\nQuestion: {user query}" , } , return response.content 0 .text def hyde retrieve user query: str, doc texts: list str , doc ids: list str , top k: int = 3 : """ Full HyDE retrieval: generate a hypothetical answer, embed THAT instead of the raw query, then find the closest real documents to it. """ Step 1: generate the hypothetical, jargon-matched answer hypothetical doc = generate hypothetical answer user query Step 2: embed the HYPOTHETICAL document, not the raw user query hyde embedding = embedder.encode hypothetical doc, normalize embeddings=True Step 3: embed the real corpus documents in production this is pre-computed and stored in a vector DB, not re-computed on every query doc embeddings = embedder.encode doc texts, normalize embeddings=True Step 4: rank real documents by similarity to the hypothetical embedding similarities = doc embeddings @ hyde embedding top indices = np.argsort similarities ::-1 :top k return doc ids i , doc texts i , float similarities i for i in top indices --- Example ----------------------------------------------------------------- corpus texts = "Throughput degradation under sustained load is typically caused by " "connection pool exhaustion or unindexed database queries.", "Our refund policy allows cancellations within 30 days of purchase.", "Latency regressions in distributed systems often trace back to N+1 " "query patterns or missing caching layers.", corpus ids = "perf doc 1", "refund policy", "perf doc 2" results = hyde retrieve "why is my code slow?", corpus texts, corpus ids for doc id, text, score in results: print f"{score:.3f} {doc id}: {text :70 }..." Without HyDE, embedding the raw casual query "why is my code slow?" might rank the perf docs only weakly, since they never use the word "slow." With HyDE, the generated hypothetical answer naturally uses words like "throughput," "latency," and "degradation" -- much closer to the real docs. Imagine ten different customers each phrase the exact same underlying question in slightly different words: "How do I integrate libX?", "I want help integrating libX," "How can I get libX working with my app?" A naive cache the kind that only matches exact strings would treat these as three totally unrelated questions and pay to compute a fresh, expensive answer for every single one — even though the right answer is identical for all three. A smarter cache recognizes that all three questions mean the same thing, and serves the same cached, already-computed answer to all of them — saving time and money on the second and third askers, without them ever noticing anything different. The core motivating fact from the notes: "Every call to LLM cost s latency and tokens" — i.e., time and money . A semantic cache sits in front of the LLM: it intercepts a query before it reaches the model and, if a semantically similar not necessarily an exact-text-match prior query exists in the cache, it returns the cached response instead of paying for a new LLM call. When this works best , per the notes: static data with no personalization — situations where the "right" answer genuinely doesn't change per-user or over short timeframes. Two named examples: Implementation, step by step from the notes : The core tension: cost vs. correctness , governed entirely by what similarity threshold you pick and how you set it. The notes are blunt about the two failure directions: The reliable way to pick a threshold , per the notes: don't guess — empirically find your "sweet spot" on your own query distribution. As a reasonable starting point, the notes suggest 0.85 cosine similarity , but this must be calibrated , not treated as gospel, because every product's query patterns differ. pip install sentence-transformers --break-system-packages from sentence transformers import SentenceTransformer import numpy as np class SemanticCache: """ A minimal in-memory semantic cache: get looks for a semantically similar prior query above similarity threshold ; set stores a new query + response pair for future lookups. """ def init self, similarity threshold: float = 0.85 : self.embedder = SentenceTransformer "all-MiniLM-L6-v2" self.similarity threshold = similarity threshold self.cached queries: list str = self.cached embeddings: list np.ndarray = self.cached responses: list str = def get self, query: str - str | None: """Return a cached response if a semantically similar query exists, else None.""" if not self.cached queries: return None query embedding = self.embedder.encode query, normalize embeddings=True similarities = np.array self.cached embeddings @ query embedding best idx = int np.argmax similarities best score = float similarities best idx if best score = self.similarity threshold: print f" cache HIT similarity={best score:.3f} matched: " f"'{self.cached queries best idx }'" return self.cached responses best idx print f" cache MISS best similarity was only {best score:.3f}" return None def set self, query: str, response: str - None: """Store a new query + response pair in the cache.""" embedding = self.embedder.encode query, normalize embeddings=True self.cached queries.append query self.cached embeddings.append embedding self.cached responses.append response --- Demonstration: threshold too low vs. too high -------------------------- cache = SemanticCache similarity threshold=0.85 cache.set "How do I integrate libX?", "To integrate libX, install it via pip and call libx.init ..." print "\n--- Reasonable threshold 0.85 ---" cache.get "I want help integrating libX" should HIT: same intent, different phrasing cache.get "What's the weather in Tokyo?" should MISS: unrelated question print "\n--- Threshold set TOO LOW 0.50 : risks wrong answers ---" loose cache = SemanticCache similarity threshold=0.50 loose cache.set "How do I cancel my subscription?", "Go to Settings Billing Cancel Plan." This is a RELATED but MEANINGFULLY DIFFERENT question -- with a low threshold it may incorrectly match and serve the wrong cached answer. loose cache.get "How do I upgrade my subscription plan?" print "\n--- Threshold set TOO HIGH 0.99 : hit rate collapses ---" strict cache = SemanticCache similarity threshold=0.99 strict cache.set "How do I integrate libX?", "To integrate libX, install it via pip..." Even a near-identical paraphrase may now fail to match because embeddings are almost never PERFECTLY identical between two different sentences. strict cache.get "I want help to integrate libX" The prototype in the notes "Semantic Caching and Similarity Calibration" describes exactly this experiment at scale: 500 queries → embeddings → index , then sweeping thresholds from 0.75 to 0.95 , populating the cache first, then computing hit rate fraction of queries found in cache and accuracy/precision . It uses three types of test queries: seed the original query, e.g. "Explain the ending of movie Arrival" , paraphrase same intent, different words, e.g. "Please clarify the conclusion of the movie Arrival" — a correct match , and near-miss looks similar but has a genuinely different intent, e.g. "Who directed the movie Arrival" — a trap , since matching this to the seed's answer would be a false positive / incorrect hit . The accuracy formula from the notes: accuracy = correct hits paraphrase matching the seed / total hits Below is a small script that runs this exact style of calibration experiment: python def evaluate threshold cache: SemanticCache, seed paraphrase pairs, near miss queries : """ seed paraphrase pairs: list of seed query, paraphrase query, response tuples -- paraphrase SHOULD correctly hit the seed's cached entry. near miss queries: list of queries that LOOK similar but have different intent -- these SHOULD ideally miss or if they hit, that's a false positive . """ for seed, response in { s, r for s, , r in seed paraphrase pairs}: cache.set seed, response total hits, correct hits = 0, 0 for seed, paraphrase, in seed paraphrase pairs: result = cache.get paraphrase if result is not None: total hits += 1 correct hits += 1 paraphrase hitting its own seed = correct false positives = 0 for near miss in near miss queries: result = cache.get near miss if result is not None: total hits += 1 false positives += 1 near-miss hitting ANY cached entry = wrong accuracy = correct hits / total hits if total hits else 0 return {"hit rate relevant": correct hits, "false positives": false positives, "accuracy": accuracy} Imagine building a "help desk" for a huge company with 10 million internal documents. A visitor walks up and asks a question in plain English. Before answering, your help desk clerk must: 1 understand what's really being asked, 2 go search both the exact-keyword index and the meaning-based index of every document, 3 combine and re-sort those results by true relevance, 4 double-check that the results are actually confident and trustworthy not just "vaguely related" , 5 draft an answer strictly grounded in what was found, and — critically — 6 go back and check every single sentence of that draft answer against the source documents before saying it out loud, refusing to say anything it can't verify. That's the whole system in one sentence: retrieve carefully, answer carefully, and never say something you can't prove. What the system does: Answers natural language queries. The one hard guarantee that shapes the entire design: "the system must not assert any claim that cannot be traced to a retrieved passage." This single sentence is why "correctness" not just relevance, not just speed drives nearly every architectural decision below. Why this guarantee matters: an internal knowledge system that confidently states wrong facts is often worse than one that admits "I don't know," because wrong confident answers erode trust and can cause real harm wrong policy info, wrong financial numbers, etc. . What the system explicitly is NOT: a document management system, an ingestion pipeline product, etc. — the notes flag this so the design stays scoped: we are not building a general-purpose CMS, just the "ask questions about documents that already exist somewhere" layer. Scale numbers why they matter : The notes lay out the exact ordered checklist a systems-design interview or real design doc would walk through: Let's walk through each, grounding every architectural choice in a "why" before the "what." 1 & 2 — Storage location and Query API. Documents live in an S3 bucket e.g. s3://myrag/doc1.txt , organized per the notes with some inspiration from Git's object storage model content-addressed, organized by structure like date / git -style hashing prefixes, e.g. folders ab/ , ac/ , ad/ — a common trick to avoid dumping literally 10 million flat files into one folder, which many filesystems and object stores handle poorly . The query API is a simple HTTP endpoint: GET /query? — the notes sketch this as @app.get '/query?' . 3 — Response shape. From the notes, a single respond call returns a structured object containing: question: <string filters: { date from, date to, source domains, tag, topics, etc. } answer: <string refused: <true/false refusal reason: <string, if refused passages: {doc id, passage id, text, score, highlight idx}, ... confidence: <number query id: <string -- used for tracing/debugging a specific request later Why this shape: it's not enough to return just an answer string. You need the passages so a user or auditor can verify the claim was actually grounded , a refused boolean plus reason so "I don't know" is a first-class, explicit outcome, not a vague hedge buried in the answer text , a confidence score so downstream systems or UI can decide whether to trust/display the answer , and a query id for tracing — being able to look up exactly what happened for a specific request later, which is essential for debugging a "why did it hallucinate here" incident. 4 — Which databases, and why: 5 — Capacity estimation, worked step by step numbers straight from the notes' math : Query load: 50,000 DAU daily active users × 8 queries/day/user × 4.5 peak multiplier ────────────────────────────────────────────────────────────────────────── = 200 QPS 86,400 seconds/day So: 50,000 × 8 = 400,000 total queries/day. 400,000 ÷ 86,400 seconds ≈ 4.6 QPS average . Multiply by a peak multiplier of 4.5 because traffic isn't spread evenly across 24 hours — there are busy hours to get roughly 200 QPS at peak . The team then designs for a bit of headroom: 250 QPS target capacity. Storage — raw documents: 10,000,000 docs × 2KB each = 20GB raw data This is small enough to be almost trivial to store redundantly — it's the derived data embeddings, indexes that actually dominates storage, as shown next. Storage — vector embeddings: 10 × 10^6 documents × 8 passages/doc avg × 1536 dimensions × 4 bytes/float32 = 10×10^6 × 8 × 1536 × 4 bytes ≈ 491 GB just the raw embedding vectors Then add HNSW graph overhead the extra data structure the ANN index needs to do fast approximate search — roughly 1.5x — bringing: Total vector index storage ≈ 491GB × 1.5 ≈ 700GB This is why the notes conclude: "Hence, we need a distributed vector database" — 700GB spread across replicas for redundancy and read throughput is well beyond what a single machine comfortably handles for a low-latency, always-on service. The notes also mention Product Quantization can reduce this by roughly 8x , but "at the cost of recall" — i.e., you trade some retrieval accuracy for a much smaller memory footprint, a classic space/accuracy tradeoff. ElasticSearch BM25 index sizing: roughly 20GB × 1.5 ≈ 30GB for the inverted index, run as a 3-node cluster with high availability HA plus 3 replicas, ≈ 90GB total , each node with 30GB RAM . Postgres sizing: the 20GB of metadata plus JSONB overhead and replication comes out to roughly 60-80GB , served with 3 read replicas since this is read-heavy traffic with a target replication lag of under 100ms . Redis: a 3-node cluster with 3 replicas , caching passages and query results — this is explicitly conditioned on the earlier semantic-caching insight: it works best "if no personalization & data is static." 6, 7, 8 — Read path, what's returned, and failure handling are covered together via the code walkthrough in Section 9.4 below the notes' own respond pseudocode . 9 — Write path is covered in Section 9.5 below. respond function , cleaned up and fully commented The notes' own hand-drawn pseudocode for the read path is the heart of the "zero hallucinations" guarantee. Here it is translated into clean, fully-commented Python: python from dataclasses import dataclass, field @dataclass class RespondResult: answer: str passages: list refused: bool refusal reason: str | None = None confidence: float | None = None def respond question: str - RespondResult: """ The full "zero hallucinations" read path: embed - hybrid search dense + sparse - RRF fusion - cross-encoder rerank - confidence threshold - LLM generate - claim extraction - NLI fact-check - final answer OR refusal. Every stage is a chance to refuse rather than guess -- that's the whole point of the "must not assert unsupported claims" guarantee. """ ------------------------------------------------------------------- Step 1: Embed the question turn text into a dense vector . ------------------------------------------------------------------- query vector = embed question ------------------------------------------------------------------- Step 2: Hybrid search -- run dense vector and sparse BM25 search CONCURRENTLY this is the "parallel tool calls" idea from Section 3 applied to retrieval itself , each returning their own top-40 shortlist. ------------------------------------------------------------------- dense results = qdrant ann search query vector, top k=40 sparse results = elasticsearch bm25 question, top k=40 ------------------------------------------------------------------- Step 3: Fuse both ranked lists with Reciprocal Rank Fusion Section 4 . ------------------------------------------------------------------- candidates = rrf fusion dense results, sparse results ------------------------------------------------------------------- Step 4: Rerank the fused candidates with a cross-encoder Section 6 for much higher-precision final ordering. Cross-encoders are too slow to run on millions of docs, but perfectly fine on ~40-80 candidates. ------------------------------------------------------------------- ranked = cross encoder rerank question, candidates, top k=40 ------------------------------------------------------------------- Step 5: CONFIDENCE GATE 1 -- if even the BEST passage scores too low, we don't have real evidence to answer with. Refuse rather than guess. ------------------------------------------------------------------- if not any passage.score 0.4 for passage in ranked : return refused response "low retrieval confidence" ------------------------------------------------------------------- Step 6: Take only the top 5 passages -- this keeps the LLM's context small, focused, and less prone to the "lost in the middle" problem Section 6 that hurts long, noisy contexts. ------------------------------------------------------------------- top passages = ranked :5 prompt = build prompt question, top passages ------------------------------------------------------------------- Step 7: Ask the LLM to generate an answer, GROUNDED ONLY in top passages. temperature=0 for maximum determinism/factuality no creative sampling . ------------------------------------------------------------------- raw answer = llm generate prompt, temperature=0 ------------------------------------------------------------------- Step 8: CONFIDENCE GATE 2 -- the prompt instructs the LLM to say "INSUFFICIENT CONTEXT" if the retrieved passages don't actually answer the question. If it says so, we honor that and refuse cleanly. ------------------------------------------------------------------- if raw answer.startswith "INSUFFICIENT CONTEXT" : return refused response "no relevant passages" ------------------------------------------------------------------- Step 9: CLAIM EXTRACTION + FACT-CHECKING -- this is the actual "zero hallucinations" enforcement mechanism. Break the raw answer into small, individually-checkable "atomic claims" e.g. one claim per sentence or per fact , then verify EACH claim is actually entailed logically supported by the retrieved passages using an NLI Natural Language Inference model -- a plain-English one-liner: "a model that decides whether a piece of text logically follows from, contradicts, or is unrelated to another piece of text." If ANY claim can't be verified, we refuse rather than let an unsupported claim slip through to the user. ------------------------------------------------------------------- claims = extract atomic claims raw answer for claim in claims: if not nli entailed claim, top passages : return refused response "unverified claim" ------------------------------------------------------------------- All gates passed: the answer is grounded, confident, and every claim has been individually verified against the retrieved evidence. ------------------------------------------------------------------- return RespondResult answer=raw answer, passages=top passages, refused=False def refused response reason: str - RespondResult: """A first-class, explicit refusal -- NOT a vague hedge buried in text.""" return RespondResult answer="", passages= , refused=True, refusal reason=reason NOTE: embed , qdrant ann search , elasticsearch bm25 , rrf fusion , cross encoder rerank , build prompt , llm generate , extract atomic claims , and nli entailed are each real functions you'd implement using the techniques from Sections 4, 6, and standard NLI models e.g. a model fine-tuned on MNLI/SNLI for entailment classification -- they are stubbed here to keep the read-path LOGIC the star of this example. The write path S3 → chunk ingestor → hierarchical chunking → embeddings → storage needs a chunker. The notes specify sentence-level boundaries using spaCy plus 32-token overlap between consecutive chunks overlap ensures a sentence that would otherwise get awkwardly split across two chunks still has enough surrounding context in at least one of them . pip install spacy --break-system-packages python -m spacy download en core web sm import spacy nlp = spacy.load "en core web sm" def chunk document text: str, max tokens per chunk: int = 256, overlap tokens: int = 32 - list dict : """ Hierarchical, sentence-boundary-respecting chunking with token overlap. Why sentence boundaries: splitting mid-sentence produces chunks that are semantically broken and embed poorly. Why overlap: a fact stated right at a chunk boundary shouldn't lose its surrounding context in EVERY chunk that references it. """ doc = nlp text sentences = sent.text.strip for sent in doc.sents chunks = current chunk sentences: list str = current token count = 0 char cursor = 0 def rough token count s: str - int: A simple approximation real systems use a proper tokenizer, e.g. tiktoken, matching whatever embedding model will consume the chunk . return len s.split for sentence in sentences: sentence tokens = rough token count sentence If adding this sentence would overflow the chunk, close the current chunk out and start a new one, carrying over the last overlap tokens worth of sentences for continuity. if current token count + sentence tokens max tokens per chunk and current chunk sentences: chunk text = " ".join current chunk sentences chunks.append { "text": chunk text, "token count": current token count, "char start": char cursor, "char end": char cursor + len chunk text , } char cursor += len chunk text Build the overlap: keep trailing sentences until we've got roughly overlap tokens worth, to seed the next chunk. overlap sentences, overlap count = , 0 for sent in reversed current chunk sentences : overlap sentences.insert 0, sent overlap count += rough token count sent if overlap count = overlap tokens: break current chunk sentences = overlap sentences current token count = overlap count current chunk sentences.append sentence current token count += sentence tokens Flush the final chunk if current chunk sentences: chunk text = " ".join current chunk sentences chunks.append { "text": chunk text, "token count": current token count, "char start": char cursor, "char end": char cursor + len chunk text , } return chunks --- Example --------------------------------------------------------------- sample doc = "Throughput degradation under sustained load is a common issue. " "It is typically caused by connection pool exhaustion. " "Another frequent cause is unindexed database queries. " "Monitoring dashboards should track p50, p95, and p99 latency. " "Alerts should fire when p99 crosses a defined SLO threshold." for i, chunk in enumerate chunk document sample doc, max tokens per chunk=15, overlap tokens=5 : print f"Chunk {i}: {chunk 'text' }" Each resulting chunk a "passage" in the notes' schema would then get an embedding computed using one of the models the notes list — microsoft/hamier , Qwen3 8B , BGE-M3 , or an OpenAI/OSS embedding API , stored in Qdrant the vector , Postgres the metadata: doc id, chunk index, char start, char end, token count , and ElasticSearch the BM25 inverted index . Flow: S3 raw doc storage → Chunk Ingestor → Hierarchical Chunking → Embeddings → Storage Postgres, Qdrant, ElasticSearch, Redis . Step by step: a document lands in S3. A chunk ingestor process picks it up typically via an event notification or polling , performs three sub-steps the notes list explicitly list files, download, read , then runs hierarchical chunking Section 9.5's sentence-boundary + overlap logic to split it into passages. Each passage is embedded, and the results are fanned out to storage: Postgres gets the passage/doc metadata, Qdrant gets the embedding vectors, and ElasticSearch gets the text for its BM25 inverted index. This asynchronous pipeline is exactly why the system can tolerate the ~15-minute eventual consistency SLO mentioned earlier — the write path doesn't need to be instantaneous, it needs to be reliable and complete within a bounded, acceptable delay. The notes also flag a timebox mechanism on the read side with two named fallback behaviors — cascade and fallback — meaning if some part of the read path takes too long e.g., the cross-encoder rerank step or the LLM generation call, described as "the long pole" of end-to-end latency , the system has a time budget after which it cascades to a simpler/cheaper strategy or falls back to a safe default like a refusal rather than hanging indefinitely. This connects directly back to the query latency numbers given in the notes: p50 = 1.5 sec, p99 = 5 sec , with the LLM inference call explicitly called out as the long pole the slowest, bottleneck stage — largely because the very first token the user sees their Time To First Token , covered next in the Appendix is gated on that call finishing its reasoning. Think about the difference between getting a text message that appears all at once after a long pause, versus watching someone type it live, word by word, right in front of you. Even though the total message might take the same amount of time to fully arrive, watching it appear progressively feels far less frustrating — you get immediate feedback that something is happening, instead of staring at a blank screen wondering if anything is working at all. That's the entire motivation for streaming: even if the total response time is identical, showing partial output as it's generated dramatically improves perceived responsiveness. The most important metric here, per the notes: TTFT — Time To First Token a plain-English one-liner: "how long the user waits before they see ANY output at all, even if the full response isn't done yet" . If a tool call takes real time to execute a slow database query, a slow external API , and the user has to wait through the entire tool call before seeing anything, that's flagged directly in the notes as bad UX — "like everything else," slow, silent waiting erodes trust in the product. The tricky part specific to streaming tool calls : "what if tool input arrives as partial JSON fragments?" When a model streams its response token by token, and part of that response is a tool use block e.g., building up {"city": "Tok ... yo"} character by character , you cannot execute the tool with a half-formed, syntactically-invalid JSON blob. The notes' answer: "wait until the entire input is available" — i.e., keep buffering accumulating the partial JSON text as it streams in, and only attempt to parse/execute the tool call once the model signals that this particular content block is fully complete. The notes include a cautionary, very concrete illustration of why this matters: imagine the model is streaming a dangerous tool call like "delete that record..." If your system tried to act on a half-baked, incomplete tool input say, a delete call where the id argument hasn't fully arrived yet , it could either error out unpredictably or — worse — "no input could be fired" i.e., act on a malformed/empty argument. Half-baked input / tool calls with no input could be fired is exactly the failure mode the buffering strategy exists to prevent. python import anthropic import json client = anthropic.Anthropic The tool definition is identical in shape to Section 1 -- streaming doesn't change WHAT a tool looks like, only HOW the response arrives. tools = { "name": "get weather", "description": "Get current weather for a location", "input schema": { "type": "object", "properties": { "location": {"type": "string"}, "unit": {"type": "string", "enum": "celsius", "fahrenheit" }, }, "required": "location" , }, } def execute tool tool name: str, tool input: dict - dict: """Stand-in for a real tool implementation see Section 1 for a fuller mock .""" if tool name == "get weather": return {"location": tool input "location" , "temp": 24, "condition": "sunny"} return {"error": "unknown tool"} with client.messages.stream model="claude-sonnet-4-6", max tokens=1024, tools=tools, messages= {"role": "user", "content": "What is the weather in Tokyo?"} , as stream: These two variables ACCUMULATE state across streamed events -- this IS the "buffering" the notes insist on. We never act on partial state. current tool input = "" current tool name = None for event in stream: ------------------------------------------------------------- "content block start": a NEW block either plain text or a tool use block has begun streaming. If it's a tool use block, note its NAME immediately, but its INPUT starts empty -- arguments arrive incrementally via later delta events. ------------------------------------------------------------- if event.type == "content block start": if event.content block.type == "tool use": current tool name = event.content block.name current tool input = "" reset the buffer for this new block ------------------------------------------------------------- "content block delta": an incremental CHUNK of the current block has arrived. For tool use blocks, this is a fragment of partial JSON event.delta.type == "input json delta" -- we must ACCUMULATE it, not parse it yet, since it's incomplete. For plain text blocks, we can print immediately since partial text is still readable this is what gives the "word by word" streaming feel for normal chat responses . ------------------------------------------------------------- elif event.type == "content block delta": if event.delta.type == "input json delta": current tool input += event.delta.partial json <- Accumulate elif event.delta.type == "text delta": print event.delta.text, end="", flush=True ------------------------------------------------------------- "content block stop": the CURRENT block text or tool use has finished streaming completely. If it was a tool use block, the buffered JSON string is now guaranteed complete and valid -- THIS is the safe moment to parse it and actually execute the tool, never before. ------------------------------------------------------------- elif event.type == "content block stop": if current tool name: parsed input = json.loads current tool input <- input complete result = execute tool current tool name, parsed input <- execute print f"\n Executed {current tool name} - {result} " current tool name = None reset for the next potential tool block The three event types to remember: content block start a new block began — note its type/name , content block delta content block stop Context Precision, simple example: think of a search-results page with 5 blue links. Some of them are genuinely useful and directly relevant to what you searched for imagine green dots next to them ; others are irrelevant filler or noise red dots . A good search engine doesn't just include some relevant results somewhere on the page — it puts them near the top . A search engine that buries the one truly relevant result at position 5, behind four irrelevant ones, is worse than one that puts it at position 1, even if both technically "found" it somewhere on the page. Faithfulness, simple example: imagine grading a student's essay by checking, sentence by sentence, whether each individual claim they made is actually backed up by the assigned textbook. If the student writes five sentences and four of them are directly supported by the textbook but the fifth one is something they simply made up that isn't in the textbook at all , you wouldn't give them full credit — you'd score the essay based on the fraction of sentences that are actually grounded in the source material. That's exactly what "faithfulness" measures for an LLM's generated answer. The motivation, per the notes: naive RAG evaluation is just qualitative — "does it seem right?" — and manual feedback loops are slow. That doesn't scale, and it's subjective. RAGAS a plain-English one-liner: "a framework and library for evaluating RAG pipelines using automated, LLM-assisted metrics instead of manual human judgment" fixes two specific pain points named in the notes: no need for human-annotated ground truth , and it lets you verify that changes/optimizations to your pipeline don't quietly downgrade quality a regression-testing mindset applied to RAG quality, not just code correctness . Context Precision — measures the retriever's or reranker's ability to rank relevant chunks higher , and it directly penalizes systems that bury relevant content below noise — this is precisely the "lost in the middle" problem flagged back in Section 6. The method: use an LLM to label each retrieved chunk as RELEVANT or NOT relevant , and then, for every position k in the ranked list, compute precision@k — the fraction of the top k results that are relevant. The notes' worked example , using a 5-result list marked green = relevant, red = not relevant : 🟢 🔴 🟢 🔴 🔴 p@2 = 0 . Reading the notes' own math carefully: Final Context Precision formula from the notes' own arithmetic : Context Precision = p@1 + p@3 / relevant chunks = 1 + 0.67 / 2 = 1.67 / 2 = 0.835 In words: you sum up the precision@k values at each rank where a relevant chunk appears , then divide by the total number of relevant chunks in the list here, 2 relevant chunks out of 5 total . This rewards systems that put their relevant chunks early — a relevant chunk found at rank 1 contributes a full 1.0, while one found deeper in the list contributes a smaller fraction, exactly reflecting the "don't bury the good stuff" intuition from the simple example. Faithfulness Score — measures factual consistency of the generated answer , and is explicitly called a "measure of hallucination" in the notes. Method: break the answer into atomic claims using an LLM — the same "claim extraction" step from the read-path pseudocode in Section 9.4 , then for each claim, score it 1 if the retrieved context supports the claim , or 0 if the context does not support the claim . Faithfulness formula: faithfulness = claims supported / claims This is the exact same underlying idea as the nli entailed check baked into the "zero hallucinations" respond function from Section 9 — faithfulness is essentially the evaluation-time metric version of the same runtime guardrail. ============================================================================= CONTEXT PRECISION -- from a list of relevant/irrelevant flags the "green dot / red dot" example from the notes . ============================================================================= def context precision relevance flags: list bool - float: """ relevance flags: e.g. True, False, True, False, False -- True = relevant chunk, False = not relevant, in the ORDER they were retrieved/ranked best first . Returns the RAGAS-style Context Precision score. """ total relevant = sum relevance flags if total relevant == 0: return 0.0 no relevant chunks at all -- precision is undefined/zero running relevant count = 0 precision sum = 0.0 for k, is relevant in enumerate relevance flags, start=1 : if is relevant: running relevant count += 1 precision@k = relevant chunks in top k / k precision at k = running relevant count / k precision sum += precision at k if NOT relevant, this rank contributes 0 and is skipped, matching the notes' worked example where p@2, p@4, p@5 = 0 return precision sum / total relevant --- Example matching the notes exactly: 🟢 🔴 🟢 🔴 🔴 ---------------------- flags = True, False, True, False, False score = context precision flags print f"Context Precision: {score:.3f}" - should print 0.835 ============================================================================= FAITHFULNESS -- given a list of claims and whether each is supported. ============================================================================= def faithfulness score claim support labels: list bool - float: """ claim support labels: e.g. True, True, True, False -- True = context supports this atomic claim, False = context does NOT support it hallucinated . """ if not claim support labels: return 1.0 no claims made -- vacuously faithful nothing to check supported = sum claim support labels total = len claim support labels return supported / total --- Example: an answer with 4 atomic claims, 1 of which is unsupported ---- claims supported = True, True, True, False faith score = faithfulness score claims supported print f"Faithfulness: {faith score:.3f}" - 0.75 python pip install ragas datasets --break-system-packages from ragas import evaluate from ragas.metrics import context precision, faithfulness from datasets import Dataset RAGAS expects a dataset shaped like this: one row per question, with the retrieved contexts and the generated answer already filled in. eval dataset = Dataset.from dict { "question": "Why is my API returning a 500 error under high load?" , "answer": "500 errors under high load are typically caused by connection " "pool exhaustion or unhandled exceptions from resource contention." , "contexts": "Throughput degradation under sustained load can trigger unhandled " "exceptions, which surface to callers as HTTP 500 errors.", "Connection pool exhaustion is a common cause of request failures " "under high concurrency.", , ground truth is optional for some metrics but required for others e.g. recall "ground truth": "500 errors under load are usually caused by connection " "pool exhaustion or unhandled exceptions." , } results = evaluate eval dataset, metrics= context precision, faithfulness print results This runs the SAME conceptual computation as the from-scratch functions above, but uses an LLM internally to a judge chunk relevance for context precision, and b extract + verify atomic claims for faithfulness.