{"slug": "tool-use-and-rag-in-production-a-complete-beginner-s-study-guide", "title": "Tool Use and RAG in Production — A Complete Beginner's Study Guide", "summary": "A developer published a beginner's study guide explaining tool use and retrieval-augmented generation (RAG) for large language models in production. The guide uses real-world analogies and runnable code to demonstrate how models delegate external tasks like weather lookups to deterministic functions. It emphasizes that the model only reasons and constructs arguments, while the application executes the actual function calls.", "body_md": "This guide assumes you know nothing about LLMs beyond \"I can chat with ChatGPT/Claude.\" Every concept is introduced with a real-world analogy first, then explained technically, then backed by runnable code.\n\nImagine you ask a friend, \"Hey, what's the weather in Tokyo right now?\" Your friend doesn't *know* this off the top of their head — nobody just knows live weather. So your friend pulls out their phone, opens a weather app, types \"Tokyo,\" reads the result, and tells you: \"It's 24°C and cloudy.\"\n\nNotice what happened: your friend (the \"brain\" doing the reasoning) never *became* a weather sensor. They recognized they needed external, live information, used a *tool* (the weather app) to fetch it, and then used that fetched information to answer you in natural language.\n\nThis is exactly what \"tool use\" (also called \"function calling\") means for an LLM like Claude or GPT. The model is the friend. It's great at reasoning and language, but it has no live connection to today's weather, your company's database, or a calculator that never makes arithmetic mistakes. So we give it a list of \"apps\" (tools) it's allowed to ask for, and something else — your application code — actually goes and uses those apps on its behalf.\n\n**Tool use** extends a model's capabilities to call \"external systems\" — anything outside the model's own weights and training data. Examples from the notes:\n\nThese are all things a language model is fundamentally bad at doing internally (it can't refresh live weather data, and it's notoriously unreliable at exact arithmetic), so instead of asking the model to guess, we let it delegate to a real, deterministic function.\n\n**How it's wired up:**\n\n`get_weather`\n\nwith `{\"city\": \"Tokyo\"}`\n\n.\"**The single most important sentence in this whole section, straight from the notes:**\n\n\"The model never calls a function itself!\"\n\nThe model's role is *reasoning and argument construction only*. It cannot reach out to the internet, touch your database, or execute code. Everything it does is emit text/JSON. Your application is the one with hands.\n\n**The full lifecycle (the loop), step by step:**\n\n`get_weather(\"Tokyo\")`\n\nfor real — hits a weather API, database, calculator, whatever the tool represents.Along the way your application also has to handle practical realities like **rate limits** on the tools you're calling (e.g., a weather API only allows so many requests per minute) — that's your application's job too, not the model's.\n\nBelow is a minimal, close-to-runnable example using the Anthropic SDK style referenced in the notes. It defines a `get_weather`\n\ntool, sends a prompt, receives a `tool_use`\n\nblock, executes a **mock** function (standing in for a real weather API), and sends the result back so the model can finish its answer.\n\n``` python\n# pip install anthropic --break-system-packages\nimport anthropic\nimport json\n\n# Step 0: create a client (assumes ANTHROPIC_API_KEY is set as an environment variable)\nclient = anthropic.Anthropic()\n\n# ---------------------------------------------------------------------------\n# Step 1: Define the tool as \"a function + metadata\"\n# This is NOT the real function. It's a JSON description that tells the model\n# \"this function exists, here's its name, what it does, and what arguments it needs.\"\n# ---------------------------------------------------------------------------\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get the current weather for a given city.\",\n        \"input_schema\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"city\": {\"type\": \"string\", \"description\": \"City name, e.g. Tokyo\"},\n                \"unit\": {\n                    \"type\": \"string\",\n                    \"enum\": [\"celsius\", \"fahrenheit\"],\n                    \"description\": \"Temperature unit to return\",\n                },\n            },\n            \"required\": [\"city\"],  # unit is optional\n        },\n    }\n]\n\n# ---------------------------------------------------------------------------\n# Step 2: This is the REAL function that will actually run.\n# In production this would call a weather API (e.g. OpenWeatherMap).\n# Here we mock it so the example is runnable without external dependencies.\n# ---------------------------------------------------------------------------\ndef get_weather(city: str, unit: str = \"celsius\") -> dict:\n    \"\"\"Pretend weather lookup — replace with a real HTTP call in production.\"\"\"\n    fake_database = {\"Tokyo\": 24, \"London\": 14, \"Delhi\": 33}\n    temp_c = fake_database.get(city, 20)\n    temp = temp_c if unit == \"celsius\" else (temp_c * 9 / 5) + 32\n    return {\"city\": city, \"temperature\": temp, \"unit\": unit}\n\n# ---------------------------------------------------------------------------\n# Step 3: Send the user's question + the tool list to the model.\n# The model will read the question and DECIDE if it needs the tool.\n# ---------------------------------------------------------------------------\nmessages = [{\"role\": \"user\", \"content\": \"What is the weather in Tokyo?\"}]\n\nresponse = client.messages.create(\n    model=\"claude-sonnet-4-6\",\n    max_tokens=1024,\n    tools=tools,\n    messages=messages,\n)\n\n# ---------------------------------------------------------------------------\n# Step 4: \"Application intercepts the response.\"\n# We look through the model's response blocks. If it contains a \"tool_use\"\n# block, that means the model wants us to run a function on its behalf.\n# The model itself did NOT run anything — it only asked.\n# ---------------------------------------------------------------------------\ntool_use_block = None\nfor block in response.content:\n    if block.type == \"tool_use\":\n        tool_use_block = block\n        break\n\nif tool_use_block:\n    print(f\"Model wants to call: {tool_use_block.name} with {tool_use_block.input}\")\n\n    # Step 5: \"Application invokes the function.\" We actually run the real code.\n    if tool_use_block.name == \"get_weather\":\n        result = get_weather(**tool_use_block.input)\n    else:\n        result = {\"error\": \"unknown tool\"}\n\n    # Step 6: \"Application sends the result back to model\" as a tool_result message.\n    # We must echo back the ORIGINAL assistant message (containing the tool_use)\n    # plus a new \"user\" message containing the tool_result, so the model has\n    # full context of what it asked for and what it got back.\n    messages.append({\"role\": \"assistant\", \"content\": response.content})\n    messages.append(\n        {\n            \"role\": \"user\",\n            \"content\": [\n                {\n                    \"type\": \"tool_result\",\n                    \"tool_use_id\": tool_use_block.id,\n                    \"content\": json.dumps(result),\n                }\n            ],\n        }\n    )\n\n    # Step 7: \"Model proceeds...\" — call the model again so it can turn the\n    # raw tool result into a natural-language final answer.\n    final_response = client.messages.create(\n        model=\"claude-sonnet-4-6\",\n        max_tokens=1024,\n        tools=tools,\n        messages=messages,\n    )\n    for block in final_response.content:\n        if block.type == \"text\":\n            print(\"Final answer:\", block.text)\nelse:\n    # The model answered directly without needing a tool.\n    for block in response.content:\n        if block.type == \"text\":\n            print(\"Final answer:\", block.text)\n```\n\nNotice the loop shape: **model decides → app intercepts → app invokes → app returns result → model proceeds.** That five-step handoff is the entire mental model for tool use, and it never changes no matter how complex the tool is.\n\nImagine you're hiring a new employee and you hand them a one-line job description: \"Search for products.\" That's it — no other instructions.\n\nOn day one, a customer asks them \"how much does this cost?\" and the new employee, trying to be helpful, uses the \"search for products\" tool because it's the only tool they were told about, even though there's actually a *dedicated* pricing tool sitting right next to it that they didn't know they should prefer. They also aren't told whether \"search\" means searching by name, by category, or by a product code — so they guess, and sometimes guess wrong.\n\nNow imagine instead you hand them a proper SOP: \"Search for products by name, category, or SKU. Use this only when the customer is explicitly trying to find or filter catalog items. Do NOT use this for pricing questions or stock checks — those have their own dedicated tools.\" Now the employee has almost no room to misinterpret their job.\n\nThis is the difference between a bad tool schema and a good one. The LLM is that new employee, and the schema is the only \"training\" it gets before being thrown into the job.\n\nFrom the notes: **\"JSON schema is [the] contract between your intent and the model's behaviour.\"** And critically: **\"Ambiguous schema is [the] most common root cause of tool failure and is 'invisible' until production.\"** It's invisible because in your own testing you tend to ask clean, obvious questions — production users ask messy, ambiguous, unexpected ones, and that's where a vague schema quietly breaks.\n\nThere are **three rules** for designing a good tool schema:\n\n**Rule 1 — Specify semantics, not just a description.**\n\nDon't just say what the function is named or roughly does. Say *when to use it*, *when not to use it*, and *what it should never be confused with*. The notes' worked example:\n\n`\"search for products\"`\n\n`\"Search for products by name, category, or SKU. Use only when user is explicitly looking to find or filter catalog items. Do not use for pricing questions or stock checks — those have other dedicated tools.\"`\n\n**Rule 2 — Be deliberate about required vs. optional parameters.**\n\nIf a parameter is optional and you don't clearly explain what happens when it's omitted, the model may **hallucinate a value** for it just to fill the slot (marked with a red ✗ in the notes). Conversely, if a parameter is marked mandatory but the model doesn't have a value for it, it may pass nothing at all or an invalid placeholder (also marked ✗). The fix from the notes: **provide default values for the model to use** so there's no ambiguity about what an \"empty\" optional parameter should look like.\n\n**Rule 3 — Enums are the cheapest reliability primitive you have.**\n\nWhenever a parameter has a fixed, known set of valid values (a category, a unit, a status), constrain it with an **enum** instead of leaving it as a free-form string. This \"keeps things readable and reduces hallucination\" — the model can only pick from the list you gave it, it can't invent `\"CategoryXYZ\"`\n\nout of thin air.\n\n**The \"too many tools\" problem, and its fix — dynamic tool loading.**\n\nThe notes flag: *\"large number of tools: model gets confused\"* — when a model is handed 50+ tools at once, two hard questions creep in: *\"What category tool?\"* and *\"Which specific tool?\"* The model starts guessing between similar-looking tools. The fix is **dynamic tool loading**: instead of always sending the model your entire tool catalog, your application first figures out (from context, intent classification, or the conversation's domain) which *subset* of tools is actually relevant, and only sends that subset to the model for this particular turn. Fewer, more relevant choices means fewer wrong guesses.\n\n```\n# =============================================================================\n# BAD SCHEMA\n# Vague description, no enum for category, ambiguous optional parameter.\n# =============================================================================\nbad_tool = {\n    \"name\": \"search_products\",\n    \"description\": \"Search for products\",  # <-- too vague: search how? for what purpose?\n    \"input_schema\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"query\": {\"type\": \"string\"},\n            \"category\": {\"type\": \"string\"},  # <-- free text: model may invent categories\n            \"include_out_of_stock\": {\"type\": \"boolean\"},  # <-- optional, no default explained\n        },\n        \"required\": [\"query\"],\n    },\n}\n# Problems this causes in production:\n# 1. Model may call this tool for pricing questions, since nothing tells it not to.\n# 2. Model may pass category=\"Electronics & Gadgets\" when your real categories\n#    are [\"electronics\", \"apparel\", \"home\"] -> zero results, silent failure.\n# 3. Model may omit include_out_of_stock, or guess True/False randomly, since\n#    there's no guidance on what happens if it's left out.\n\n# =============================================================================\n# GOOD SCHEMA\n# Specifies semantics (when to use / not use), enum for category,\n# explicit default behaviour for optional params.\n# =============================================================================\ngood_tool = {\n    \"name\": \"search_products\",\n    \"description\": (\n        \"Search for products by name, category, or SKU. \"\n        \"Use only when the user is explicitly looking to find or filter catalog \"\n        \"items. Do NOT use this for pricing questions or stock-level checks — \"\n        \"those have their own dedicated tools (get_price, check_stock).\"\n    ),\n    \"input_schema\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"query\": {\n                \"type\": \"string\",\n                \"description\": \"Free-text search term, e.g. a product name or SKU.\",\n            },\n            \"category\": {\n                \"type\": \"string\",\n                # Enum = the model can ONLY pick from these exact values.\n                \"enum\": [\"electronics\", \"apparel\", \"home\", \"books\", \"toys\"],\n                \"description\": \"Optional. Restrict results to one catalog category.\",\n            },\n            \"include_out_of_stock\": {\n                \"type\": \"boolean\",\n                \"description\": (\n                    \"Whether to include out-of-stock items. \"\n                    \"If omitted, DEFAULTS to false (only show in-stock items).\"\n                ),\n            },\n        },\n        \"required\": [\"query\"],\n    },\n}\n\n# The good schema tells the model: what it's for, what it's NOT for, exactly\n# which category strings are valid, and exactly what \"not specifying\" means.\n# That triple clarity is what the notes call the \"contract\" between your\n# intent and the model's behaviour.\n# A tiny illustration of picking a relevant SUBSET of tools before calling\n# the model, instead of always sending your entire tool catalog.\n\nALL_TOOLS = {\n    \"search_products\": {...},   # schema dicts omitted for brevity\n    \"get_price\": {...},\n    \"check_stock\": {...},\n    \"get_weather\": {...},\n    \"run_bash_command\": {...},\n    \"search_hr_docs\": {...},\n}\n\n# A simple keyword-based intent router. In production this is often a small\n# classifier model or an embedding-similarity lookup, not just keywords —\n# but the principle (narrow the toolset BEFORE calling the main model) is identical.\ndef select_relevant_tools(user_message: str) -> list[str]:\n    msg = user_message.lower()\n    if any(word in msg for word in [\"price\", \"cost\", \"how much\"]):\n        return [\"get_price\"]\n    if any(word in msg for word in [\"stock\", \"available\", \"in stock\"]):\n        return [\"check_stock\"]\n    if any(word in msg for word in [\"find\", \"search\", \"looking for\"]):\n        return [\"search_products\"]\n    if \"weather\" in msg:\n        return [\"get_weather\"]\n    # Fallback: give a small default set rather than everything\n    return [\"search_products\", \"get_price\"]\n\ndef build_tools_for_request(user_message: str) -> list[dict]:\n    relevant_names = select_relevant_tools(user_message)\n    return [ALL_TOOLS[name] for name in relevant_names if name in ALL_TOOLS]\n\n# Usage: only the relevant tools go into the API call, not all 6+.\n# tools_for_this_call = build_tools_for_request(\"How much does the red mug cost?\")\n# -> only sends the get_price tool, so the model can't confuse it with search_products.\n```\n\nSuppose you ask a travel-planning friend: \"Can you check the weather in Paris, Tokyo, and New York for me?\" A slow friend would check Paris, wait, tell you, then check Tokyo, wait, tell you, then check New York. A smart friend instead opens three browser tabs at once, checks all three simultaneously, and gives you all three answers together. Same total work, far less waiting.\n\nNow imagine one of those three lookups fails — say the New York weather site is down. Does your friend refuse to tell you about Paris and Tokyo just because New York failed? Or do they hand you what they *do* have, and mention New York didn't work? That's a judgment call — and it's the same judgment call your application has to make about tool calls.\n\n**Parallel tool calls:** the notes describe a model that can emit **multiple tool-call requests in a single turn** — e.g., \"compare weather across N cities\" or a \"trip plan\" that needs both `search_flight`\n\nand `search_hotel`\n\nat once. Your **orchestration layer** (the application code, not the model) is responsible for \"fanning out\" these calls **concurrently**, collecting all the results, and sending them back together. The direct payoff: it **improves overall time to completion** — you pay for the *slowest* of the N calls, not the *sum* of all N. The notes' guidance: **use this whenever the operations are independent of one another** (one call's result doesn't depend on another's).\n\n**Failure handling — \"it depends\".** The notes are refreshingly honest here: *\"Answer to any question could well be IT DEPENDS.\"* There isn't one universal right answer; it's a judgment call based on your product's tolerance for incompleteness vs. correctness. Three named strategies:\n\nThe notes give a concrete example of when parallel fan-out matters: *\"4 websearch calls, lookup across multiple knowledge bases.\"* If a question needs facts from 4 different knowledge bases, firing all 4 lookups concurrently instead of one after another is the entire difference between a fast, responsive system and a sluggish one.\n\n``` python\nimport asyncio\nimport random\n\n# ---------------------------------------------------------------------------\n# Mock tool: fetch weather for one city. Sometimes fails, to demonstrate\n# failure handling (simulating a flaky downstream API).\n# ---------------------------------------------------------------------------\nasync def fetch_weather(city: str) -> dict:\n    await asyncio.sleep(random.uniform(0.1, 0.5))  # simulate network latency\n    if city == \"New York\" and random.random() < 0.5:\n        raise ConnectionError(f\"Weather API timed out for {city}\")\n    return {\"city\": city, \"temp_c\": random.randint(10, 35)}\n\n# =============================================================================\n# STRATEGY 1: PREVENT\n# Validate/guard BEFORE attempting the call, so we never even try a call\n# we already know is unsafe (e.g. an unsupported city, a rate-limited tool).\n# =============================================================================\nSUPPORTED_CITIES = {\"Paris\", \"Tokyo\", \"New York\", \"London\"}\n\nasync def call_with_prevention(city: str) -> dict:\n    if city not in SUPPORTED_CITIES:\n        # We \"prevent\" the failure by never calling the API for a city we\n        # know isn't supported, instead of letting it fail downstream.\n        return {\"city\": city, \"error\": \"unsupported_city\", \"skipped_call\": True}\n    return await fetch_weather(city)\n\n# =============================================================================\n# STRATEGY 2: ABSORB\n# Run all calls concurrently; if one fails, catch the exception, keep going,\n# and return only the successful results (plus a note about what failed).\n# =============================================================================\nasync def call_with_absorption(cities: list[str]) -> dict:\n    tasks = [fetch_weather(city) for city in cities]\n    # return_exceptions=True means a failed task returns its Exception object\n    # instead of crashing the whole asyncio.gather() call.\n    results = await asyncio.gather(*tasks, return_exceptions=True)\n\n    successes, failures = [], []\n    for city, result in zip(cities, results):\n        if isinstance(result, Exception):\n            failures.append({\"city\": city, \"error\": str(result)})\n        else:\n            successes.append(result)\n\n    return {\"successful_results\": successes, \"failed_cities\": failures}\n\n# =============================================================================\n# STRATEGY 3: FAIL GRACEFULLY\n# If ANY call fails and a partial answer would be misleading (e.g. this tool\n# call was mandatory context for the user's question), abort entirely and\n# surface a clear, honest message instead of a silently incomplete answer.\n# =============================================================================\nasync def call_with_graceful_failure(cities: list[str]) -> dict:\n    tasks = [fetch_weather(city) for city in cities]\n    try:\n        results = await asyncio.gather(*tasks)  # no return_exceptions: first failure raises\n        return {\"ok\": True, \"results\": results}\n    except Exception as exc:\n        return {\n            \"ok\": False,\n            \"refused\": True,\n            \"reason\": f\"Could not complete weather comparison: {exc}\",\n        }\n\nasync def main():\n    cities = [\"Paris\", \"Tokyo\", \"New York\"]\n\n    print(\"PREVENT strategy (one unsupported city):\")\n    print(await asyncio.gather(*[call_with_prevention(c) for c in [\"Paris\", \"Atlantis\"]]))\n\n    print(\"\\nABSORB strategy (best-effort, partial results ok):\")\n    print(await call_with_absorption(cities))\n\n    print(\"\\nFAIL GRACEFULLY strategy (all-or-nothing):\")\n    print(await call_with_graceful_failure(cities))\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nWhich strategy to pick is a product decision, not a technical one — that's the \"it depends\" in the notes. A weather-comparison feature can tolerate \"absorb\" (2 out of 3 cities is still useful). A financial transaction pipeline probably needs \"fail gracefully\" (a half-completed transfer is dangerous, not just incomplete).\n\nSay you're searching your email for something. If you search \"birthday party invite,\" a smart search that understands *meaning* (semantic search) will find an email titled \"Come celebrate turning 30!\" even though none of your exact words appear in it — because it understands the *concept* of a birthday party.\n\nBut now say you search for an exact error code, like `1099-MISC`\n\n. A meaning-based search might get confused and return generic tax documents about \"how do I reset my password\" style topics — it doesn't realize you need an *exact string match*, not a conceptual one. A classic old-school keyword search (like Ctrl+F, but smarter) would nail this instantly because it's built for exact-term matching.\n\nNeither approach alone is good at everything. The fix: run **both kinds of search at the same time**, and combine (merge and rerank) their results — a bit like asking two friends with different strengths for restaurant recommendations, and combining their two rankings into one better, more trustworthy ranking.\n\n**Dense retrieval** (semantic search): your query and your documents are both converted into vectors (\"embeddings\") that capture *meaning*. Two pieces of text that mean similar things end up as vectors that are close together in space, even if they don't share any exact words. This is great when there's genuine **semantic similarity** — the notes' example: \"How do I reset my password?\" naturally matches a document titled \"Account Recovery Steps,\" even with zero overlapping words.\n\n**Sparse retrieval / BM25**: this is the notes' name for **traditional keyword-based search** (\"BM25\" = \"Best Matching 25,\" a well-known statistical formula scoring how well a document's exact terms match a query's exact terms, weighted by term rarity and frequency). BM25 is the *opposite* strength of dense search: it's excellent at exact terms, codes, IDs, acronyms — things like \"error code 1099-MISC\" — but it has no concept of meaning, so it's poor at \"How do I reset my password?\" style natural-language questions where the right document doesn't share your exact wording.\n\n**Hybrid search = dense + sparse, run together.** The notes' rule of thumb: *\"20|50 documents certainly having the right answer\"* — hybrid retrieval **improves recall** (the fraction of genuinely relevant documents that get retrieved at all), because you're covering both failure modes (semantic gap AND exact-term gap) simultaneously. The fix line from the notes: **\"Run both and merge and rerank!\"**\n\n**Reciprocal Rank Fusion (RRF)** is the merging technique. Analogy: imagine two friends each independently rank the same 10 restaurants from 1 (best) to 10 (worst). To get one combined ranking that reflects both opinions, you don't just average their scores (their internal scoring \"scales\" may be totally different: one friend rates out of 5 stars, another out of 100 points). Instead, RRF says: *only the rank position matters, not the internal score* — a restaurant ranked #1 by both friends should shoot to the top of the combined list, even if their raw scores were on incompatible scales. This \"brings reciprocal ranks together\" as the notes put it.\n\n**The formula**, straight from the notes:\n\n```\nRRF(d) = Σ  1 / (k + rank_r(d))\n         r∈R\n```\n\nRead this piece by piece:\n\n`d`\n\nis a particular document you're scoring.`R`\n\nis the set of ranked result lists you're fusing (in our case: the dense-search ranking and the BM25 ranking).`rank_r(d)`\n\nis the position of document `d`\n\nin ranking `r`\n\n(1st place, 2nd place, etc.).`k`\n\nis a small constant, `1 / (k + rank)`\n\nfor The notes' visual intuition: dense list gives fractions like `1/1, 1/2, ...`\n\nand sparse list gives `1/61, 1/62, ...`\n\n(i.e., `1/(60+1), 1/(60+2), ...`\n\n) — \"**Fusion is better**\" because a document doesn't need to win outright in either individual list; it just needs to consistently rank *reasonably well* across both to accumulate a high combined score.\n\nOne more practical note from the notes: **Weaviate and Elasticsearch (ES) support hybrid search natively** — meaning they've built RRF-style fusion directly into the database. Otherwise, you have to do it yourself with a **scatter-gather** pattern (fire off both queries yourself, gather both result sets back in your application, then fuse them).\n\n```\n# =============================================================================\n# Reciprocal Rank Fusion (RRF) — implemented from the raw formula.\n# RRF(d) = sum over each ranked list r containing d of  1 / (k + rank_r(d))\n# =============================================================================\n\ndef reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:\n    \"\"\"\n    ranked_lists: a list of ranked result lists. Each inner list is already\n                  sorted best-to-worst, e.g. [\"doc3\", \"doc1\", \"doc9\"].\n    k: the RRF damping constant (60 is the standard default from the notes).\n\n    Returns a list of (doc_id, fused_score) sorted from best to worst.\n    \"\"\"\n    scores: dict[str, float] = {}\n\n    for ranked_list in ranked_lists:\n        for position, doc_id in enumerate(ranked_list):\n            rank = position + 1  # ranks start at 1, not 0\n            # This is the exact formula: 1 / (k + rank)\n            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)\n\n    # Sort documents by their fused score, highest first\n    fused_ranking = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)\n    return fused_ranking\n\n# --- Example usage ---------------------------------------------------------\n\n# Dense (semantic) search results for \"how do I reset my password?\"\ndense_results = [\"account_recovery_steps\", \"login_faq\", \"security_settings\", \"billing_faq\"]\n\n# BM25 (keyword) search results for the SAME query — note it ranks things\n# differently because it only sees exact word overlap\nsparse_results = [\"login_faq\", \"account_recovery_steps\", \"terms_of_service\", \"billing_faq\"]\n\nfused = reciprocal_rank_fusion([dense_results, sparse_results], k=60)\n\nfor doc_id, score in fused:\n    print(f\"{doc_id}: {score:.5f}\")\n\n# Expected behaviour: \"account_recovery_steps\" and \"login_faq\" both rank\n# highly in BOTH lists, so their fused scores rise above documents that only\n# appeared in one list or ranked poorly in both.\n# This shows the SHAPE of a real hybrid search call: one sparse (BM25) query,\n# one dense (vector) query, run independently, then fused with RRF.\n# (rank_bm25 and sentence-transformers are common lightweight libraries.)\n\n# pip install rank_bm25 sentence-transformers --break-system-packages\nfrom rank_bm25 import BM25Okapi\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\ndocuments = [\n    \"To reset your password, go to Account Recovery and click 'Forgot Password'.\",\n    \"Error code 1099-MISC indicates a mismatch in the tax reporting form.\",\n    \"Our billing FAQ covers refunds, invoices, and subscription changes.\",\n]\ndoc_ids = [\"account_recovery\", \"error_1099_misc\", \"billing_faq\"]\n\nquery = \"how do I reset my password\"\n\n# --- Sparse (BM25) leg: pure keyword/token overlap scoring -----------------\ntokenized_corpus = [doc.lower().split() for doc in documents]\nbm25 = BM25Okapi(tokenized_corpus)\nbm25_scores = bm25.get_scores(query.lower().split())\nsparse_ranking = [doc_ids[i] for i in np.argsort(bm25_scores)[::-1]]\n\n# --- Dense (semantic) leg: embed query + docs, rank by cosine similarity ---\nembedder = SentenceTransformer(\"all-MiniLM-L6-v2\")  # small, fast embedding model\ndoc_embeddings = embedder.encode(documents, normalize_embeddings=True)\nquery_embedding = embedder.encode(query, normalize_embeddings=True)\ncosine_scores = doc_embeddings @ query_embedding  # dot product of normalized vectors = cosine similarity\ndense_ranking = [doc_ids[i] for i in np.argsort(cosine_scores)[::-1]]\n\nprint(\"Sparse (BM25) ranking:\", sparse_ranking)\nprint(\"Dense (semantic) ranking:\", dense_ranking)\n\n# --- Fuse both rankings with RRF (reusing the function from section 4.3) ---\nfinal_ranking = reciprocal_rank_fusion([dense_ranking, sparse_ranking], k=60)\nprint(\"Final hybrid ranking:\", final_ranking)\n```\n\nImagine you're searching your email for a specific message and you know it arrived last week. You could (a) tell your email client \"only show me emails from last week,\" and *then* search the text — that's fast and narrow. Or you could (b) search the text across your *entire* mailbox history, get back thousands of results, and manually throw away everything not from last week — that's slow and wasteful, since your search engine did a ton of unnecessary work scanning old emails it was always going to discard.\n\nNow a second, more serious example: at a company, an HR employee should be able to see every department's HR documents, but a regular software engineer should *only* see documents relevant to their own team — they should never even see, let alone search inside, another department's confidential HR files. This isn't just a nice-to-have; it's a hard security requirement, often called **RBAC** (Role-Based Access Control — a plain-English one-liner: *\"restricting what a user can see or do based on their assigned role\"*).\n\n**Metadata filtering** applies **hard constraints** (like date ranges, document owner, department, tags) either **before** or **after** the vector search runs, on top of whatever relevance ranking your retrieval produces. The notes' example query: *\"What was our Q1 2026 revenue?\"* — you'd want to filter to `date range = Q1 2026`\n\nbefore even letting semantic search loose on the whole corpus.\n\nBeyond just retrieval quality, the notes stress this is *especially* important for **Enterprise RAG and RBAC**: an employee simply must not be able to retrieve another department's HR docs, no matter how semantically relevant those docs might look to their query. This is a security boundary, not just a relevance tweak.\n\nThere are two places you can apply the filter:\n\n```\n# =============================================================================\n# PRE-FILTERING: the filter is passed INTO the vector database query itself.\n# The DB never even considers documents outside the RBAC/date boundary, so\n# ranking + filtering happen together, efficiently, in one pass.\n# (Qdrant-style filter syntax, as referenced in the notes' prototype.)\n# =============================================================================\n\n# pip install qdrant-client --break-system-packages\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.models import Filter, FieldCondition, MatchValue, Range\n\nclient = QdrantClient(url=\"http://localhost:6333\")  # assumes a running Qdrant instance\n\nquery_vector = [0.12, 0.98, 0.31]  # a real embedding would be much longer, e.g. 1536-dim\n\nresults = client.search(\n    collection_name=\"hr_documents\",\n    query_vector=query_vector,\n    query_filter=Filter(\n        must=[\n            # RBAC constraint: only documents this user's department may see\n            FieldCondition(key=\"department\", match=MatchValue(value=\"engineering\")),\n            # Date constraint: only Q1 2026 documents\n            FieldCondition(key=\"date\", range=Range(gte=\"2026-01-01\", lte=\"2026-03-31\")),\n        ]\n    ),\n    limit=10,\n)\n# Qdrant applies the filter WHILE searching, so all 10 results returned are\n# guaranteed to satisfy both constraints — no wasted compute, no leakage risk.\n\n# =============================================================================\n# POST-FILTERING: search everything first, THEN discard invalid results in\n# plain Python. Shown for contrast — this is what the notes call \"wasteful.\"\n# =============================================================================\n\ndef post_filter_search(all_search_results: list[dict], user_department: str) -> list[dict]:\n    \"\"\"\n    all_search_results: e.g. the top-K matches from an UNFILTERED vector search\n                         across the entire corpus, regardless of department.\n    \"\"\"\n    filtered = [\n        doc for doc in all_search_results\n        if doc[\"department\"] == user_department\n        and \"2026-01-01\" <= doc[\"date\"] <= \"2026-03-31\"\n    ]\n    return filtered\n\n# Danger illustrated: if the unfiltered top-10 search returns mostly documents\n# from OTHER departments (because they happened to be semantically similar),\n# post-filtering might leave you with just 1-2 usable results, or worse —\n# if the filtering logic has a bug, a forbidden document could slip through\n# into the LLM's context before being filtered, which is a real security risk\n# with RBAC-sensitive data. Pre-filtering avoids this entirely.\n```\n\n**Bi-encoder analogy:** imagine two strangers each write their own short bio independently — \"I like hiking, sci-fi movies, and cooking Italian food\" and \"I enjoy the outdoors, watching space films, and making pasta.\" You then compare these two bios *after the fact* to guess how compatible these two people might be. It's fast (you can pre-write and store millions of bios ahead of time and just compare later), but it's a bit shallow — the two people never actually talked to each other, so subtle nuances get missed.\n\n**Cross-encoder analogy:** now imagine instead you sit those same two strangers down together for an actual conversation. They can react to each other in real time, clarify ambiguous statements, and pick up on nuance that never would have shown up in two separately-written bios. You get a *much* better read on true compatibility — but it only works one pair at a time, and a live conversation takes real time and effort per pair. You couldn't have every person in a stadium have a real conversation with every other person — it simply wouldn't scale.\n\n**Bi-encoders** (used for the *first* stage of dense retrieval, described back in Section 4): the query is turned into an embedding, and each document is turned into an embedding, **completely independently** of each other. Relevance is then just **vector proximity** — how close the two embeddings are (measured via cosine similarity). Because documents can be embedded once, offline, ahead of time, bi-encoders are **fast at query time and scalable** — this is why they're \"good enough for most\" use cases, per the notes.\n\nTheir key weakness, from the notes: **\"No cross attention, i.e. query and doc never see each other.\"** Because the two embeddings were produced in isolation, the model can't attend to specific interactions between query terms and document terms. This makes bi-encoders **weaker on negation, exact terms, etc.** — e.g., a bi-encoder might not clearly distinguish \"the flight is NOT delayed\" from \"the flight IS delayed,\" because both sentences produce very similar embeddings overall (they share almost all the same words).\n\n**Cross-encoders**, by contrast, take the ** <query, document> pair as a single input** — literally concatenating the query and the candidate document together — and output\n\nThe notes draw a small attention diagram illustrating the **\"lost in the middle\" problem**: attention/relevance tends to concentrate most heavily near the query token and decays as you move deeper into a long document — meaning information buried in the *middle* of a long document or a long context window can get comparatively under-weighted relative to information at the start or end. This is a known failure mode of transformer attention over long inputs, and it's part of why reranking with a small, focused candidate set (rather than dumping a giant unranked context at the LLM) genuinely helps.\n\n**Cross-encoders: much higher accuracy** — they can compare every token in the query to every token in the document, so they **handle negation, exact matches, etc.** far better than bi-encoders. Example model named in the notes: `cross-encoder/ms-marco-MiniLM-L-6-v2`\n\n.\n\n**The catch — cross-encoders are slow and don't scale.** From the notes: they work well on \"50 doc[s] v[s] million doc[s] per query\" — meaning cross-encoders are only practical on a *small* shortlist, never the entire corpus. There's **no indexing** possible (because the score depends on the specific query, you can't precompute it ahead of time the way you can with bi-encoder embeddings), and each pair costs roughly **~10ms of latency**.\n\n**The latency math, worked out step by step (from the notes):**\n\n`<query, doc>`\n\npairThat's obviously unusable in a real product. The fix, stated directly in the notes: **\"could be done on candidate docs only.\"** This is exactly why the two techniques are used *together* in a pipeline, not as alternatives: use the cheap, scalable bi-encoder (plus BM25/hybrid search) to narrow millions of documents down to a small shortlist (say, the top 40), and *then* run the expensive, high-accuracy cross-encoder reranker only on that small shortlist. You get bi-encoder speed at scale, and cross-encoder accuracy where it matters most — the final ranking of the few documents that actually make it into the LLM's context.\n\n```\n# pip install sentence-transformers --break-system-packages\nfrom sentence_transformers import SentenceTransformer, CrossEncoder\nimport numpy as np\n\nquery = \"Why is my API returning a 500 error under high load?\"\n\ncandidate_docs = [\n    \"Throughput degradation under load can trigger unhandled exceptions, \"\n    \"which surface to callers as HTTP 500 errors.\",\n    \"To reset your password, click 'Forgot Password' on the login screen.\",\n    \"500 errors are NOT related to client-side input validation issues.\",\n    \"Our billing FAQ covers refunds, invoices, and subscription cancellations.\",\n]\n\n# =============================================================================\n# STAGE 1 (fast, scalable): BI-ENCODER — embed query and docs SEPARATELY,\n# then compare with cosine similarity. This is what you'd run across\n# millions of documents to get an initial shortlist.\n# =============================================================================\nbi_encoder = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nquery_vec = bi_encoder.encode(query, normalize_embeddings=True)\ndoc_vecs = bi_encoder.encode(candidate_docs, normalize_embeddings=True)\n\n# cosine similarity = dot product of normalized vectors\nbi_encoder_scores = doc_vecs @ query_vec\n\nprint(\"Bi-encoder ranking (fast, first-pass):\")\nfor idx in np.argsort(bi_encoder_scores)[::-1]:\n    print(f\"  {bi_encoder_scores[idx]:.3f}  {candidate_docs[idx][:60]}...\")\n\n# =============================================================================\n# STAGE 2 (slow, accurate): CROSS-ENCODER — feed the query+doc TOGETHER as a\n# single pair, get one relevance score per pair. Only run this on the small\n# shortlist that stage 1 already narrowed down (here, all 4, since it's tiny).\n# =============================================================================\ncross_encoder = CrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n\npairs = [[query, doc] for doc in candidate_docs]  # literal <query, doc> pairing\ncross_encoder_scores = cross_encoder.predict(pairs)\n\nprint(\"\\nCross-encoder ranking (slow, high-precision rerank):\")\nfor idx in np.argsort(cross_encoder_scores)[::-1]:\n    print(f\"  {cross_encoder_scores[idx]:.3f}  {candidate_docs[idx][:60]}...\")\n\n# You'll typically notice the cross-encoder more confidently separates the\n# TRUE match (the throughput/500-error doc) from the negation trap (doc #3,\n# which mentions \"500 errors\" but explicitly says they are NOT related to\n# the topic) — a distinction bi-encoders are known to struggle with.\n```\n\nImagine you ask a librarian, \"Why is my code slow?\" The librarian's catalog, though, doesn't use casual words like \"slow\" — the technical manuals on the shelf are indexed under formal terms like \"throughput degradation,\" \"latency regression,\" or \"bottleneck analysis.\" If the librarian searches the catalog using your exact casual phrase, they might get nothing useful back, even though the *perfect* manual for your problem exists on the shelf. The words just don't line up.\n\nA clever librarian's trick: before searching, they first jot down (in their head) roughly *what the ideal, technical-sounding manual would probably say* about your problem — then they search the catalog using *that* imagined, more formal description instead of your casual phrasing. That imagined \"ideal answer\" acts as a bridge between your everyday words and the library's formal vocabulary.\n\nThat's exactly what **HyDE** (Hypothetical Document Embedding) does for a RAG system.\n\nThe notes call the **\"Query-Document Gap\"** the *most persistent failure mode in RAG*. It shows up because user queries tend to be **short and conversational**, while the underlying documents/corpus are often **semantically distant** in their vocabulary — the classic example: \"why is my pipeline slow?\" (user's words) vs. \"Throughput degradation is...\" (the document's actual wording). Even though these mean almost the same thing conceptually, their raw embeddings may not be close enough for a naive semantic search to reliably connect them.\n\nThere are two named fixes in the notes:\n\n**1. Expand or rephrase the user query.** Simply ask an LLM: *\"generate 3 alternate phrasings...\"* of the user's question, and search using all of them (or a combination), increasing the odds that at least one phrasing's embedding lands close to the real document.\n\n**2. HyDE — Hypothetical Document Embedding.** Instead of embedding the user's raw, short question, you first ask an LLM to *write a short document that would answer this question* — the exact prompt style from the notes: **\"Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like.\"** You then take *that generated hypothetical document* — not the original question — and embed *it*. Because this hypothetical answer is written in a style and vocabulary much closer to how real documents in your corpus are likely written, its embedding tends to land much closer to the *real* relevant document's embedding than the original short, casual question would have.\n\nTechnically, HyDE reframes retrieval as a **similarity search between d and d ∈ D** — that is, comparing a (hypothetical) document to the real documents in your corpus\n\n`D`\n\n, rather than comparing a short question to documents (which is a fundamentally harder, more mismatched comparison).**When does this actually help?** The notes are precise about this: HyDE **\"works when corpus vocab is very different than user query.\"** Good candidates: **scientific papers, legal texts, internal jargon-heavy documentation** — domains where the \"real\" documents are written in dense, formal, or highly technical language that a typical user's question would never naturally use.\n\n``` python\n# pip install anthropic sentence-transformers --break-system-packages\nimport anthropic\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\nclient = anthropic.Anthropic()\nembedder = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\ndef generate_hypothetical_answer(user_query: str) -> str:\n    \"\"\"\n    Ask an LLM to write a short, FACTUAL-SOUNDING document that would answer\n    the user's question. This is the exact prompt shape from the notes:\n    \"Write a short document that will answer this question. Be factual.\n    If you don't know the answer, write what the answer would typically\n    look like.\"\n    We use this generated text purely to bridge the vocabulary gap — we do\n    NOT show this hypothetical text to the end user; it's an internal\n    retrieval-only artifact.\n    \"\"\"\n    response = client.messages.create(\n        model=\"claude-sonnet-4-6\",\n        max_tokens=300,\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": (\n                    \"Write a short document that will answer this question. \"\n                    \"Be factual. If you don't know the answer, write what the \"\n                    f\"answer would typically look like.\\n\\nQuestion: {user_query}\"\n                ),\n            }\n        ],\n    )\n    return response.content[0].text\n\ndef hyde_retrieve(user_query: str, doc_texts: list[str], doc_ids: list[str], top_k: int = 3):\n    \"\"\"\n    Full HyDE retrieval: generate a hypothetical answer, embed THAT instead\n    of the raw query, then find the closest real documents to it.\n    \"\"\"\n    # Step 1: generate the hypothetical, jargon-matched answer\n    hypothetical_doc = generate_hypothetical_answer(user_query)\n\n    # Step 2: embed the HYPOTHETICAL document, not the raw user query\n    hyde_embedding = embedder.encode(hypothetical_doc, normalize_embeddings=True)\n\n    # Step 3: embed the real corpus documents (in production this is pre-computed\n    # and stored in a vector DB, not re-computed on every query)\n    doc_embeddings = embedder.encode(doc_texts, normalize_embeddings=True)\n\n    # Step 4: rank real documents by similarity to the hypothetical embedding\n    similarities = doc_embeddings @ hyde_embedding\n    top_indices = np.argsort(similarities)[::-1][:top_k]\n\n    return [(doc_ids[i], doc_texts[i], float(similarities[i])) for i in top_indices]\n\n# --- Example -----------------------------------------------------------------\ncorpus_texts = [\n    \"Throughput degradation under sustained load is typically caused by \"\n    \"connection pool exhaustion or unindexed database queries.\",\n    \"Our refund policy allows cancellations within 30 days of purchase.\",\n    \"Latency regressions in distributed systems often trace back to N+1 \"\n    \"query patterns or missing caching layers.\",\n]\ncorpus_ids = [\"perf_doc_1\", \"refund_policy\", \"perf_doc_2\"]\n\nresults = hyde_retrieve(\"why is my code slow?\", corpus_texts, corpus_ids)\nfor doc_id, text, score in results:\n    print(f\"{score:.3f}  {doc_id}: {text[:70]}...\")\n\n# Without HyDE, embedding the raw casual query \"why is my code slow?\" might\n# rank the perf docs only weakly, since they never use the word \"slow.\"\n# With HyDE, the generated hypothetical answer naturally uses words like\n# \"throughput,\" \"latency,\" and \"degradation\" -- much closer to the real docs.\n```\n\nImagine ten different customers each phrase the exact same underlying question in slightly different words: \"How do I integrate libX?\", \"I want help integrating libX,\" \"How can I get libX working with my app?\" A naive cache (the kind that only matches *exact* strings) would treat these as three totally unrelated questions and pay to compute a fresh, expensive answer for every single one — even though the *right* answer is identical for all three.\n\nA smarter cache recognizes that all three questions *mean* the same thing, and serves the same cached, already-computed answer to all of them — saving time and money on the second and third askers, without them ever noticing anything different.\n\nThe core motivating fact from the notes: **\"Every call to LLM cost[s] latency and tokens\"** — i.e., **time and money**. A **semantic cache** sits in front of the LLM: it **intercepts a query before it reaches the model** and, if a *semantically similar* (not necessarily an exact-text-match) prior query exists in the cache, it returns the cached response instead of paying for a new LLM call.\n\n**When this works best**, per the notes: **static data with no personalization** — situations where the \"right\" answer genuinely doesn't change per-user or over short timeframes. Two named examples:\n\n**Implementation, step by step (from the notes):**\n\n**The core tension: cost vs. correctness**, governed entirely by *what similarity threshold you pick and how* you set it. The notes are blunt about the two failure directions:\n\n**The reliable way to pick a threshold**, per the notes: don't guess — **empirically find your \"sweet spot\" on your own query distribution.** As a reasonable starting point, the notes suggest **0.85 cosine similarity**, but this must be *calibrated*, not treated as gospel, because every product's query patterns differ.\n\n```\n# pip install sentence-transformers --break-system-packages\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\nclass SemanticCache:\n    \"\"\"\n    A minimal in-memory semantic cache: get() looks for a semantically\n    similar prior query above `similarity_threshold`; set() stores a new\n    query + response pair for future lookups.\n    \"\"\"\n\n    def __init__(self, similarity_threshold: float = 0.85):\n        self.embedder = SentenceTransformer(\"all-MiniLM-L6-v2\")\n        self.similarity_threshold = similarity_threshold\n        self.cached_queries: list[str] = []\n        self.cached_embeddings: list[np.ndarray] = []\n        self.cached_responses: list[str] = []\n\n    def get(self, query: str) -> str | None:\n        \"\"\"Return a cached response if a semantically similar query exists, else None.\"\"\"\n        if not self.cached_queries:\n            return None\n\n        query_embedding = self.embedder.encode(query, normalize_embeddings=True)\n        similarities = np.array(self.cached_embeddings) @ query_embedding\n        best_idx = int(np.argmax(similarities))\n        best_score = float(similarities[best_idx])\n\n        if best_score >= self.similarity_threshold:\n            print(f\"[cache HIT] similarity={best_score:.3f} matched: \"\n                  f\"'{self.cached_queries[best_idx]}'\")\n            return self.cached_responses[best_idx]\n\n        print(f\"[cache MISS] best similarity was only {best_score:.3f}\")\n        return None\n\n    def set(self, query: str, response: str) -> None:\n        \"\"\"Store a new query + response pair in the cache.\"\"\"\n        embedding = self.embedder.encode(query, normalize_embeddings=True)\n        self.cached_queries.append(query)\n        self.cached_embeddings.append(embedding)\n        self.cached_responses.append(response)\n\n# --- Demonstration: threshold too low vs. too high --------------------------\n\ncache = SemanticCache(similarity_threshold=0.85)\ncache.set(\"How do I integrate libX?\", \"To integrate libX, install it via pip and call libx.init()...\")\n\nprint(\"\\n--- Reasonable threshold (0.85) ---\")\ncache.get(\"I want help integrating libX\")   # should HIT: same intent, different phrasing\ncache.get(\"What's the weather in Tokyo?\")   # should MISS: unrelated question\n\nprint(\"\\n--- Threshold set TOO LOW (0.50): risks wrong answers ---\")\nloose_cache = SemanticCache(similarity_threshold=0.50)\nloose_cache.set(\"How do I cancel my subscription?\", \"Go to Settings > Billing > Cancel Plan.\")\n# This is a RELATED but MEANINGFULLY DIFFERENT question -- with a low\n# threshold it may incorrectly match and serve the wrong cached answer.\nloose_cache.get(\"How do I upgrade my subscription plan?\")\n\nprint(\"\\n--- Threshold set TOO HIGH (0.99): hit rate collapses ---\")\nstrict_cache = SemanticCache(similarity_threshold=0.99)\nstrict_cache.set(\"How do I integrate libX?\", \"To integrate libX, install it via pip...\")\n# Even a near-identical paraphrase may now fail to match because embeddings\n# are almost never PERFECTLY identical between two different sentences.\nstrict_cache.get(\"I want help to integrate libX\")\n```\n\nThe prototype in the notes (\"Semantic Caching and Similarity Calibration\") describes exactly this experiment at scale: **500 queries → embeddings → index**, then sweeping thresholds from **0.75 to 0.95**, populating the cache first, then computing **hit rate** (fraction of queries found in cache) and **accuracy/precision**. It uses three types of test queries: **seed** (the original query, e.g. \"Explain the ending of movie Arrival\"), **paraphrase** (same intent, different words, e.g. \"Please clarify the conclusion of the movie Arrival\" — a *correct* match), and **near-miss** (looks similar but has a genuinely different intent, e.g. \"Who directed the movie Arrival\" — a *trap*, since matching this to the seed's answer would be a **false positive / incorrect hit**). The accuracy formula from the notes:\n\n```\naccuracy = correct hits (paraphrase matching the seed) / total hits\n```\n\nBelow is a small script that runs this exact style of calibration experiment:\n\n``` python\ndef evaluate_threshold(cache: SemanticCache, seed_paraphrase_pairs, near_miss_queries):\n    \"\"\"\n    seed_paraphrase_pairs: list of (seed_query, paraphrase_query, response) tuples\n                           -- paraphrase SHOULD correctly hit the seed's cached entry.\n    near_miss_queries: list of queries that LOOK similar but have different intent\n                       -- these SHOULD ideally miss (or if they hit, that's a false positive).\n    \"\"\"\n    for seed, response in {(s, r) for s, _, r in seed_paraphrase_pairs}:\n        cache.set(seed, response)\n\n    total_hits, correct_hits = 0, 0\n\n    for seed, paraphrase, _ in seed_paraphrase_pairs:\n        result = cache.get(paraphrase)\n        if result is not None:\n            total_hits += 1\n            correct_hits += 1  # paraphrase hitting its own seed = correct\n\n    false_positives = 0\n    for near_miss in near_miss_queries:\n        result = cache.get(near_miss)\n        if result is not None:\n            total_hits += 1\n            false_positives += 1  # near-miss hitting ANY cached entry = wrong\n\n    accuracy = correct_hits / total_hits if total_hits else 0\n    return {\"hit_rate_relevant\": correct_hits, \"false_positives\": false_positives, \"accuracy\": accuracy}\n```\n\nImagine building a \"help desk\" for a huge company with 10 million internal documents. A visitor walks up and asks a question in plain English. Before answering, your help desk clerk must: (1) understand what's really being asked, (2) go search *both* the exact-keyword index and the meaning-based index of every document, (3) combine and re-sort those results by true relevance, (4) double-check that the results are actually confident and trustworthy (not just \"vaguely related\"), (5) draft an answer strictly grounded in what was found, and — critically — (6) go back and check every single sentence of that draft answer against the source documents *before* saying it out loud, refusing to say anything it can't verify. That's the whole system in one sentence: **retrieve carefully, answer carefully, and never say something you can't prove.**\n\n**What the system does:** Answers natural language queries. **The one hard guarantee that shapes the entire design:** *\"the system must not assert any claim that cannot be traced to [a] retrieved passage.\"* This single sentence is why \"correctness\" (not just relevance, not just speed) drives nearly every architectural decision below. Why this guarantee matters: an internal knowledge system that confidently states wrong facts is often *worse* than one that admits \"I don't know,\" because wrong confident answers erode trust and can cause real harm (wrong policy info, wrong financial numbers, etc.).\n\n**What the system explicitly is NOT:** a document management system, an ingestion pipeline product, etc. — the notes flag this so the design stays scoped: we are *not* building a general-purpose CMS, just the \"ask questions about documents that already exist somewhere\" layer.\n\n**Scale numbers (why they matter):**\n\nThe notes lay out the exact ordered checklist a systems-design interview or real design doc would walk through:\n\nLet's walk through each, grounding every architectural choice in a \"why\" before the \"what.\"\n\n**1 & 2 — Storage location and Query API.** Documents live in an **S3 bucket** (e.g. `s3://myrag/doc1.txt`\n\n), organized per the notes with some inspiration from Git's object storage model (content-addressed, organized by structure like `date`\n\n/`git`\n\n-style hashing prefixes, e.g. folders `ab/`\n\n, `ac/`\n\n, `ad/`\n\n— a common trick to avoid dumping literally 10 million flat files into one folder, which many filesystems and object stores handle poorly). The query API is a simple HTTP endpoint: `GET /query?`\n\n— the notes sketch this as `@app.get('/query?')`\n\n.\n\n**3 — Response shape.** From the notes, a single `respond()`\n\ncall returns a structured object containing:\n\n```\nquestion: <string>\nfilters: { date_from, date_to, source_domains, tag, topics, etc. }\nanswer: <string>\nrefused: <true/false>\nrefusal_reason: <string, if refused>\npassages: [ {doc_id, passage_id, text, score, highlight_idx}, ... ]\nconfidence: <number>\nquery_id: <string>   -- used for tracing/debugging a specific request later\n```\n\nWhy this shape: it's not enough to return just an answer string. You need the **passages** (so a user or auditor can verify the claim was actually grounded), a **refused** boolean plus **reason** (so \"I don't know\" is a first-class, explicit outcome, not a vague hedge buried in the answer text), a **confidence** score (so downstream systems or UI can decide whether to trust/display the answer), and a **query_id** for **tracing** — being able to look up exactly what happened for a specific request later, which is essential for debugging a \"why did it hallucinate here\" incident.\n\n**4 — Which databases, and why:**\n\n**5 — Capacity estimation, worked step by step (numbers straight from the notes' math):**\n\n*Query load:*\n\n```\n50,000 DAU (daily active users) × 8 queries/day/user × 4.5 (peak multiplier)\n──────────────────────────────────────────────────────────────────────────  = 200 QPS\n                          86,400 seconds/day\n```\n\nSo: 50,000 × 8 = 400,000 total queries/day. 400,000 ÷ 86,400 seconds ≈ 4.6 QPS *average*. Multiply by a **peak multiplier of 4.5** (because traffic isn't spread evenly across 24 hours — there are busy hours) to get roughly **200 QPS at peak**. The team then designs for a bit of headroom: **250 QPS** target capacity.\n\n*Storage — raw documents:*\n\n```\n10,000,000 docs × 2KB each = 20GB raw data\n```\n\nThis is small enough to be almost trivial to store redundantly — it's the *derived* data (embeddings, indexes) that actually dominates storage, as shown next.\n\n*Storage — vector embeddings:*\n\n```\n10 × 10^6 documents × 8 passages/doc (avg) × 1536 dimensions × 4 bytes/float32\n= 10×10^6 × 8 × 1536 × 4 bytes\n≈ 491 GB (just the raw embedding vectors)\n```\n\nThen add **HNSW graph overhead** (the extra data structure the ANN index needs to do fast approximate search) — roughly **1.5x** — bringing:\n\n```\nTotal vector index storage ≈ 491GB × 1.5 ≈ 700GB\n```\n\nThis is why the notes conclude: **\"Hence, we need a distributed vector database\"** — 700GB (spread across replicas for redundancy and read throughput) is well beyond what a single machine comfortably handles for a low-latency, always-on service. The notes also mention **Product Quantization** can reduce this by roughly **8x**, but *\"at [the] cost of recall\"* — i.e., you trade some retrieval accuracy for a much smaller memory footprint, a classic space/accuracy tradeoff.\n\n*ElasticSearch (BM25 index) sizing:* roughly `20GB × 1.5 ≈ 30GB`\n\nfor the inverted index, run as a **3-node cluster with high availability (HA)** plus 3 replicas, ≈ **90GB total**, each node with **30GB RAM**.\n\n*Postgres sizing:* the 20GB of metadata plus JSONB overhead and replication comes out to roughly **60-80GB**, served with **3 read replicas** (since this is read-heavy traffic) with a target replication lag of **under 100ms**.\n\n*Redis:* a **3-node cluster with 3 replicas**, caching passages and query results — this is explicitly conditioned on the earlier semantic-caching insight: it works best **\"if no personalization & data is static.\"**\n\n**6, 7, 8 — Read path, what's returned, and failure handling** are covered together via the code walkthrough in Section 9.4 below (the notes' own `respond()`\n\npseudocode).\n\n**9 — Write path** is covered in Section 9.5 below.\n\n`respond()`\n\nfunction), cleaned up and fully commented\nThe notes' own hand-drawn pseudocode for the read path is the heart of the \"zero hallucinations\" guarantee. Here it is translated into clean, fully-commented Python:\n\n``` python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass RespondResult:\n    answer: str\n    passages: list\n    refused: bool\n    refusal_reason: str | None = None\n    confidence: float | None = None\n\ndef respond(question: str) -> RespondResult:\n    \"\"\"\n    The full \"zero hallucinations\" read path:\n    embed -> hybrid search (dense + sparse) -> RRF fusion -> cross-encoder\n    rerank -> confidence threshold -> LLM generate -> claim extraction ->\n    NLI fact-check -> final answer OR refusal.\n\n    Every stage is a chance to refuse rather than guess -- that's the whole\n    point of the \"must not assert unsupported claims\" guarantee.\n    \"\"\"\n\n    # -------------------------------------------------------------------\n    # Step 1: Embed the question (turn text into a dense vector).\n    # -------------------------------------------------------------------\n    query_vector = embed(question)\n\n    # -------------------------------------------------------------------\n    # Step 2: Hybrid search -- run dense (vector) and sparse (BM25) search\n    # CONCURRENTLY (this is the \"parallel tool calls\" idea from Section 3\n    # applied to retrieval itself), each returning their own top-40 shortlist.\n    # -------------------------------------------------------------------\n    dense_results = qdrant_ann_search(query_vector, top_k=40)\n    sparse_results = elasticsearch_bm25(question, top_k=40)\n\n    # -------------------------------------------------------------------\n    # Step 3: Fuse both ranked lists with Reciprocal Rank Fusion (Section 4).\n    # -------------------------------------------------------------------\n    candidates = rrf_fusion(dense_results, sparse_results)\n\n    # -------------------------------------------------------------------\n    # Step 4: Rerank the fused candidates with a cross-encoder (Section 6)\n    # for much higher-precision final ordering. Cross-encoders are too slow\n    # to run on millions of docs, but perfectly fine on ~40-80 candidates.\n    # -------------------------------------------------------------------\n    ranked = cross_encoder_rerank(question, candidates, top_k=40)\n\n    # -------------------------------------------------------------------\n    # Step 5: CONFIDENCE GATE #1 -- if even the BEST passage scores too low,\n    # we don't have real evidence to answer with. Refuse rather than guess.\n    # -------------------------------------------------------------------\n    if not any(passage.score > 0.4 for passage in ranked):\n        return refused_response(\"low_retrieval_confidence\")\n\n    # -------------------------------------------------------------------\n    # Step 6: Take only the top 5 passages -- this keeps the LLM's context\n    # small, focused, and less prone to the \"lost in the middle\" problem\n    # (Section 6) that hurts long, noisy contexts.\n    # -------------------------------------------------------------------\n    top_passages = ranked[:5]\n    prompt = build_prompt(question, top_passages)\n\n    # -------------------------------------------------------------------\n    # Step 7: Ask the LLM to generate an answer, GROUNDED ONLY in top_passages.\n    # temperature=0 for maximum determinism/factuality (no creative sampling).\n    # -------------------------------------------------------------------\n    raw_answer = llm_generate(prompt, temperature=0)\n\n    # -------------------------------------------------------------------\n    # Step 8: CONFIDENCE GATE #2 -- the prompt instructs the LLM to say\n    # \"INSUFFICIENT_CONTEXT\" if the retrieved passages don't actually answer\n    # the question. If it says so, we honor that and refuse cleanly.\n    # -------------------------------------------------------------------\n    if raw_answer.startswith(\"INSUFFICIENT_CONTEXT\"):\n        return refused_response(\"no_relevant_passages\")\n\n    # -------------------------------------------------------------------\n    # Step 9: CLAIM EXTRACTION + FACT-CHECKING -- this is the actual\n    # \"zero hallucinations\" enforcement mechanism. Break the raw answer\n    # into small, individually-checkable \"atomic claims\" (e.g. one claim\n    # per sentence or per fact), then verify EACH claim is actually\n    # entailed (logically supported) by the retrieved passages using an\n    # NLI (Natural Language Inference) model -- a plain-English one-liner:\n    # \"a model that decides whether a piece of text logically follows from,\n    # contradicts, or is unrelated to another piece of text.\"\n    # If ANY claim can't be verified, we refuse rather than let an\n    # unsupported claim slip through to the user.\n    # -------------------------------------------------------------------\n    claims = extract_atomic_claims(raw_answer)\n    for claim in claims:\n        if not nli_entailed(claim, top_passages):\n            return refused_response(\"unverified_claim\")\n\n    # -------------------------------------------------------------------\n    # All gates passed: the answer is grounded, confident, and every claim\n    # has been individually verified against the retrieved evidence.\n    # -------------------------------------------------------------------\n    return RespondResult(answer=raw_answer, passages=top_passages, refused=False)\n\ndef refused_response(reason: str) -> RespondResult:\n    \"\"\"A first-class, explicit refusal -- NOT a vague hedge buried in text.\"\"\"\n    return RespondResult(answer=\"\", passages=[], refused=True, refusal_reason=reason)\n\n# NOTE: embed(), qdrant_ann_search(), elasticsearch_bm25(), rrf_fusion(),\n# cross_encoder_rerank(), build_prompt(), llm_generate(), extract_atomic_claims(),\n# and nli_entailed() are each real functions you'd implement using the\n# techniques from Sections 4, 6, and standard NLI models (e.g. a model\n# fine-tuned on MNLI/SNLI for entailment classification) -- they are stubbed\n# here to keep the read-path LOGIC the star of this example.\n```\n\nThe write path (S3 → chunk ingestor → hierarchical chunking → embeddings → storage) needs a chunker. The notes specify **sentence-level boundaries (using spaCy)** plus **32-token overlap** between consecutive chunks (overlap ensures a sentence that would otherwise get awkwardly split across two chunks still has enough surrounding context in at least one of them).\n\n```\n# pip install spacy --break-system-packages\n# python -m spacy download en_core_web_sm\nimport spacy\n\nnlp = spacy.load(\"en_core_web_sm\")\n\ndef chunk_document(text: str, max_tokens_per_chunk: int = 256, overlap_tokens: int = 32) -> list[dict]:\n    \"\"\"\n    Hierarchical, sentence-boundary-respecting chunking with token overlap.\n\n    Why sentence boundaries: splitting mid-sentence produces chunks that are\n    semantically broken and embed poorly. Why overlap: a fact stated right at\n    a chunk boundary shouldn't lose its surrounding context in EVERY chunk\n    that references it.\n    \"\"\"\n    doc = nlp(text)\n    sentences = [sent.text.strip() for sent in doc.sents]\n\n    chunks = []\n    current_chunk_sentences: list[str] = []\n    current_token_count = 0\n    char_cursor = 0\n\n    def rough_token_count(s: str) -> int:\n        # A simple approximation (real systems use a proper tokenizer, e.g.\n        # tiktoken, matching whatever embedding model will consume the chunk).\n        return len(s.split())\n\n    for sentence in sentences:\n        sentence_tokens = rough_token_count(sentence)\n\n        # If adding this sentence would overflow the chunk, close the\n        # current chunk out and start a new one, carrying over the last\n        # `overlap_tokens` worth of sentences for continuity.\n        if current_token_count + sentence_tokens > max_tokens_per_chunk and current_chunk_sentences:\n            chunk_text = \" \".join(current_chunk_sentences)\n            chunks.append({\n                \"text\": chunk_text,\n                \"token_count\": current_token_count,\n                \"char_start\": char_cursor,\n                \"char_end\": char_cursor + len(chunk_text),\n            })\n            char_cursor += len(chunk_text)\n\n            # Build the overlap: keep trailing sentences until we've got\n            # roughly `overlap_tokens` worth, to seed the next chunk.\n            overlap_sentences, overlap_count = [], 0\n            for sent in reversed(current_chunk_sentences):\n                overlap_sentences.insert(0, sent)\n                overlap_count += rough_token_count(sent)\n                if overlap_count >= overlap_tokens:\n                    break\n\n            current_chunk_sentences = overlap_sentences\n            current_token_count = overlap_count\n\n        current_chunk_sentences.append(sentence)\n        current_token_count += sentence_tokens\n\n    # Flush the final chunk\n    if current_chunk_sentences:\n        chunk_text = \" \".join(current_chunk_sentences)\n        chunks.append({\n            \"text\": chunk_text,\n            \"token_count\": current_token_count,\n            \"char_start\": char_cursor,\n            \"char_end\": char_cursor + len(chunk_text),\n        })\n\n    return chunks\n\n# --- Example ---------------------------------------------------------------\nsample_doc = (\n    \"Throughput degradation under sustained load is a common issue. \"\n    \"It is typically caused by connection pool exhaustion. \"\n    \"Another frequent cause is unindexed database queries. \"\n    \"Monitoring dashboards should track p50, p95, and p99 latency. \"\n    \"Alerts should fire when p99 crosses a defined SLO threshold.\"\n)\n\nfor i, chunk in enumerate(chunk_document(sample_doc, max_tokens_per_chunk=15, overlap_tokens=5)):\n    print(f\"Chunk {i}: {chunk['text']}\")\n```\n\nEach resulting chunk (a \"passage\" in the notes' schema) would then get an embedding computed (using one of the models the notes list — `microsoft/hamier`\n\n, `Qwen3 (8B)`\n\n, `BGE-M3`\n\n, or an OpenAI/OSS embedding API), stored in Qdrant (the vector), Postgres (the metadata: doc_id, chunk_index, char_start, char_end, token_count), and ElasticSearch (the BM25 inverted index).\n\nFlow: **S3 (raw doc storage) → Chunk Ingestor → Hierarchical Chunking → Embeddings → Storage (Postgres, Qdrant, ElasticSearch, Redis).**\n\nStep by step: a document lands in S3. A **chunk ingestor** process picks it up (typically via an event notification or polling), performs three sub-steps the notes list explicitly (**list files, download, read**), then runs **hierarchical chunking** (Section 9.5's sentence-boundary + overlap logic) to split it into passages. Each passage is embedded, and the results are fanned out to storage: **Postgres** gets the passage/doc metadata, **Qdrant** gets the embedding vectors, and **ElasticSearch** gets the text for its BM25 inverted index. This asynchronous pipeline is exactly why the system can tolerate the **~15-minute eventual consistency SLO** mentioned earlier — the write path doesn't need to be instantaneous, it needs to be reliable and complete within a bounded, acceptable delay.\n\nThe notes also flag a **timebox** mechanism on the read side with two named fallback behaviors — **cascade** and **fallback** — meaning if some part of the read path takes too long (e.g., the cross-encoder rerank step or the LLM generation call, described as \"the long pole\" of end-to-end latency), the system has a time budget after which it cascades to a simpler/cheaper strategy or falls back to a safe default (like a refusal) rather than hanging indefinitely. This connects directly back to the **query latency numbers** given in the notes: **p50 = 1.5 sec, p99 = 5 sec**, with the **LLM inference call** explicitly called out as the **long pole** (the slowest, bottleneck stage) — largely because the very first token the user sees (their **Time To First Token**, covered next in the Appendix) is gated on that call finishing its reasoning.\n\nThink about the difference between getting a text message that appears **all at once** after a long pause, versus watching someone type it live, word by word, right in front of you. Even though the *total* message might take the same amount of time to fully arrive, watching it appear progressively *feels* far less frustrating — you get immediate feedback that something is happening, instead of staring at a blank screen wondering if anything is working at all.\n\nThat's the entire motivation for streaming: even if the total response time is identical, showing partial output as it's generated dramatically improves perceived responsiveness.\n\nThe most important metric here, per the notes: **TTFT — Time To First Token** (a plain-English one-liner: *\"how long the user waits before they see ANY output at all, even if the full response isn't done yet\"*). If a tool call takes real time to execute (a slow database query, a slow external API), and the user has to wait through the *entire* tool call before seeing anything, that's flagged directly in the notes as **bad UX** — \"like everything else,\" slow, silent waiting erodes trust in the product.\n\nThe tricky part specific to *streaming tool calls*: **\"what if tool input arrives as partial JSON fragments?\"** When a model streams its response token by token, and part of that response is a `tool_use`\n\nblock (e.g., building up `{\"city\": \"Tok`\n\n... `yo\"}`\n\ncharacter by character), you cannot execute the tool with a half-formed, syntactically-invalid JSON blob. The notes' answer: **\"wait until the entire input is available\"** — i.e., **keep buffering** (accumulating) the partial JSON text as it streams in, and only attempt to parse/execute the tool call once the model signals that this particular content block is fully complete.\n\nThe notes include a cautionary, very concrete illustration of *why* this matters: imagine the model is streaming a dangerous tool call like **\"delete that record...\"** If your system tried to act on a half-baked, incomplete tool input (say, a delete call where the *id* argument hasn't fully arrived yet), it could either error out unpredictably or — worse — **\"no input could be fired\"** i.e., act on a malformed/empty argument. **Half-baked input / tool calls with no input could be fired** is exactly the failure mode the buffering strategy exists to prevent.\n\n``` python\nimport anthropic\nimport json\n\nclient = anthropic.Anthropic()\n\n# The tool definition is identical in shape to Section 1 -- streaming\n# doesn't change WHAT a tool looks like, only HOW the response arrives.\ntools = [{\n    \"name\": \"get_weather\",\n    \"description\": \"Get current weather for a location\",\n    \"input_schema\": {\n        \"type\": \"object\",\n        \"properties\": {\n            \"location\": {\"type\": \"string\"},\n            \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n        },\n        \"required\": [\"location\"],\n    },\n}]\n\ndef execute_tool(tool_name: str, tool_input: dict) -> dict:\n    \"\"\"Stand-in for a real tool implementation (see Section 1 for a fuller mock).\"\"\"\n    if tool_name == \"get_weather\":\n        return {\"location\": tool_input[\"location\"], \"temp\": 24, \"condition\": \"sunny\"}\n    return {\"error\": \"unknown tool\"}\n\nwith client.messages.stream(\n    model=\"claude-sonnet-4-6\",\n    max_tokens=1024,\n    tools=tools,\n    messages=[{\"role\": \"user\", \"content\": \"What is the weather in Tokyo?\"}],\n) as stream:\n\n    # These two variables ACCUMULATE state across streamed events -- this IS\n    # the \"buffering\" the notes insist on. We never act on partial state.\n    current_tool_input = \"\"\n    current_tool_name = None\n\n    for event in stream:\n        # -------------------------------------------------------------\n        # \"content_block_start\": a NEW block (either plain text or a\n        # tool_use block) has begun streaming. If it's a tool_use block,\n        # note its NAME immediately, but its INPUT starts empty --\n        # arguments arrive incrementally via later delta events.\n        # -------------------------------------------------------------\n        if event.type == \"content_block_start\":\n            if event.content_block.type == \"tool_use\":\n                current_tool_name = event.content_block.name\n                current_tool_input = \"\"  # reset the buffer for this new block\n\n        # -------------------------------------------------------------\n        # \"content_block_delta\": an incremental CHUNK of the current\n        # block has arrived. For tool_use blocks, this is a fragment of\n        # partial JSON (event.delta.type == \"input_json_delta\") -- we\n        # must ACCUMULATE it, not parse it yet, since it's incomplete.\n        # For plain text blocks, we can print immediately since partial\n        # text is still readable (this is what gives the \"word by word\"\n        # streaming feel for normal chat responses).\n        # -------------------------------------------------------------\n        elif event.type == \"content_block_delta\":\n            if event.delta.type == \"input_json_delta\":\n                current_tool_input += event.delta.partial_json  # <- Accumulate\n            elif event.delta.type == \"text_delta\":\n                print(event.delta.text, end=\"\", flush=True)\n\n        # -------------------------------------------------------------\n        # \"content_block_stop\": the CURRENT block (text or tool_use) has\n        # finished streaming completely. If it was a tool_use block, the\n        # buffered JSON string is now guaranteed complete and valid --\n        # THIS is the safe moment to parse it and actually execute the\n        # tool, never before.\n        # -------------------------------------------------------------\n        elif event.type == \"content_block_stop\":\n            if current_tool_name:\n                parsed_input = json.loads(current_tool_input)  # <- input complete\n                result = execute_tool(current_tool_name, parsed_input)  # <- execute\n                print(f\"\\n[Executed {current_tool_name} -> {result}]\")\n                current_tool_name = None  # reset for the next potential tool block\n```\n\nThe three event types to remember: ** content_block_start** (a new block began — note its type/name),\n\n`content_block_delta`\n\n`content_block_stop`\n\n**Context Precision, simple example:** think of a search-results page with 5 blue links. Some of them are genuinely useful and directly relevant to what you searched for (imagine green dots next to them); others are irrelevant filler or noise (red dots). A *good* search engine doesn't just include some relevant results somewhere on the page — it puts them **near the top**. A search engine that buries the one truly relevant result at position #5, behind four irrelevant ones, is worse than one that puts it at position #1, even if both technically \"found\" it somewhere on the page.\n\n**Faithfulness, simple example:** imagine grading a student's essay by checking, sentence by sentence, whether each individual claim they made is actually backed up by the assigned textbook. If the student writes five sentences and four of them are directly supported by the textbook but the fifth one is something they simply made up (that isn't in the textbook at all), you wouldn't give them full credit — you'd score the essay based on the *fraction* of sentences that are actually grounded in the source material. That's exactly what \"faithfulness\" measures for an LLM's generated answer.\n\nThe motivation, per the notes: **naive RAG evaluation is just qualitative — \"does it seem right?\" — and manual feedback loops are slow.** That doesn't scale, and it's subjective. **RAGAS** (a plain-English one-liner: *\"a framework and library for evaluating RAG pipelines using automated, LLM-assisted metrics instead of manual human judgment\"*) fixes two specific pain points named in the notes: **no need for human-annotated ground truth**, and it lets you **verify that changes/optimizations to your pipeline don't quietly downgrade quality** (a regression-testing mindset applied to RAG quality, not just code correctness).\n\n**Context Precision** — measures the **retriever's** (or reranker's) **ability to rank relevant chunks higher**, and it directly **penalizes systems that bury relevant content below noise** — this is precisely the **\"lost in the middle\" problem** flagged back in Section 6. The method: use an LLM to label each retrieved chunk as **RELEVANT** or **NOT relevant**, and then, for every position `k`\n\nin the ranked list, compute **precision@k** — the fraction of the *top k* results that are relevant.\n\n**The notes' worked example**, using a 5-result list marked (green = relevant, red = not relevant): `🟢 🔴 🟢 🔴 🔴`\n\n`p@2 = 0`\n\n. Reading the notes' own math carefully: **Final Context Precision formula (from the notes' own arithmetic):**\n\n```\nContext Precision = (p@1 + p@3) / (# relevant chunks)\n                   = (1 + 0.67) / 2\n                   = 1.67 / 2\n                   = 0.835\n```\n\nIn words: you sum up the precision@k values *at each rank where a relevant chunk appears*, then divide by the **total number of relevant chunks** in the list (here, 2 relevant chunks out of 5 total). This rewards systems that put their relevant chunks *early* — a relevant chunk found at rank 1 contributes a full 1.0, while one found deeper in the list contributes a smaller fraction, exactly reflecting the \"don't bury the good stuff\" intuition from the simple example.\n\n**Faithfulness Score** — measures **factual consistency of the generated answer**, and is explicitly called a **\"measure of hallucination\"** in the notes. Method: **break the answer into atomic claims** (using an LLM — the same \"claim extraction\" step from the read-path pseudocode in Section 9.4), then for each claim, score it **1 if the retrieved context supports the claim**, or **0 if the context does not support the claim**.\n\n**Faithfulness formula:**\n\n```\nfaithfulness = (# claims supported) / (# claims)\n```\n\nThis is the exact same underlying idea as the `nli_entailed()`\n\ncheck baked into the \"zero hallucinations\" `respond()`\n\nfunction from Section 9 — faithfulness is essentially the *evaluation-time metric version* of the same *runtime* guardrail.\n\n```\n# =============================================================================\n# CONTEXT PRECISION -- from a list of relevant/irrelevant flags\n# (the \"green dot / red dot\" example from the notes).\n# =============================================================================\n\ndef context_precision(relevance_flags: list[bool]) -> float:\n    \"\"\"\n    relevance_flags: e.g. [True, False, True, False, False]\n                      -- True = relevant chunk, False = not relevant,\n                      in the ORDER they were retrieved/ranked (best first).\n\n    Returns the RAGAS-style Context Precision score.\n    \"\"\"\n    total_relevant = sum(relevance_flags)\n    if total_relevant == 0:\n        return 0.0  # no relevant chunks at all -- precision is undefined/zero\n\n    running_relevant_count = 0\n    precision_sum = 0.0\n\n    for k, is_relevant in enumerate(relevance_flags, start=1):\n        if is_relevant:\n            running_relevant_count += 1\n            # precision@k = (# relevant chunks in top k) / k\n            precision_at_k = running_relevant_count / k\n            precision_sum += precision_at_k\n        # if NOT relevant, this rank contributes 0 and is skipped, matching\n        # the notes' worked example where p@2, p@4, p@5 = 0\n\n    return precision_sum / total_relevant\n\n# --- Example matching the notes exactly: 🟢 🔴 🟢 🔴 🔴 ----------------------\nflags = [True, False, True, False, False]\nscore = context_precision(flags)\nprint(f\"Context Precision: {score:.3f}\")  # -> should print 0.835\n\n# =============================================================================\n# FAITHFULNESS -- given a list of claims and whether each is supported.\n# =============================================================================\n\ndef faithfulness_score(claim_support_labels: list[bool]) -> float:\n    \"\"\"\n    claim_support_labels: e.g. [True, True, True, False]\n                           -- True = context supports this atomic claim,\n                           False = context does NOT support it (hallucinated).\n    \"\"\"\n    if not claim_support_labels:\n        return 1.0  # no claims made -- vacuously faithful (nothing to check)\n\n    supported = sum(claim_support_labels)\n    total = len(claim_support_labels)\n    return supported / total\n\n# --- Example: an answer with 4 atomic claims, 1 of which is unsupported ----\nclaims_supported = [True, True, True, False]\nfaith_score = faithfulness_score(claims_supported)\nprint(f\"Faithfulness: {faith_score:.3f}\")  # -> 0.75\npython\n# pip install ragas datasets --break-system-packages\nfrom ragas import evaluate\nfrom ragas.metrics import context_precision, faithfulness\nfrom datasets import Dataset\n\n# RAGAS expects a dataset shaped like this: one row per question, with the\n# retrieved contexts and the generated answer already filled in.\neval_dataset = Dataset.from_dict({\n    \"question\": [\"Why is my API returning a 500 error under high load?\"],\n    \"answer\": [\"500 errors under high load are typically caused by connection \"\n               \"pool exhaustion or unhandled exceptions from resource contention.\"],\n    \"contexts\": [[\n        \"Throughput degradation under sustained load can trigger unhandled \"\n        \"exceptions, which surface to callers as HTTP 500 errors.\",\n        \"Connection pool exhaustion is a common cause of request failures \"\n        \"under high concurrency.\",\n    ]],\n    # ground_truth is optional for some metrics but required for others (e.g. recall)\n    \"ground_truth\": [\"500 errors under load are usually caused by connection \"\n                     \"pool exhaustion or unhandled exceptions.\"],\n})\n\nresults = evaluate(eval_dataset, metrics=[context_precision, faithfulness])\nprint(results)\n# This runs the SAME conceptual computation as the from-scratch functions\n# above, but uses an LLM internally to (a) judge chunk relevance for\n# context_precision, and (b) extract + verify atomic claims for faithfulness.\n```\n\n", "url": "https://wpnews.pro/news/tool-use-and-rag-in-production-a-complete-beginner-s-study-guide", "canonical_source": "https://dev.to/digital_subham/tool-use-and-rag-in-production-a-complete-beginners-study-guide-379i", "published_at": "2026-07-12 18:11:27+00:00", "updated_at": "2026-07-12 18:45:26.036282+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "developer-tools"], "entities": ["Claude", "GPT", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/tool-use-and-rag-in-production-a-complete-beginner-s-study-guide", "markdown": "https://wpnews.pro/news/tool-use-and-rag-in-production-a-complete-beginner-s-study-guide.md", "text": "https://wpnews.pro/news/tool-use-and-rag-in-production-a-complete-beginner-s-study-guide.txt", "jsonld": "https://wpnews.pro/news/tool-use-and-rag-in-production-a-complete-beginner-s-study-guide.jsonld"}}