This guide assumes you know nothing about LLMs beyond "I can chat with ChatGPT/Claude." Every concept is introduced with a real-world analogy first, then explained technically, then backed by runnable code.
Imagine you ask a friend, "Hey, what's the weather in Tokyo right now?" Your friend doesn't know this off the top of their head — nobody just knows live weather. So your friend pulls out their phone, opens a weather app, types "Tokyo," reads the result, and tells you: "It's 24°C and cloudy."
Notice what happened: your friend (the "brain" doing the reasoning) never became a weather sensor. They recognized they needed external, live information, used a tool (the weather app) to fetch it, and then used that fetched information to answer you in natural language.
This is exactly what "tool use" (also called "function calling") means for an LLM like Claude or GPT. The model is the friend. It's great at reasoning and language, but it has no live connection to today's weather, your company's database, or a calculator that never makes arithmetic mistakes. So we give it a list of "apps" (tools) it's allowed to ask for, and something else — your application code — actually goes and uses those apps on its behalf.
Tool use extends a model's capabilities to call "external systems" — anything outside the model's own weights and training data. Examples from the notes:
These are all things a language model is fundamentally bad at doing internally (it can't refresh live weather data, and it's notoriously unreliable at exact arithmetic), so instead of asking the model to guess, we let it delegate to a real, deterministic function.
How it's wired up:
get_weather
with {"city": "Tokyo"}
."The single most important sentence in this whole section, straight from the notes:
"The model never calls a function itself!"
The model's role is reasoning and argument construction only. It cannot reach out to the internet, touch your database, or execute code. Everything it does is emit text/JSON. Your application is the one with hands.
The full lifecycle (the loop), step by step:
get_weather("Tokyo")
for real — hits a weather API, database, calculator, whatever the tool represents.Along the way your application also has to handle practical realities like rate limits on the tools you're calling (e.g., a weather API only allows so many requests per minute) — that's your application's job too, not the model's.
Below is a minimal, close-to-runnable example using the Anthropic SDK style referenced in the notes. It defines a get_weather
tool, sends a prompt, receives a tool_use
block, executes a mock function (standing in for a real weather API), and sends the result back so the model can finish its answer.
import anthropic
import json
client = anthropic.Anthropic()
tools = [
{
"name": "get_weather",
"description": "Get the current weather for a given city.",
"input_schema": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name, e.g. Tokyo"},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "Temperature unit to return",
},
},
"required": ["city"], # unit is optional
},
}
]
def get_weather(city: str, unit: str = "celsius") -> dict:
"""Pretend weather lookup — replace with a real HTTP call in production."""
fake_database = {"Tokyo": 24, "London": 14, "Delhi": 33}
temp_c = fake_database.get(city, 20)
temp = temp_c if unit == "celsius" else (temp_c * 9 / 5) + 32
return {"city": city, "temperature": temp, "unit": unit}
messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=messages,
)
tool_use_block = None
for block in response.content:
if block.type == "tool_use":
tool_use_block = block
break
if tool_use_block:
print(f"Model wants to call: {tool_use_block.name} with {tool_use_block.input}")
if tool_use_block.name == "get_weather":
result = get_weather(**tool_use_block.input)
else:
result = {"error": "unknown tool"}
messages.append({"role": "assistant", "content": response.content})
messages.append(
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": tool_use_block.id,
"content": json.dumps(result),
}
],
}
)
final_response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=messages,
)
for block in final_response.content:
if block.type == "text":
print("Final answer:", block.text)
else:
for block in response.content:
if block.type == "text":
print("Final answer:", block.text)
Notice the loop shape: model decides → app intercepts → app invokes → app returns result → model proceeds. That five-step handoff is the entire mental model for tool use, and it never changes no matter how complex the tool is.
Imagine you're hiring a new employee and you hand them a one-line job description: "Search for products." That's it — no other instructions.
On day one, a customer asks them "how much does this cost?" and the new employee, trying to be helpful, uses the "search for products" tool because it's the only tool they were told about, even though there's actually a dedicated pricing tool sitting right next to it that they didn't know they should prefer. They also aren't told whether "search" means searching by name, by category, or by a product code — so they guess, and sometimes guess wrong.
Now imagine instead you hand them a proper SOP: "Search for products by name, category, or SKU. Use this only when the customer is explicitly trying to find or filter catalog items. Do NOT use this for pricing questions or stock checks — those have their own dedicated tools." Now the employee has almost no room to misinterpret their job.
This is the difference between a bad tool schema and a good one. The LLM is that new employee, and the schema is the only "training" it gets before being thrown into the job.
From the notes: "JSON schema is [the] contract between your intent and the model's behaviour." And critically: "Ambiguous schema is [the] most common root cause of tool failure and is 'invisible' until production." It's invisible because in your own testing you tend to ask clean, obvious questions — production users ask messy, ambiguous, unexpected ones, and that's where a vague schema quietly breaks.
There are three rules for designing a good tool schema:
Rule 1 — Specify semantics, not just a description.
Don't just say what the function is named or roughly does. Say when to use it, when not to use it, and what it should never be confused with. The notes' worked example:
"search for products"
"Search for products by name, category, or SKU. Use only when user is explicitly looking to find or filter catalog items. Do not use for pricing questions or stock checks — those have other dedicated tools."
Rule 2 — Be deliberate about required vs. optional parameters.
If a parameter is optional and you don't clearly explain what happens when it's omitted, the model may hallucinate a value for it just to fill the slot (marked with a red ✗ in the notes). Conversely, if a parameter is marked mandatory but the model doesn't have a value for it, it may pass nothing at all or an invalid placeholder (also marked ✗). The fix from the notes: provide default values for the model to use so there's no ambiguity about what an "empty" optional parameter should look like.
Rule 3 — Enums are the cheapest reliability primitive you have.
Whenever a parameter has a fixed, known set of valid values (a category, a unit, a status), constrain it with an enum instead of leaving it as a free-form string. This "keeps things readable and reduces hallucination" — the model can only pick from the list you gave it, it can't invent "CategoryXYZ"
out of thin air.
The "too many tools" problem, and its fix — dynamic tool .
The notes flag: "large number of tools: model gets confused" — when a model is handed 50+ tools at once, two hard questions creep in: "What category tool?" and "Which specific tool?" The model starts guessing between similar-looking tools. The fix is **dynamic tool **: instead of always sending the model your entire tool catalog, your application first figures out (from context, intent classification, or the conversation's domain) which subset of tools is actually relevant, and only sends that subset to the model for this particular turn. Fewer, more relevant choices means fewer wrong guesses.
bad_tool = {
"name": "search_products",
"description": "Search for products", # <-- too vague: search how? for what purpose?
"input_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"category": {"type": "string"}, # <-- free text: model may invent categories
"include_out_of_stock": {"type": "boolean"}, # <-- optional, no default explained
},
"required": ["query"],
},
}
good_tool = {
"name": "search_products",
"description": (
"Search for products by name, category, or SKU. "
"Use only when the user is explicitly looking to find or filter catalog "
"items. Do NOT use this for pricing questions or stock-level checks — "
"those have their own dedicated tools (get_price, check_stock)."
),
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Free-text search term, e.g. a product name or SKU.",
},
"category": {
"type": "string",
"enum": ["electronics", "apparel", "home", "books", "toys"],
"description": "Optional. Restrict results to one catalog category.",
},
"include_out_of_stock": {
"type": "boolean",
"description": (
"Whether to include out-of-stock items. "
"If omitted, DEFAULTS to false (only show in-stock items)."
),
},
},
"required": ["query"],
},
}
ALL_TOOLS = {
"search_products": {...}, # schema dicts omitted for brevity
"get_price": {...},
"check_stock": {...},
"get_weather": {...},
"run_bash_command": {...},
"search_hr_docs": {...},
}
def select_relevant_tools(user_message: str) -> list[str]:
msg = user_message.lower()
if any(word in msg for word in ["price", "cost", "how much"]):
return ["get_price"]
if any(word in msg for word in ["stock", "available", "in stock"]):
return ["check_stock"]
if any(word in msg for word in ["find", "search", "looking for"]):
return ["search_products"]
if "weather" in msg:
return ["get_weather"]
return ["search_products", "get_price"]
def build_tools_for_request(user_message: str) -> list[dict]:
relevant_names = select_relevant_tools(user_message)
return [ALL_TOOLS[name] for name in relevant_names if name in ALL_TOOLS]
Suppose you ask a travel-planning friend: "Can you check the weather in Paris, Tokyo, and New York for me?" A slow friend would check Paris, wait, tell you, then check Tokyo, wait, tell you, then check New York. A smart friend instead opens three browser tabs at once, checks all three simultaneously, and gives you all three answers together. Same total work, far less waiting.
Now imagine one of those three lookups fails — say the New York weather site is down. Does your friend refuse to tell you about Paris and Tokyo just because New York failed? Or do they hand you what they do have, and mention New York didn't work? That's a judgment call — and it's the same judgment call your application has to make about tool calls.
Parallel tool calls: the notes describe a model that can emit multiple tool-call requests in a single turn — e.g., "compare weather across N cities" or a "trip plan" that needs both search_flight
and search_hotel
at once. Your orchestration layer (the application code, not the model) is responsible for "fanning out" these calls concurrently, collecting all the results, and sending them back together. The direct payoff: it improves overall time to completion — you pay for the slowest of the N calls, not the sum of all N. The notes' guidance: use this whenever the operations are independent of one another (one call's result doesn't depend on another's).
Failure handling — "it depends". The notes are refreshingly honest here: "Answer to any question could well be IT DEPENDS." There isn't one universal right answer; it's a judgment call based on your product's tolerance for incompleteness vs. correctness. Three named strategies:
The notes give a concrete example of when parallel fan-out matters: "4 websearch calls, lookup across multiple knowledge bases." If a question needs facts from 4 different knowledge bases, firing all 4 lookups concurrently instead of one after another is the entire difference between a fast, responsive system and a sluggish one.
import asyncio
import random
async def fetch_weather(city: str) -> dict:
await asyncio.sleep(random.uniform(0.1, 0.5)) # simulate network latency
if city == "New York" and random.random() < 0.5:
raise ConnectionError(f"Weather API timed out for {city}")
return {"city": city, "temp_c": random.randint(10, 35)}
SUPPORTED_CITIES = {"Paris", "Tokyo", "New York", "London"}
async def call_with_prevention(city: str) -> dict:
if city not in SUPPORTED_CITIES:
return {"city": city, "error": "unsupported_city", "skipped_call": True}
return await fetch_weather(city)
async def call_with_absorption(cities: list[str]) -> dict:
tasks = [fetch_weather(city) for city in cities]
results = await asyncio.gather(*tasks, return_exceptions=True)
successes, failures = [], []
for city, result in zip(cities, results):
if isinstance(result, Exception):
failures.append({"city": city, "error": str(result)})
else:
successes.append(result)
return {"successful_results": successes, "failed_cities": failures}
async def call_with_graceful_failure(cities: list[str]) -> dict:
tasks = [fetch_weather(city) for city in cities]
try:
results = await asyncio.gather(*tasks) # no return_exceptions: first failure raises
return {"ok": True, "results": results}
except Exception as exc:
return {
"ok": False,
"refused": True,
"reason": f"Could not complete weather comparison: {exc}",
}
async def main():
cities = ["Paris", "Tokyo", "New York"]
print("PREVENT strategy (one unsupported city):")
print(await asyncio.gather(*[call_with_prevention(c) for c in ["Paris", "Atlantis"]]))
print("\nABSORB strategy (best-effort, partial results ok):")
print(await call_with_absorption(cities))
print("\nFAIL GRACEFULLY strategy (all-or-nothing):")
print(await call_with_graceful_failure(cities))
if __name__ == "__main__":
asyncio.run(main())
Which strategy to pick is a product decision, not a technical one — that's the "it depends" in the notes. A weather-comparison feature can tolerate "absorb" (2 out of 3 cities is still useful). A financial transaction pipeline probably needs "fail gracefully" (a half-completed transfer is dangerous, not just incomplete).
Say you're searching your email for something. If you search "birthday party invite," a smart search that understands meaning (semantic search) will find an email titled "Come celebrate turning 30!" even though none of your exact words appear in it — because it understands the concept of a birthday party.
But now say you search for an exact error code, like 1099-MISC
. A meaning-based search might get confused and return generic tax documents about "how do I reset my password" style topics — it doesn't realize you need an exact string match, not a conceptual one. A classic old-school keyword search (like Ctrl+F, but smarter) would nail this instantly because it's built for exact-term matching.
Neither approach alone is good at everything. The fix: run both kinds of search at the same time, and combine (merge and rerank) their results — a bit like asking two friends with different strengths for restaurant recommendations, and combining their two rankings into one better, more trustworthy ranking.
Dense retrieval (semantic search): your query and your documents are both converted into vectors ("embeddings") that capture meaning. Two pieces of text that mean similar things end up as vectors that are close together in space, even if they don't share any exact words. This is great when there's genuine semantic similarity — the notes' example: "How do I reset my password?" naturally matches a document titled "Account Recovery Steps," even with zero overlapping words.
Sparse retrieval / BM25: this is the notes' name for traditional keyword-based search ("BM25" = "Best Matching 25," a well-known statistical formula scoring how well a document's exact terms match a query's exact terms, weighted by term rarity and frequency). BM25 is the opposite strength of dense search: it's excellent at exact terms, codes, IDs, acronyms — things like "error code 1099-MISC" — but it has no concept of meaning, so it's poor at "How do I reset my password?" style natural-language questions where the right document doesn't share your exact wording.
Hybrid search = dense + sparse, run together. The notes' rule of thumb: "20|50 documents certainly having the right answer" — hybrid retrieval improves recall (the fraction of genuinely relevant documents that get retrieved at all), because you're covering both failure modes (semantic gap AND exact-term gap) simultaneously. The fix line from the notes: "Run both and merge and rerank!"
Reciprocal Rank Fusion (RRF) is the merging technique. Analogy: imagine two friends each independently rank the same 10 restaurants from 1 (best) to 10 (worst). To get one combined ranking that reflects both opinions, you don't just average their scores (their internal scoring "scales" may be totally different: one friend rates out of 5 stars, another out of 100 points). Instead, RRF says: only the rank position matters, not the internal score — a restaurant ranked #1 by both friends should shoot to the top of the combined list, even if their raw scores were on incompatible scales. This "brings reciprocal ranks together" as the notes put it.
The formula, straight from the notes:
RRF(d) = Σ 1 / (k + rank_r(d))
r∈R
Read this piece by piece:
d
is a particular document you're scoring.R
is the set of ranked result lists you're fusing (in our case: the dense-search ranking and the BM25 ranking).rank_r(d)
is the position of document d
in ranking r
(1st place, 2nd place, etc.).k
is a small constant, 1 / (k + rank)
for The notes' visual intuition: dense list gives fractions like 1/1, 1/2, ...
and sparse list gives 1/61, 1/62, ...
(i.e., 1/(60+1), 1/(60+2), ...
) — "Fusion is better" because a document doesn't need to win outright in either individual list; it just needs to consistently rank reasonably well across both to accumulate a high combined score.
One more practical note from the notes: Weaviate and Elasticsearch (ES) support hybrid search natively — meaning they've built RRF-style fusion directly into the database. Otherwise, you have to do it yourself with a scatter-gather pattern (fire off both queries yourself, gather both result sets back in your application, then fuse them).
def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
"""
ranked_lists: a list of ranked result lists. Each inner list is already
sorted best-to-worst, e.g. ["doc3", "doc1", "doc9"].
k: the RRF damping constant (60 is the standard default from the notes).
Returns a list of (doc_id, fused_score) sorted from best to worst.
"""
scores: dict[str, float] = {}
for ranked_list in ranked_lists:
for position, doc_id in enumerate(ranked_list):
rank = position + 1 # ranks start at 1, not 0
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
fused_ranking = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
return fused_ranking
dense_results = ["account_recovery_steps", "login_faq", "security_settings", "billing_faq"]
sparse_results = ["login_faq", "account_recovery_steps", "terms_of_service", "billing_faq"]
fused = reciprocal_rank_fusion([dense_results, sparse_results], k=60)
for doc_id, score in fused:
print(f"{doc_id}: {score:.5f}")
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np
documents = [
"To reset your password, go to Account Recovery and click 'Forgot Password'.",
"Error code 1099-MISC indicates a mismatch in the tax reporting form.",
"Our billing FAQ covers refunds, invoices, and subscription changes.",
]
doc_ids = ["account_recovery", "error_1099_misc", "billing_faq"]
query = "how do I reset my password"
tokenized_corpus = [doc.lower().split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.lower().split())
sparse_ranking = [doc_ids[i] for i in np.argsort(bm25_scores)[::-1]]
embedder = SentenceTransformer("all-MiniLM-L6-v2") # small, fast embedding model
doc_embeddings = embedder.encode(documents, normalize_embeddings=True)
query_embedding = embedder.encode(query, normalize_embeddings=True)
cosine_scores = doc_embeddings @ query_embedding # dot product of normalized vectors = cosine similarity
dense_ranking = [doc_ids[i] for i in np.argsort(cosine_scores)[::-1]]
print("Sparse (BM25) ranking:", sparse_ranking)
print("Dense (semantic) ranking:", dense_ranking)
final_ranking = reciprocal_rank_fusion([dense_ranking, sparse_ranking], k=60)
print("Final hybrid ranking:", final_ranking)
Imagine you're searching your email for a specific message and you know it arrived last week. You could (a) tell your email client "only show me emails from last week," and then search the text — that's fast and narrow. Or you could (b) search the text across your entire mailbox history, get back thousands of results, and manually throw away everything not from last week — that's slow and wasteful, since your search engine did a ton of unnecessary work scanning old emails it was always going to discard.
Now a second, more serious example: at a company, an HR employee should be able to see every department's HR documents, but a regular software engineer should only see documents relevant to their own team — they should never even see, let alone search inside, another department's confidential HR files. This isn't just a nice-to-have; it's a hard security requirement, often called RBAC (Role-Based Access Control — a plain-English one-liner: "restricting what a user can see or do based on their assigned role").
Metadata filtering applies hard constraints (like date ranges, document owner, department, tags) either before or after the vector search runs, on top of whatever relevance ranking your retrieval produces. The notes' example query: "What was our Q1 2026 revenue?" — you'd want to filter to date range = Q1 2026
before even letting semantic search loose on the whole corpus.
Beyond just retrieval quality, the notes stress this is especially important for Enterprise RAG and RBAC: an employee simply must not be able to retrieve another department's HR docs, no matter how semantically relevant those docs might look to their query. This is a security boundary, not just a relevance tweak.
There are two places you can apply the filter:
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
client = QdrantClient(url="http://localhost:6333") # assumes a running Qdrant instance
query_vector = [0.12, 0.98, 0.31] # a real embedding would be much longer, e.g. 1536-dim
results = client.search(
collection_name="hr_documents",
query_vector=query_vector,
query_filter=Filter(
must=[
FieldCondition(key="department", match=MatchValue(value="engineering")),
FieldCondition(key="date", range=Range(gte="2026-01-01", lte="2026-03-31")),
]
),
limit=10,
)
def post_filter_search(all_search_results: list[dict], user_department: str) -> list[dict]:
"""
all_search_results: e.g. the top-K matches from an UNFILTERED vector search
across the entire corpus, regardless of department.
"""
filtered = [
doc for doc in all_search_results
if doc["department"] == user_department
and "2026-01-01" <= doc["date"] <= "2026-03-31"
]
return filtered
Bi-encoder analogy: imagine two strangers each write their own short bio independently — "I like hiking, sci-fi movies, and cooking Italian food" and "I enjoy the outdoors, watching space films, and making pasta." You then compare these two bios after the fact to guess how compatible these two people might be. It's fast (you can pre-write and store millions of bios ahead of time and just compare later), but it's a bit shallow — the two people never actually talked to each other, so subtle nuances get missed.
Cross-encoder analogy: now imagine instead you sit those same two strangers down together for an actual conversation. They can react to each other in real time, clarify ambiguous statements, and pick up on nuance that never would have shown up in two separately-written bios. You get a much better read on true compatibility — but it only works one pair at a time, and a live conversation takes real time and effort per pair. You couldn't have every person in a stadium have a real conversation with every other person — it simply wouldn't scale.
Bi-encoders (used for the first stage of dense retrieval, described back in Section 4): the query is turned into an embedding, and each document is turned into an embedding, completely independently of each other. Relevance is then just vector proximity — how close the two embeddings are (measured via cosine similarity). Because documents can be embedded once, offline, ahead of time, bi-encoders are fast at query time and scalable — this is why they're "good enough for most" use cases, per the notes.
Their key weakness, from the notes: "No cross attention, i.e. query and doc never see each other." Because the two embeddings were produced in isolation, the model can't attend to specific interactions between query terms and document terms. This makes bi-encoders weaker on negation, exact terms, etc. — e.g., a bi-encoder might not clearly distinguish "the flight is NOT delayed" from "the flight IS delayed," because both sentences produce very similar embeddings overall (they share almost all the same words).
Cross-encoders, by contrast, take the ** <query, document> pair as a single input** — literally concatenating the query and the candidate document together — and output
The notes draw a small attention diagram illustrating the "lost in the middle" problem: attention/relevance tends to concentrate most heavily near the query token and decays as you move deeper into a long document — meaning information buried in the middle of a long document or a long context window can get comparatively under-weighted relative to information at the start or end. This is a known failure mode of transformer attention over long inputs, and it's part of why reranking with a small, focused candidate set (rather than dumping a giant unranked context at the LLM) genuinely helps.
Cross-encoders: much higher accuracy — they can compare every token in the query to every token in the document, so they handle negation, exact matches, etc. far better than bi-encoders. Example model named in the notes: cross-encoder/ms-marco-MiniLM-L-6-v2
.
The catch — cross-encoders are slow and don't scale. From the notes: they work well on "50 doc[s] v[s] million doc[s] per query" — meaning cross-encoders are only practical on a small shortlist, never the entire corpus. There's no indexing possible (because the score depends on the specific query, you can't precompute it ahead of time the way you can with bi-encoder embeddings), and each pair costs roughly ~10ms of latency.
The latency math, worked out step by step (from the notes):
<query, doc>
pairThat's obviously unusable in a real product. The fix, stated directly in the notes: "could be done on candidate docs only." This is exactly why the two techniques are used together in a pipeline, not as alternatives: use the cheap, scalable bi-encoder (plus BM25/hybrid search) to narrow millions of documents down to a small shortlist (say, the top 40), and then run the expensive, high-accuracy cross-encoder reranker only on that small shortlist. You get bi-encoder speed at scale, and cross-encoder accuracy where it matters most — the final ranking of the few documents that actually make it into the LLM's context.
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np
query = "Why is my API returning a 500 error under high load?"
candidate_docs = [
"Throughput degradation under load can trigger unhandled exceptions, "
"which surface to callers as HTTP 500 errors.",
"To reset your password, click 'Forgot Password' on the login screen.",
"500 errors are NOT related to client-side input validation issues.",
"Our billing FAQ covers refunds, invoices, and subscription cancellations.",
]
bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")
query_vec = bi_encoder.encode(query, normalize_embeddings=True)
doc_vecs = bi_encoder.encode(candidate_docs, normalize_embeddings=True)
bi_encoder_scores = doc_vecs @ query_vec
print("Bi-encoder ranking (fast, first-pass):")
for idx in np.argsort(bi_encoder_scores)[::-1]:
print(f" {bi_encoder_scores[idx]:.3f} {candidate_docs[idx][:60]}...")
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [[query, doc] for doc in candidate_docs] # literal <query, doc> pairing
cross_encoder_scores = cross_encoder.predict(pairs)
print("\nCross-encoder ranking (slow, high-precision rerank):")
for idx in np.argsort(cross_encoder_scores)[::-1]:
print(f" {cross_encoder_scores[idx]:.3f} {candidate_docs[idx][:60]}...")
Imagine you ask a librarian, "Why is my code slow?" The librarian's catalog, though, doesn't use casual words like "slow" — the technical manuals on the shelf are indexed under formal terms like "throughput degradation," "latency regression," or "bottleneck analysis." If the librarian searches the catalog using your exact casual phrase, they might get nothing useful back, even though the perfect manual for your problem exists on the shelf. The words just don't line up.
A clever librarian's trick: before searching, they first jot down (in their head) roughly what the ideal, technical-sounding manual would probably say about your problem — then they search the catalog using that imagined, more formal description instead of your casual phrasing. That imagined "ideal answer" acts as a bridge between your everyday words and the library's formal vocabulary.
That's exactly what HyDE (Hypothetical Document Embedding) does for a RAG system.
The notes call the "Query-Document Gap" the most persistent failure mode in RAG. It shows up because user queries tend to be short and conversational, while the underlying documents/corpus are often semantically distant in their vocabulary — the classic example: "why is my pipeline slow?" (user's words) vs. "Throughput degradation is..." (the document's actual wording). Even though these mean almost the same thing conceptually, their raw embeddings may not be close enough for a naive semantic search to reliably connect them.
There are two named fixes in the notes:
1. Expand or rephrase the user query. Simply ask an LLM: "generate 3 alternate phrasings..." of the user's question, and search using all of them (or a combination), increasing the odds that at least one phrasing's embedding lands close to the real document.
2. HyDE — Hypothetical Document Embedding. Instead of embedding the user's raw, short question, you first ask an LLM to write a short document that would answer this question — the exact prompt style from the notes: "Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like." You then take that generated hypothetical document — not the original question — and embed it. Because this hypothetical answer is written in a style and vocabulary much closer to how real documents in your corpus are likely written, its embedding tends to land much closer to the real relevant document's embedding than the original short, casual question would have.
Technically, HyDE reframes retrieval as a similarity search between d and d ∈ D — that is, comparing a (hypothetical) document to the real documents in your corpus
D
, rather than comparing a short question to documents (which is a fundamentally harder, more mismatched comparison).When does this actually help? The notes are precise about this: HyDE "works when corpus vocab is very different than user query." Good candidates: scientific papers, legal texts, internal jargon-heavy documentation — domains where the "real" documents are written in dense, formal, or highly technical language that a typical user's question would never naturally use.
import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np
client = anthropic.Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def generate_hypothetical_answer(user_query: str) -> str:
"""
Ask an LLM to write a short, FACTUAL-SOUNDING document that would answer
the user's question. This is the exact prompt shape from the notes:
"Write a short document that will answer this question. Be factual.
If you don't know the answer, write what the answer would typically
look like."
We use this generated text purely to bridge the vocabulary gap — we do
NOT show this hypothetical text to the end user; it's an internal
retrieval-only artifact.
"""
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=300,
messages=[
{
"role": "user",
"content": (
"Write a short document that will answer this question. "
"Be factual. If you don't know the answer, write what the "
f"answer would typically look like.\n\nQuestion: {user_query}"
),
}
],
)
return response.content[0].text
def hyde_retrieve(user_query: str, doc_texts: list[str], doc_ids: list[str], top_k: int = 3):
"""
Full HyDE retrieval: generate a hypothetical answer, embed THAT instead
of the raw query, then find the closest real documents to it.
"""
hypothetical_doc = generate_hypothetical_answer(user_query)
hyde_embedding = embedder.encode(hypothetical_doc, normalize_embeddings=True)
doc_embeddings = embedder.encode(doc_texts, normalize_embeddings=True)
similarities = doc_embeddings @ hyde_embedding
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(doc_ids[i], doc_texts[i], float(similarities[i])) for i in top_indices]
corpus_texts = [
"Throughput degradation under sustained load is typically caused by "
"connection pool exhaustion or unindexed database queries.",
"Our refund policy allows cancellations within 30 days of purchase.",
"Latency regressions in distributed systems often trace back to N+1 "
"query patterns or missing caching layers.",
]
corpus_ids = ["perf_doc_1", "refund_policy", "perf_doc_2"]
results = hyde_retrieve("why is my code slow?", corpus_texts, corpus_ids)
for doc_id, text, score in results:
print(f"{score:.3f} {doc_id}: {text[:70]}...")
Imagine ten different customers each phrase the exact same underlying question in slightly different words: "How do I integrate libX?", "I want help integrating libX," "How can I get libX working with my app?" A naive cache (the kind that only matches exact strings) would treat these as three totally unrelated questions and pay to compute a fresh, expensive answer for every single one — even though the right answer is identical for all three.
A smarter cache recognizes that all three questions mean the same thing, and serves the same cached, already-computed answer to all of them — saving time and money on the second and third askers, without them ever noticing anything different.
The core motivating fact from the notes: "Every call to LLM cost[s] latency and tokens" — i.e., time and money. A semantic cache sits in front of the LLM: it intercepts a query before it reaches the model and, if a semantically similar (not necessarily an exact-text-match) prior query exists in the cache, it returns the cached response instead of paying for a new LLM call.
When this works best, per the notes: static data with no personalization — situations where the "right" answer genuinely doesn't change per-user or over short timeframes. Two named examples:
Implementation, step by step (from the notes):
The core tension: cost vs. correctness, governed entirely by what similarity threshold you pick and how you set it. The notes are blunt about the two failure directions:
The reliable way to pick a threshold, per the notes: don't guess — empirically find your "sweet spot" on your own query distribution. As a reasonable starting point, the notes suggest 0.85 cosine similarity, but this must be calibrated, not treated as gospel, because every product's query patterns differ.
from sentence_transformers import SentenceTransformer
import numpy as np
class SemanticCache:
"""
A minimal in-memory semantic cache: get() looks for a semantically
similar prior query above `similarity_threshold`; set() stores a new
query + response pair for future lookups.
"""
def __init__(self, similarity_threshold: float = 0.85):
self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
self.similarity_threshold = similarity_threshold
self.cached_queries: list[str] = []
self.cached_embeddings: list[np.ndarray] = []
self.cached_responses: list[str] = []
def get(self, query: str) -> str | None:
"""Return a cached response if a semantically similar query exists, else None."""
if not self.cached_queries:
return None
query_embedding = self.embedder.encode(query, normalize_embeddings=True)
similarities = np.array(self.cached_embeddings) @ query_embedding
best_idx = int(np.argmax(similarities))
best_score = float(similarities[best_idx])
if best_score >= self.similarity_threshold:
print(f"[cache HIT] similarity={best_score:.3f} matched: "
f"'{self.cached_queries[best_idx]}'")
return self.cached_responses[best_idx]
print(f"[cache MISS] best similarity was only {best_score:.3f}")
return None
def set(self, query: str, response: str) -> None:
"""Store a new query + response pair in the cache."""
embedding = self.embedder.encode(query, normalize_embeddings=True)
self.cached_queries.append(query)
self.cached_embeddings.append(embedding)
self.cached_responses.append(response)
cache = SemanticCache(similarity_threshold=0.85)
cache.set("How do I integrate libX?", "To integrate libX, install it via pip and call libx.init()...")
print("\n--- Reasonable threshold (0.85) ---")
cache.get("I want help integrating libX") # should HIT: same intent, different phrasing
cache.get("What's the weather in Tokyo?") # should MISS: unrelated question
print("\n--- Threshold set TOO LOW (0.50): risks wrong answers ---")
loose_cache = SemanticCache(similarity_threshold=0.50)
loose_cache.set("How do I cancel my subscription?", "Go to Settings > Billing > Cancel Plan.")
loose_cache.get("How do I upgrade my subscription plan?")
print("\n--- Threshold set TOO HIGH (0.99): hit rate collapses ---")
strict_cache = SemanticCache(similarity_threshold=0.99)
strict_cache.set("How do I integrate libX?", "To integrate libX, install it via pip...")
strict_cache.get("I want help to integrate libX")
The prototype in the notes ("Semantic Caching and Similarity Calibration") describes exactly this experiment at scale: 500 queries → embeddings → index, then sweeping thresholds from 0.75 to 0.95, populating the cache first, then computing hit rate (fraction of queries found in cache) and accuracy/precision. It uses three types of test queries: seed (the original query, e.g. "Explain the ending of movie Arrival"), paraphrase (same intent, different words, e.g. "Please clarify the conclusion of the movie Arrival" — a correct match), and near-miss (looks similar but has a genuinely different intent, e.g. "Who directed the movie Arrival" — a trap, since matching this to the seed's answer would be a false positive / incorrect hit). The accuracy formula from the notes:
accuracy = correct hits (paraphrase matching the seed) / total hits
Below is a small script that runs this exact style of calibration experiment:
def evaluate_threshold(cache: SemanticCache, seed_paraphrase_pairs, near_miss_queries):
"""
seed_paraphrase_pairs: list of (seed_query, paraphrase_query, response) tuples
-- paraphrase SHOULD correctly hit the seed's cached entry.
near_miss_queries: list of queries that LOOK similar but have different intent
-- these SHOULD ideally miss (or if they hit, that's a false positive).
"""
for seed, response in {(s, r) for s, _, r in seed_paraphrase_pairs}:
cache.set(seed, response)
total_hits, correct_hits = 0, 0
for seed, paraphrase, _ in seed_paraphrase_pairs:
result = cache.get(paraphrase)
if result is not None:
total_hits += 1
correct_hits += 1 # paraphrase hitting its own seed = correct
false_positives = 0
for near_miss in near_miss_queries:
result = cache.get(near_miss)
if result is not None:
total_hits += 1
false_positives += 1 # near-miss hitting ANY cached entry = wrong
accuracy = correct_hits / total_hits if total_hits else 0
return {"hit_rate_relevant": correct_hits, "false_positives": false_positives, "accuracy": accuracy}
Imagine building a "help desk" for a huge company with 10 million internal documents. A visitor walks up and asks a question in plain English. Before answering, your help desk clerk must: (1) understand what's really being asked, (2) go search both the exact-keyword index and the meaning-based index of every document, (3) combine and re-sort those results by true relevance, (4) double-check that the results are actually confident and trustworthy (not just "vaguely related"), (5) draft an answer strictly grounded in what was found, and — critically — (6) go back and check every single sentence of that draft answer against the source documents before saying it out loud, refusing to say anything it can't verify. That's the whole system in one sentence: retrieve carefully, answer carefully, and never say something you can't prove.
What the system does: Answers natural language queries. The one hard guarantee that shapes the entire design: "the system must not assert any claim that cannot be traced to [a] retrieved passage." This single sentence is why "correctness" (not just relevance, not just speed) drives nearly every architectural decision below. Why this guarantee matters: an internal knowledge system that confidently states wrong facts is often worse than one that admits "I don't know," because wrong confident answers erode trust and can cause real harm (wrong policy info, wrong financial numbers, etc.).
What the system explicitly is NOT: a document management system, an ingestion pipeline product, etc. — the notes flag this so the design stays scoped: we are not building a general-purpose CMS, just the "ask questions about documents that already exist somewhere" layer.
Scale numbers (why they matter):
The notes lay out the exact ordered checklist a systems-design interview or real design doc would walk through:
Let's walk through each, grounding every architectural choice in a "why" before the "what."
1 & 2 — Storage location and Query API. Documents live in an S3 bucket (e.g. s3://myrag/doc1.txt
), organized per the notes with some inspiration from Git's object storage model (content-addressed, organized by structure like date
/git
-style hashing prefixes, e.g. folders ab/
, ac/
, ad/
— a common trick to avoid dumping literally 10 million flat files into one folder, which many filesystems and object stores handle poorly). The query API is a simple HTTP endpoint: GET /query?
— the notes sketch this as @app.get('/query?')
.
3 — Response shape. From the notes, a single respond()
call returns a structured object containing:
question: <string>
filters: { date_from, date_to, source_domains, tag, topics, etc. }
answer: <string>
refused: <true/false>
refusal_reason: <string, if refused>
passages: [ {doc_id, passage_id, text, score, highlight_idx}, ... ]
confidence: <number>
query_id: <string> -- used for tracing/debugging a specific request later
Why this shape: it's not enough to return just an answer string. You need the passages (so a user or auditor can verify the claim was actually grounded), a refused boolean plus reason (so "I don't know" is a first-class, explicit outcome, not a vague hedge buried in the answer text), a confidence score (so downstream systems or UI can decide whether to trust/display the answer), and a query_id for tracing — being able to look up exactly what happened for a specific request later, which is essential for debugging a "why did it hallucinate here" incident.
4 — Which databases, and why:
5 — Capacity estimation, worked step by step (numbers straight from the notes' math):
Query load:
50,000 DAU (daily active users) × 8 queries/day/user × 4.5 (peak multiplier)
────────────────────────────────────────────────────────────────────────── = 200 QPS
86,400 seconds/day
So: 50,000 × 8 = 400,000 total queries/day. 400,000 ÷ 86,400 seconds ≈ 4.6 QPS average. Multiply by a peak multiplier of 4.5 (because traffic isn't spread evenly across 24 hours — there are busy hours) to get roughly 200 QPS at peak. The team then designs for a bit of headroom: 250 QPS target capacity.
Storage — raw documents:
10,000,000 docs × 2KB each = 20GB raw data
This is small enough to be almost trivial to store redundantly — it's the derived data (embeddings, indexes) that actually dominates storage, as shown next.
Storage — vector embeddings:
10 × 10^6 documents × 8 passages/doc (avg) × 1536 dimensions × 4 bytes/float32
= 10×10^6 × 8 × 1536 × 4 bytes
≈ 491 GB (just the raw embedding vectors)
Then add HNSW graph overhead (the extra data structure the ANN index needs to do fast approximate search) — roughly 1.5x — bringing:
Total vector index storage ≈ 491GB × 1.5 ≈ 700GB
This is why the notes conclude: "Hence, we need a distributed vector database" — 700GB (spread across replicas for redundancy and read throughput) is well beyond what a single machine comfortably handles for a low-latency, always-on service. The notes also mention Product Quantization can reduce this by roughly 8x, but "at [the] cost of recall" — i.e., you trade some retrieval accuracy for a much smaller memory footprint, a classic space/accuracy tradeoff.
ElasticSearch (BM25 index) sizing: roughly 20GB × 1.5 ≈ 30GB
for the inverted index, run as a 3-node cluster with high availability (HA) plus 3 replicas, ≈ 90GB total, each node with 30GB RAM.
Postgres sizing: the 20GB of metadata plus JSONB overhead and replication comes out to roughly 60-80GB, served with 3 read replicas (since this is read-heavy traffic) with a target replication lag of under 100ms.
Redis: a 3-node cluster with 3 replicas, caching passages and query results — this is explicitly conditioned on the earlier semantic-caching insight: it works best "if no personalization & data is static."
6, 7, 8 — Read path, what's returned, and failure handling are covered together via the code walkthrough in Section 9.4 below (the notes' own respond()
pseudocode).
9 — Write path is covered in Section 9.5 below.
respond()
function), cleaned up and fully commented The notes' own hand-drawn pseudocode for the read path is the heart of the "zero hallucinations" guarantee. Here it is translated into clean, fully-commented Python:
from dataclasses import dataclass, field
@dataclass
class RespondResult:
answer: str
passages: list
refused: bool
refusal_reason: str | None = None
confidence: float | None = None
def respond(question: str) -> RespondResult:
"""
The full "zero hallucinations" read path:
embed -> hybrid search (dense + sparse) -> RRF fusion -> cross-encoder
rerank -> confidence threshold -> LLM generate -> claim extraction ->
NLI fact-check -> final answer OR refusal.
Every stage is a chance to refuse rather than guess -- that's the whole
point of the "must not assert unsupported claims" guarantee.
"""
query_vector = embed(question)
dense_results = qdrant_ann_search(query_vector, top_k=40)
sparse_results = elasticsearch_bm25(question, top_k=40)
candidates = rrf_fusion(dense_results, sparse_results)
ranked = cross_encoder_rerank(question, candidates, top_k=40)
if not any(passage.score > 0.4 for passage in ranked):
return refused_response("low_retrieval_confidence")
top_passages = ranked[:5]
prompt = build_prompt(question, top_passages)
raw_answer = llm_generate(prompt, temperature=0)
if raw_answer.startswith("INSUFFICIENT_CONTEXT"):
return refused_response("no_relevant_passages")
claims = extract_atomic_claims(raw_answer)
for claim in claims:
if not nli_entailed(claim, top_passages):
return refused_response("unverified_claim")
return RespondResult(answer=raw_answer, passages=top_passages, refused=False)
def refused_response(reason: str) -> RespondResult:
"""A first-class, explicit refusal -- NOT a vague hedge buried in text."""
return RespondResult(answer="", passages=[], refused=True, refusal_reason=reason)
The write path (S3 → chunk ingestor → hierarchical chunking → embeddings → storage) needs a chunker. The notes specify sentence-level boundaries (using spaCy) plus 32-token overlap between consecutive chunks (overlap ensures a sentence that would otherwise get awkwardly split across two chunks still has enough surrounding context in at least one of them).
import spacy
nlp = spacy.load("en_core_web_sm")
def chunk_document(text: str, max_tokens_per_chunk: int = 256, overlap_tokens: int = 32) -> list[dict]:
"""
Hierarchical, sentence-boundary-respecting chunking with token overlap.
Why sentence boundaries: splitting mid-sentence produces chunks that are
semantically broken and embed poorly. Why overlap: a fact stated right at
a chunk boundary shouldn't lose its surrounding context in EVERY chunk
that references it.
"""
doc = nlp(text)
sentences = [sent.text.strip() for sent in doc.sents]
chunks = []
current_chunk_sentences: list[str] = []
current_token_count = 0
char_cursor = 0
def rough_token_count(s: str) -> int:
return len(s.split())
for sentence in sentences:
sentence_tokens = rough_token_count(sentence)
if current_token_count + sentence_tokens > max_tokens_per_chunk and current_chunk_sentences:
chunk_text = " ".join(current_chunk_sentences)
chunks.append({
"text": chunk_text,
"token_count": current_token_count,
"char_start": char_cursor,
"char_end": char_cursor + len(chunk_text),
})
char_cursor += len(chunk_text)
overlap_sentences, overlap_count = [], 0
for sent in reversed(current_chunk_sentences):
overlap_sentences.insert(0, sent)
overlap_count += rough_token_count(sent)
if overlap_count >= overlap_tokens:
break
current_chunk_sentences = overlap_sentences
current_token_count = overlap_count
current_chunk_sentences.append(sentence)
current_token_count += sentence_tokens
if current_chunk_sentences:
chunk_text = " ".join(current_chunk_sentences)
chunks.append({
"text": chunk_text,
"token_count": current_token_count,
"char_start": char_cursor,
"char_end": char_cursor + len(chunk_text),
})
return chunks
sample_doc = (
"Throughput degradation under sustained load is a common issue. "
"It is typically caused by connection pool exhaustion. "
"Another frequent cause is unindexed database queries. "
"Monitoring dashboards should track p50, p95, and p99 latency. "
"Alerts should fire when p99 crosses a defined SLO threshold."
)
for i, chunk in enumerate(chunk_document(sample_doc, max_tokens_per_chunk=15, overlap_tokens=5)):
print(f"Chunk {i}: {chunk['text']}")
Each resulting chunk (a "passage" in the notes' schema) would then get an embedding computed (using one of the models the notes list — microsoft/hamier
, Qwen3 (8B)
, BGE-M3
, or an OpenAI/OSS embedding API), stored in Qdrant (the vector), Postgres (the metadata: doc_id, chunk_index, char_start, char_end, token_count), and ElasticSearch (the BM25 inverted index).
Flow: S3 (raw doc storage) → Chunk Ingestor → Hierarchical Chunking → Embeddings → Storage (Postgres, Qdrant, ElasticSearch, Redis).
Step by step: a document lands in S3. A chunk ingestor process picks it up (typically via an event notification or polling), performs three sub-steps the notes list explicitly (list files, download, read), then runs hierarchical chunking (Section 9.5's sentence-boundary + overlap logic) to split it into passages. Each passage is embedded, and the results are fanned out to storage: Postgres gets the passage/doc metadata, Qdrant gets the embedding vectors, and ElasticSearch gets the text for its BM25 inverted index. This asynchronous pipeline is exactly why the system can tolerate the ~15-minute eventual consistency SLO mentioned earlier — the write path doesn't need to be instantaneous, it needs to be reliable and complete within a bounded, acceptable delay.
The notes also flag a timebox mechanism on the read side with two named fallback behaviors — cascade and fallback — meaning if some part of the read path takes too long (e.g., the cross-encoder rerank step or the LLM generation call, described as "the long pole" of end-to-end latency), the system has a time budget after which it cascades to a simpler/cheaper strategy or falls back to a safe default (like a refusal) rather than hanging indefinitely. This connects directly back to the query latency numbers given in the notes: p50 = 1.5 sec, p99 = 5 sec, with the LLM inference call explicitly called out as the long pole (the slowest, bottleneck stage) — largely because the very first token the user sees (their Time To First Token, covered next in the Appendix) is gated on that call finishing its reasoning.
Think about the difference between getting a text message that appears all at once after a long , versus watching someone type it live, word by word, right in front of you. Even though the total message might take the same amount of time to fully arrive, watching it appear progressively feels far less frustrating — you get immediate feedback that something is happening, instead of staring at a blank screen wondering if anything is working at all.
That's the entire motivation for streaming: even if the total response time is identical, showing partial output as it's generated dramatically improves perceived responsiveness.
The most important metric here, per the notes: TTFT — Time To First Token (a plain-English one-liner: "how long the user waits before they see ANY output at all, even if the full response isn't done yet"). If a tool call takes real time to execute (a slow database query, a slow external API), and the user has to wait through the entire tool call before seeing anything, that's flagged directly in the notes as bad UX — "like everything else," slow, silent waiting erodes trust in the product.
The tricky part specific to streaming tool calls: "what if tool input arrives as partial JSON fragments?" When a model streams its response token by token, and part of that response is a tool_use
block (e.g., building up {"city": "Tok
... yo"}
character by character), you cannot execute the tool with a half-formed, syntactically-invalid JSON blob. The notes' answer: "wait until the entire input is available" — i.e., keep buffering (accumulating) the partial JSON text as it streams in, and only attempt to parse/execute the tool call once the model signals that this particular content block is fully complete.
The notes include a cautionary, very concrete illustration of why this matters: imagine the model is streaming a dangerous tool call like "delete that record..." If your system tried to act on a half-baked, incomplete tool input (say, a delete call where the id argument hasn't fully arrived yet), it could either error out unpredictably or — worse — "no input could be fired" i.e., act on a malformed/empty argument. Half-baked input / tool calls with no input could be fired is exactly the failure mode the buffering strategy exists to prevent.
import anthropic
import json
client = anthropic.Anthropic()
tools = [{
"name": "get_weather",
"description": "Get current weather for a location",
"input_schema": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
}]
def execute_tool(tool_name: str, tool_input: dict) -> dict:
"""Stand-in for a real tool implementation (see Section 1 for a fuller mock)."""
if tool_name == "get_weather":
return {"location": tool_input["location"], "temp": 24, "condition": "sunny"}
return {"error": "unknown tool"}
with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
tools=tools,
messages=[{"role": "user", "content": "What is the weather in Tokyo?"}],
) as stream:
current_tool_input = ""
current_tool_name = None
for event in stream:
if event.type == "content_block_start":
if event.content_block.type == "tool_use":
current_tool_name = event.content_block.name
current_tool_input = "" # reset the buffer for this new block
elif event.type == "content_block_delta":
if event.delta.type == "input_json_delta":
current_tool_input += event.delta.partial_json # <- Accumulate
elif event.delta.type == "text_delta":
print(event.delta.text, end="", flush=True)
elif event.type == "content_block_stop":
if current_tool_name:
parsed_input = json.loads(current_tool_input) # <- input complete
result = execute_tool(current_tool_name, parsed_input) # <- execute
print(f"\n[Executed {current_tool_name} -> {result}]")
current_tool_name = None # reset for the next potential tool block
The three event types to remember: ** content_block_start** (a new block began — note its type/name),
content_block_delta
content_block_stop
Context Precision, simple example: think of a search-results page with 5 blue links. Some of them are genuinely useful and directly relevant to what you searched for (imagine green dots next to them); others are irrelevant filler or noise (red dots). A good search engine doesn't just include some relevant results somewhere on the page — it puts them near the top. A search engine that buries the one truly relevant result at position #5, behind four irrelevant ones, is worse than one that puts it at position #1, even if both technically "found" it somewhere on the page.
Faithfulness, simple example: imagine grading a student's essay by checking, sentence by sentence, whether each individual claim they made is actually backed up by the assigned textbook. If the student writes five sentences and four of them are directly supported by the textbook but the fifth one is something they simply made up (that isn't in the textbook at all), you wouldn't give them full credit — you'd score the essay based on the fraction of sentences that are actually grounded in the source material. That's exactly what "faithfulness" measures for an LLM's generated answer.
The motivation, per the notes: naive RAG evaluation is just qualitative — "does it seem right?" — and manual feedback loops are slow. That doesn't scale, and it's subjective. RAGAS (a plain-English one-liner: "a framework and library for evaluating RAG pipelines using automated, LLM-assisted metrics instead of manual human judgment") fixes two specific pain points named in the notes: no need for human-annotated ground truth, and it lets you verify that changes/optimizations to your pipeline don't quietly downgrade quality (a regression-testing mindset applied to RAG quality, not just code correctness).
Context Precision — measures the retriever's (or reranker's) ability to rank relevant chunks higher, and it directly penalizes systems that bury relevant content below noise — this is precisely the "lost in the middle" problem flagged back in Section 6. The method: use an LLM to label each retrieved chunk as RELEVANT or NOT relevant, and then, for every position k
in the ranked list, compute precision@k — the fraction of the top k results that are relevant.
The notes' worked example, using a 5-result list marked (green = relevant, red = not relevant): 🟢 🔴 🟢 🔴 🔴
p@2 = 0
. Reading the notes' own math carefully: Final Context Precision formula (from the notes' own arithmetic):
Context Precision = (p@1 + p@3) / (# relevant chunks)
= (1 + 0.67) / 2
= 1.67 / 2
= 0.835
In words: you sum up the precision@k values at each rank where a relevant chunk appears, then divide by the total number of relevant chunks in the list (here, 2 relevant chunks out of 5 total). This rewards systems that put their relevant chunks early — a relevant chunk found at rank 1 contributes a full 1.0, while one found deeper in the list contributes a smaller fraction, exactly reflecting the "don't bury the good stuff" intuition from the simple example.
Faithfulness Score — measures factual consistency of the generated answer, and is explicitly called a "measure of hallucination" in the notes. Method: break the answer into atomic claims (using an LLM — the same "claim extraction" step from the read-path pseudocode in Section 9.4), then for each claim, score it 1 if the retrieved context supports the claim, or 0 if the context does not support the claim.
Faithfulness formula:
faithfulness = (# claims supported) / (# claims)
This is the exact same underlying idea as the nli_entailed()
check baked into the "zero hallucinations" respond()
function from Section 9 — faithfulness is essentially the evaluation-time metric version of the same runtime guardrail.
def context_precision(relevance_flags: list[bool]) -> float:
"""
relevance_flags: e.g. [True, False, True, False, False]
-- True = relevant chunk, False = not relevant,
in the ORDER they were retrieved/ranked (best first).
Returns the RAGAS-style Context Precision score.
"""
total_relevant = sum(relevance_flags)
if total_relevant == 0:
return 0.0 # no relevant chunks at all -- precision is undefined/zero
running_relevant_count = 0
precision_sum = 0.0
for k, is_relevant in enumerate(relevance_flags, start=1):
if is_relevant:
running_relevant_count += 1
precision_at_k = running_relevant_count / k
precision_sum += precision_at_k
return precision_sum / total_relevant
flags = [True, False, True, False, False]
score = context_precision(flags)
print(f"Context Precision: {score:.3f}") # -> should print 0.835
def faithfulness_score(claim_support_labels: list[bool]) -> float:
"""
claim_support_labels: e.g. [True, True, True, False]
-- True = context supports this atomic claim,
False = context does NOT support it (hallucinated).
"""
if not claim_support_labels:
return 1.0 # no claims made -- vacuously faithful (nothing to check)
supported = sum(claim_support_labels)
total = len(claim_support_labels)
return supported / total
claims_supported = [True, True, True, False]
faith_score = faithfulness_score(claims_supported)
print(f"Faithfulness: {faith_score:.3f}") # -> 0.75
python
from ragas import evaluate
from ragas.metrics import context_precision, faithfulness
from datasets import Dataset
eval_dataset = Dataset.from_dict({
"question": ["Why is my API returning a 500 error under high load?"],
"answer": ["500 errors under high load are typically caused by connection "
"pool exhaustion or unhandled exceptions from resource contention."],
"contexts": [[
"Throughput degradation under sustained load can trigger unhandled "
"exceptions, which surface to callers as HTTP 500 errors.",
"Connection pool exhaustion is a common cause of request failures "
"under high concurrency.",
]],
"ground_truth": ["500 errors under load are usually caused by connection "
"pool exhaustion or unhandled exceptions."],
})
results = evaluate(eval_dataset, metrics=[context_precision, faithfulness])
print(results)