Here is a chunk from a quarterly filing: "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? Grew from what? Inside the filing, the sentence made sense. Cut out, embedded and indexed on its own, it carries none of that, and a query about ACME's Q2 2023 revenue has almost nothing to match against.
That example comes from Anthropic's Contextual Retrieval write-up, published September 19, 2024. The fix they proposed is small. Before you embed a chunk or index it for BM25, ask a model to write a short context for it using the whole document, and glue that context to the front of the chunk. Their chunk becomes: "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."
On their test sets, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, from 5.7% to 2.9%. Adding a reranker on top took the reduction to 67%, down to 1.9%. Those are their numbers on their data. This post gets you yours: about 80 lines of Python that index your docs both ways, contextualize every chunk with prompt caching so each document is paid for once, and print recall@5 before and after on 30 questions you label yourself.
The metric in the write-up is 1 minus recall@20: the share of relevant chunks that did not show up in the top 20 results. The test data covered codebases, fiction, ArXiv papers and science papers. The headline figures used Gemini Text 004 embeddings and Claude 3 Haiku to write the contexts, which usually came out at 50 to 100 tokens (same source).
The companion cookbook notebook reports at k=5, closer to what this post measures. Its printed run shows Pass@5 going from 80.92% to 88.12% with contextual embeddings alone. The same notebook's summary table and prose don't fully agree with its printed output, which is one more reason to treat any published number as a hint about direction.
None of this tells you whether your docs have the problem. Chunks that lean on their surroundings ("the endpoint above", "this setting", a table whose header lives two chunks back) have the most to gain. A folder of self-contained FAQ entries may barely move. You find out by measuring.
Anthropic's 2024 write-up also suggests that if your whole knowledge base is under 200,000 tokens, you put all of it in the prompt and skip retrieval.
The whole trick: the chunk keeps a short note about where it came from.
Install the four dependencies and set ANTHROPIC_API_KEY and OPENAI_API_KEY:
pip install anthropic openai rank-bm25 numpy
The script uses the Anthropic Python SDK for the contexts, the OpenAI Python SDK for embeddings, rank-bm25 for keyword search and NumPy for the vector math. Save it as ctx_eval.py:
import json, re, sys
import numpy as np
from anthropic import Anthropic
from openai import OpenAI
from rank_bm25 import BM25Okapi
claude, oai = Anthropic(), OpenAI()
usage = {"written": 0, "read": 0, "uncached": 0}
PROMPT = (
"Here is the chunk we want to situate within the whole "
"document\n<chunk>\n{chunk}\n</chunk>\n"
"Please give a short succinct context to situate this "
"chunk within the overall document for the purposes of "
"improving search retrieval of the chunk. Answer only "
"with the succinct context and nothing else."
)
def chunk(text, size=500):
words = text.split()
return [" ".join(words[i:i + size])
for i in range(0, len(words), size)]
def situate(doc, piece):
resp = claude.messages.create(
model="claude-haiku-5-5",
max_tokens=300,
thinking={"type": "disabled"},
messages=[{"role": "user", "content": [
{"type": "text",
"text": f"<document>\n{doc}\n</document>",
"cache_control": {"type": "ephemeral"}},
{"type": "text", "text": PROMPT.format(chunk=piece)},
]}],
)
u = resp.usage
usage["written"] += u.cache_creation_input_tokens or 0
usage["read"] += u.cache_read_input_tokens or 0
usage["uncached"] += u.input_tokens
return resp.content[0].text.strip()
def embed(texts):
vecs = []
for i in range(0, len(texts), 100):
r = oai.embeddings.create(
model="text-embedding-3-small",
input=texts[i:i + 100])
vecs += [d.embedding for d in r.data]
return np.array(vecs)
def toks(s):
return re.findall(r"\w+", s.lower())
class Index:
def __init__(self, texts):
self.vecs = embed(texts)
self.bm25 = BM25Okapi([toks(t) for t in texts])
def search(self, q, k=5, pool=20):
dense = np.argsort(-(self.vecs @ embed([q])[0]))[:pool]
bm = self.bm25.get_scores(toks(q))
sparse = [i for i in np.argsort(-bm)[:pool] if bm[i] > 0]
fused = {}
for ranking in (dense, sparse):
for rank, i in enumerate(ranking, 1):
fused[i] = fused.get(i, 0) + 1 / (60 + rank)
return sorted(fused, key=fused.get, reverse=True)[:k]
def build(docs, contextual):
ids, texts = [], []
for name, doc in docs.items():
for n, piece in enumerate(chunk(doc)):
if contextual:
piece = situate(doc, piece) + "\n\n" + piece
ids.append(f"{name}#{n}")
texts.append(piece)
return ids, Index(texts)
def recall(ids, index, qa, k=5):
def hit(item):
gold = set(item["gold"])
top = {ids[i] for i in index.search(item["q"], k)}
return len(top & gold) / len(gold)
return sum(hit(x) for x in qa) / len(qa)
if __name__ == "__main__":
qa = json.load(open(sys.argv[1]))
docs = {p: open(p).read() for p in sys.argv[2:]}
for contextual in (False, True):
ids, index = build(docs, contextual)
r = recall(ids, index, qa)
print(f"contextual={contextual} recall@5={r:.1%}")
print("prompt tokens:", usage)
Run it as python ctx_eval.py qa.json docs/*.md. Here is what each part is doing.
chunk splits on fixed 500-word windows. Swap in your production chunker; both runs just have to use the same one, because the gold labels point at chunk positions.
situate sends Anthropic's contextualizer prompt, copied from the cookbook. The document goes in the first content block with cache_control, and the chunk goes in a second block that is not cached. That ordering matters: the cache covers the prefix up to the marked block, so the document has to come first. The model is Claude Haiku 5.5, whose thinking is on by default. The thinking docs say Haiku 5.5 accepts {"type": "disabled"} at high effort or below, and the models overview lists its default effort as medium. Writing one sentence of context does not need reasoning tokens billed as output.
Index.search runs both retrievers. The dense side is a plain dot product, which works as cosine similarity because OpenAI embeddings are normalized to length 1. The BM25 side lowercases and splits on word characters itself, since rank-bm25 does no preprocessing of its own. Chunks with a zero BM25 score are dropped so they don't sneak into the fusion. Anthropic's write-up says to merge the two result sets with "rank fusion techniques" without naming one, so this uses reciprocal rank fusion with the k=60 constant Elasticsearch defaults to.
build is where the experiment lives. The contextual flag changes the text that gets embedded and indexed, and nothing else. Chunk IDs stay the same in both runs, so one label file scores both.
recall averages, over all questions, the fraction of gold chunks that land in the top 5. For a question with one gold chunk, that is hit or miss.
Without caching, every chunk call re-sends the entire document. Ten chunks from one document means paying for that document ten times as input. With cache_control on the document block, the first call writes it to the cache and the next nine read it back.
The prompt caching docs price a 5-minute cache write at 1.25 times base input and a cache read at 0.1 times. On the pricing page as of October 2026, Claude Haiku 5.5 for prompts up to 100,000 tokens is $0.10 per million input tokens, $0.125 for 5-minute cache writes, $0.01 for cache hits and $0.50 for output. Prompts over 100,000 tokens move to a rate five times higher, so very long documents cost more per chunk.
Some arithmetic from that table, using the shape Anthropic used for its own estimate: an 8,000-token document, ten 800-token chunks, about 50 tokens of instructions and 100 tokens of generated context per chunk. With caching, that is roughly $0.001 for the one cache write, $0.0007 for nine cache reads, $0.0009 for the uncached chunk text and $0.0005 for output. About $0.003 per document, or around $0.38 per million document tokens. Without caching, the same job comes to roughly $1.17 per million. That is my arithmetic from list prices, not a measured bill. Anthropic's 2024 figure for the same shape was $1.02 per million document tokens, on Claude 3 Haiku pricing.
One write, many cheap reads: the document is cached and each chunk call reuses it.
Three details from the caching docs decide whether you actually get those reads:
The prompt tokens line at the end tells you how it went on your run. For comparison, the cookbook's own 737-chunk ingestion reported that 61.83% of all input tokens were read from cache, lower than the 70 to 80% its prose suggests. Read your counters before you trust anyone's estimate, including the one above.
The number this script prints is only as honest as qa.json. To label anything, you first need to see the chunk IDs. A helper that reuses the same chunker:
import sys
from ctx_eval import chunk
for path in sys.argv[1:]:
for n, piece in enumerate(chunk(open(path).read())):
print(f"{path}#{n}\t{piece[:90]}")
Importing ctx_eval creates both API clients, so keep the environment variables set. Then write the label file:
[
{"q": "How long do refunds take on annual plans?",
"gold": ["docs/billing.md#3"]},
{"q": "Which header carries the idempotency key?",
"gold": ["docs/api.md#7", "docs/api.md#8"]}
]
A few rules keep the set from flattering either run:
With 30 questions, each one is worth about 3.3 points of recall@5. A gap of one or two questions between runs is within noise. So look at the flips as well as the totals. Inside the __main__ block, after each build, this prints a hit or miss per question:
for item in qa:
top = {ids[i] for i in index.search(item["q"])}
print(bool(top & set(item["gold"])), item["q"][:60])
Questions that went from miss to hit tell you where the prefix helped. Questions that went from hit to miss deserve a closer read: when every chunk of one document gets a similar opening sentence, those chunks may start to look alike to the retriever.
If recall@5 barely moves, that is a real result. Your chunks may already be self-contained, or your baseline may already be high, and you just saved yourself an indexing step on every document update.
If it moves, Anthropic's write-up lists the next levers. They reranked by pulling the top 150 and narrowing to 20 with a Cohere reranker, which is where the 67% figure came from. They tried passing 5, 10 and 20 chunks and found 20 worked best for them, while telling readers to test it on their own use case. They also suggest a contextualizer prompt tailored to your domain, such as one that includes a glossary of terms defined elsewhere in the knowledge base. Change one lever at a time and rerun the same frozen 30 questions.
Keep the indexing cost in mind once this goes past an experiment. Contexts are generated at index time from the whole document, so when a document changes, any of its chunks' contexts can go stale. Re-contextualize the whole document, not only the edited chunk.
The script is the measuring stick; the decisions around it (how you chunk, when hybrid search beats dense alone, when a reranker earns its latency) are what the RAG Pocket Guide covers, under its subtitle of retrieval, chunking and reranking patterns for production. Run the 30 questions first, then pick the chapter that matches the misses.