Contextual Retrieval in 80 Lines: Anthropic's Chunk-Prefix Trick, Measured on Your Docs

A developer published an ~80-line Python script that reproduces Anthropic's Contextual Retrieval technique, prepending an LLM-generated context note to each chunk before embedding and BM25 indexing, and measuring recall@5 before and after on user-labeled questions. The script uses the Anthropic and OpenAI SDKs, rank-bm25 and NumPy, and relies on prompt caching so each document is paid for once. It cites Anthropic's September 2024 write-up, which reported a 49% reduction in top-20 retrieval failure (5.7% to 2.9%) with contextual embeddings plus contextual BM25, and 67% with a reranker added.

Here is a chunk from a quarterly filing: "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? Grew from what? Inside the filing, the sentence made sense. Cut out, embedded and indexed on its own, it carries none of that, and a query about ACME's Q2 2023 revenue has almost nothing to match against. That example comes from Anthropic's Contextual Retrieval write-up https://www.anthropic.com/news/contextual-retrieval , published September 19, 2024. The fix they proposed is small. Before you embed a chunk or index it for BM25, ask a model to write a short context for it using the whole document, and glue that context to the front of the chunk. Their chunk becomes: "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter." On their test sets, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49% https://www.anthropic.com/news/contextual-retrieval , from 5.7% to 2.9%. Adding a reranker on top took the reduction to 67%, down to 1.9%. Those are their numbers on their data. This post gets you yours: about 80 lines of Python that index your docs both ways, contextualize every chunk with prompt caching so each document is paid for once, and print recall@5 before and after on 30 questions you label yourself. The metric in the write-up is 1 minus recall@20: the share of relevant chunks that did not show up in the top 20 results. The test data covered codebases, fiction, ArXiv papers and science papers. The headline figures used Gemini Text 004 embeddings and Claude 3 Haiku to write the contexts, which usually came out at 50 to 100 tokens same source https://www.anthropic.com/news/contextual-retrieval . The companion cookbook notebook https://github.com/anthropics/claude-cookbooks/blob/main/capabilities/contextual-embeddings/guide.ipynb reports at k=5, closer to what this post measures. Its printed run shows Pass@5 going from 80.92% to 88.12% with contextual embeddings alone. The same notebook's summary table and prose don't fully agree with its printed output, which is one more reason to treat any published number as a hint about direction. None of this tells you whether your docs have the problem. Chunks that lean on their surroundings "the endpoint above", "this setting", a table whose header lives two chunks back have the most to gain. A folder of self-contained FAQ entries may barely move. You find out by measuring. Anthropic's 2024 write-up also suggests https://www.anthropic.com/news/contextual-retrieval that if your whole knowledge base is under 200,000 tokens, you put all of it in the prompt and skip retrieval. The whole trick: the chunk keeps a short note about where it came from. Install the four dependencies and set ANTHROPIC API KEY and OPENAI API KEY : pip install anthropic openai rank-bm25 numpy The script uses the Anthropic Python SDK https://github.com/anthropics/anthropic-sdk-python for the contexts, the OpenAI Python SDK https://github.com/openai/openai-python for embeddings, rank-bm25 https://pypi.org/project/rank-bm25/ for keyword search and NumPy https://numpy.org for the vector math. Save it as ctx eval.py : python import json, re, sys import numpy as np from anthropic import Anthropic from openai import OpenAI from rank bm25 import BM25Okapi claude, oai = Anthropic , OpenAI usage = {"written": 0, "read": 0, "uncached": 0} PROMPT = "Here is the chunk we want to situate within the whole " "document\n<chunk \n{chunk}\n</chunk \n" "Please give a short succinct context to situate this " "chunk within the overall document for the purposes of " "improving search retrieval of the chunk. Answer only " "with the succinct context and nothing else." def chunk text, size=500 : words = text.split return " ".join words i:i + size for i in range 0, len words , size def situate doc, piece : resp = claude.messages.create model="claude-haiku-5-5", max tokens=300, thinking={"type": "disabled"}, messages= {"role": "user", "content": {"type": "text", "text": f"<document \n{doc}\n</document ", "cache control": {"type": "ephemeral"}}, {"type": "text", "text": PROMPT.format chunk=piece }, } , u = resp.usage usage "written" += u.cache creation input tokens or 0 usage "read" += u.cache read input tokens or 0 usage "uncached" += u.input tokens return resp.content 0 .text.strip def embed texts : vecs = for i in range 0, len texts , 100 : r = oai.embeddings.create model="text-embedding-3-small", input=texts i:i + 100 vecs += d.embedding for d in r.data return np.array vecs def toks s : return re.findall r"\w+", s.lower class Index: def init self, texts : self.vecs = embed texts self.bm25 = BM25Okapi toks t for t in texts def search self, q, k=5, pool=20 : dense = np.argsort - self.vecs @ embed q 0 :pool bm = self.bm25.get scores toks q sparse = i for i in np.argsort -bm :pool if bm i 0 fused = {} for ranking in dense, sparse : for rank, i in enumerate ranking, 1 : fused i = fused.get i, 0 + 1 / 60 + rank return sorted fused, key=fused.get, reverse=True :k def build docs, contextual : ids, texts = , for name, doc in docs.items : for n, piece in enumerate chunk doc : if contextual: piece = situate doc, piece + "\n\n" + piece ids.append f"{name} {n}" texts.append piece return ids, Index texts def recall ids, index, qa, k=5 : def hit item : gold = set item "gold" top = {ids i for i in index.search item "q" , k } return len top & gold / len gold return sum hit x for x in qa / len qa if name == " main ": qa = json.load open sys.argv 1 docs = {p: open p .read for p in sys.argv 2: } for contextual in False, True : ids, index = build docs, contextual r = recall ids, index, qa print f"contextual={contextual} recall@5={r:.1%}" print "prompt tokens:", usage Run it as python ctx eval.py qa.json docs/ .md . Here is what each part is doing. chunk splits on fixed 500-word windows. Swap in your production chunker; both runs just have to use the same one, because the gold labels point at chunk positions. situate sends Anthropic's contextualizer prompt, copied from the cookbook https://github.com/anthropics/claude-cookbooks/blob/main/capabilities/contextual-embeddings/guide.ipynb . The document goes in the first content block with cache control , and the chunk goes in a second block that is not cached. That ordering matters: the cache covers the prefix up to the marked block, so the document has to come first. The model is Claude Haiku 5.5 https://platform.claude.com/docs/en/about-claude/models/overview , whose thinking is on by default. The thinking docs https://platform.claude.com/docs/en/build-with-claude/thinking say Haiku 5.5 accepts {"type": "disabled"} at high effort or below, and the models overview lists its default effort as medium. Writing one sentence of context does not need reasoning tokens billed as output. Index.search runs both retrievers. The dense side is a plain dot product, which works as cosine similarity because OpenAI embeddings are normalized to length 1 https://developers.openai.com/api/docs/guides/embeddings . The BM25 side lowercases and splits on word characters itself, since rank-bm25 does no preprocessing of its own. Chunks with a zero BM25 score are dropped so they don't sneak into the fusion. Anthropic's write-up says to merge the two result sets with "rank fusion techniques" without naming one, so this uses reciprocal rank fusion with the k=60 constant Elasticsearch defaults to https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion . build is where the experiment lives. The contextual flag changes the text that gets embedded and indexed, and nothing else. Chunk IDs stay the same in both runs, so one label file scores both. recall averages, over all questions, the fraction of gold chunks that land in the top 5. For a question with one gold chunk, that is hit or miss. Without caching, every chunk call re-sends the entire document. Ten chunks from one document means paying for that document ten times as input. With cache control on the document block, the first call writes it to the cache and the next nine read it back. The prompt caching docs https://platform.claude.com/docs/en/build-with-claude/prompt-caching price a 5-minute cache write at 1.25 times base input and a cache read at 0.1 times. On the pricing page https://platform.claude.com/docs/en/about-claude/pricing as of October 2026, Claude Haiku 5.5 for prompts up to 100,000 tokens is $0.10 per million input tokens, $0.125 for 5-minute cache writes, $0.01 for cache hits and $0.50 for output. Prompts over 100,000 tokens move to a rate five times higher, so very long documents cost more per chunk. Some arithmetic from that table, using the shape Anthropic used for its own estimate: an 8,000-token document, ten 800-token chunks, about 50 tokens of instructions and 100 tokens of generated context per chunk. With caching, that is roughly $0.001 for the one cache write, $0.0007 for nine cache reads, $0.0009 for the uncached chunk text and $0.0005 for output. About $0.003 per document, or around $0.38 per million document tokens. Without caching, the same job comes to roughly $1.17 per million. That is my arithmetic from list prices, not a measured bill. Anthropic's 2024 figure for the same shape was $1.02 per million document tokens https://www.anthropic.com/news/contextual-retrieval , on Claude 3 Haiku pricing. One write, many cheap reads: the document is cached and each chunk call reuses it. Three details from the caching docs decide whether you actually get those reads: The prompt tokens line at the end tells you how it went on your run. For comparison, the cookbook's own 737-chunk ingestion reported that 61.83% of all input tokens were read from cache, lower than the 70 to 80% its prose suggests. Read your counters before you trust anyone's estimate, including the one above. The number this script prints is only as honest as qa.json . To label anything, you first need to see the chunk IDs. A helper that reuses the same chunker: python import sys from ctx eval import chunk for path in sys.argv 1: : for n, piece in enumerate chunk open path .read : print f"{path} {n}\t{piece :90 }" Importing ctx eval creates both API clients, so keep the environment variables set. Then write the label file: {"q": "How long do refunds take on annual plans?", "gold": "docs/billing.md 3" }, {"q": "Which header carries the idempotency key?", "gold": "docs/api.md 7", "docs/api.md 8" } A few rules keep the set from flattering either run: With 30 questions, each one is worth about 3.3 points of recall@5 mailto:recall@5 . A gap of one or two questions between runs is within noise. So look at the flips as well as the totals. Inside the main block, after each build , this prints a hit or miss per question: for item in qa: top = {ids i for i in index.search item "q" } print bool top & set item "gold" , item "q" :60 Questions that went from miss to hit tell you where the prefix helped. Questions that went from hit to miss deserve a closer read: when every chunk of one document gets a similar opening sentence, those chunks may start to look alike to the retriever. If recall@5 barely moves, that is a real result. Your chunks may already be self-contained, or your baseline may already be high, and you just saved yourself an indexing step on every document update. If it moves, Anthropic's write-up https://www.anthropic.com/news/contextual-retrieval lists the next levers. They reranked by pulling the top 150 and narrowing to 20 with a Cohere reranker https://docs.cohere.com/docs/rerank , which is where the 67% figure came from. They tried passing 5, 10 and 20 chunks and found 20 worked best for them, while telling readers to test it on their own use case. They also suggest a contextualizer prompt tailored to your domain, such as one that includes a glossary of terms defined elsewhere in the knowledge base. Change one lever at a time and rerun the same frozen 30 questions. Keep the indexing cost in mind once this goes past an experiment. Contexts are generated at index time from the whole document, so when a document changes, any of its chunks' contexts can go stale. Re-contextualize the whole document, not only the edited chunk. The script is the measuring stick; the decisions around it how you chunk, when hybrid search beats dense alone, when a reranker earns its latency are what the RAG Pocket Guide https://www.amazon.com/dp/B0GX2YDC5Z covers, under its subtitle of retrieval, chunking and reranking patterns for production. Run the 30 questions first, then pick the chapter that matches the misses.