{"slug": "contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your", "title": "Contextual Retrieval in 80 Lines: Anthropic's Chunk-Prefix Trick, Measured on Your Docs", "summary": "A developer published an ~80-line Python script that reproduces Anthropic's Contextual Retrieval technique, prepending an LLM-generated context note to each chunk before embedding and BM25 indexing, and measuring recall@5 before and after on user-labeled questions. The script uses the Anthropic and OpenAI SDKs, rank-bm25 and NumPy, and relies on prompt caching so each document is paid for once. It cites Anthropic's September 2024 write-up, which reported a 49% reduction in top-20 retrieval failure (5.7% to 2.9%) with contextual embeddings plus contextual BM25, and 67% with a reranker added.", "body_md": "Here is a chunk from a quarterly filing: \"The company's revenue grew by 3% over the previous quarter.\" Which company? Which quarter? Grew from what? Inside the filing, the sentence made sense. Cut out, embedded and indexed on its own, it carries none of that, and a query about ACME's Q2 2023 revenue has almost nothing to match against.\n\nThat example comes from Anthropic's [Contextual Retrieval write-up](https://www.anthropic.com/news/contextual-retrieval), published September 19, 2024. The fix they proposed is small. Before you embed a chunk or index it for BM25, ask a model to write a short context for it using the whole document, and glue that context to the front of the chunk. Their chunk becomes: \"This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter.\"\n\nOn their test sets, [contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%](https://www.anthropic.com/news/contextual-retrieval), from 5.7% to 2.9%. Adding a reranker on top took the reduction to 67%, down to 1.9%. Those are their numbers on their data. This post gets you yours: about 80 lines of Python that index your docs both ways, contextualize every chunk with prompt caching so each document is paid for once, and print recall@5 before and after on 30 questions you label yourself.\n\nThe metric in the write-up is 1 minus recall@20: the share of relevant chunks that did not show up in the top 20 results. The test data covered codebases, fiction, ArXiv papers and science papers. The headline figures used Gemini Text 004 embeddings and Claude 3 Haiku to write the contexts, which usually came out at 50 to 100 tokens ([same source](https://www.anthropic.com/news/contextual-retrieval)).\n\nThe companion [cookbook notebook](https://github.com/anthropics/claude-cookbooks/blob/main/capabilities/contextual-embeddings/guide.ipynb) reports at k=5, closer to what this post measures. Its printed run shows Pass@5 going from 80.92% to 88.12% with contextual embeddings alone. The same notebook's summary table and prose don't fully agree with its printed output, which is one more reason to treat any published number as a hint about direction.\n\nNone of this tells you whether your docs have the problem. Chunks that lean on their surroundings (\"the endpoint above\", \"this setting\", a table whose header lives two chunks back) have the most to gain. A folder of self-contained FAQ entries may barely move. You find out by measuring.\n\nAnthropic's 2024 write-up also [suggests](https://www.anthropic.com/news/contextual-retrieval) that if your whole knowledge base is under 200,000 tokens, you put all of it in the prompt and skip retrieval.\n\n*The whole trick: the chunk keeps a short note about where it came from.*\n\nInstall the four dependencies and set `ANTHROPIC_API_KEY` and `OPENAI_API_KEY`:\n\n```\npip install anthropic openai rank-bm25 numpy\n```\n\nThe script uses the [Anthropic Python SDK](https://github.com/anthropics/anthropic-sdk-python) for the contexts, the [OpenAI Python SDK](https://github.com/openai/openai-python) for embeddings, [rank-bm25](https://pypi.org/project/rank-bm25/) for keyword search and [NumPy](https://numpy.org) for the vector math. Save it as `ctx_eval.py`:\n\n``` python\nimport json, re, sys\nimport numpy as np\nfrom anthropic import Anthropic\nfrom openai import OpenAI\nfrom rank_bm25 import BM25Okapi\n\nclaude, oai = Anthropic(), OpenAI()\nusage = {\"written\": 0, \"read\": 0, \"uncached\": 0}\n\nPROMPT = (\n    \"Here is the chunk we want to situate within the whole \"\n    \"document\\n<chunk>\\n{chunk}\\n</chunk>\\n\"\n    \"Please give a short succinct context to situate this \"\n    \"chunk within the overall document for the purposes of \"\n    \"improving search retrieval of the chunk. Answer only \"\n    \"with the succinct context and nothing else.\"\n)\n\ndef chunk(text, size=500):\n    words = text.split()\n    return [\" \".join(words[i:i + size])\n            for i in range(0, len(words), size)]\n\ndef situate(doc, piece):\n    resp = claude.messages.create(\n        model=\"claude-haiku-5-5\",\n        max_tokens=300,\n        thinking={\"type\": \"disabled\"},\n        messages=[{\"role\": \"user\", \"content\": [\n            {\"type\": \"text\",\n             \"text\": f\"<document>\\n{doc}\\n</document>\",\n             \"cache_control\": {\"type\": \"ephemeral\"}},\n            {\"type\": \"text\", \"text\": PROMPT.format(chunk=piece)},\n        ]}],\n    )\n    u = resp.usage\n    usage[\"written\"] += u.cache_creation_input_tokens or 0\n    usage[\"read\"] += u.cache_read_input_tokens or 0\n    usage[\"uncached\"] += u.input_tokens\n    return resp.content[0].text.strip()\n\ndef embed(texts):\n    vecs = []\n    for i in range(0, len(texts), 100):\n        r = oai.embeddings.create(\n            model=\"text-embedding-3-small\",\n            input=texts[i:i + 100])\n        vecs += [d.embedding for d in r.data]\n    return np.array(vecs)\n\ndef toks(s):\n    return re.findall(r\"\\w+\", s.lower())\n\nclass Index:\n    def __init__(self, texts):\n        self.vecs = embed(texts)\n        self.bm25 = BM25Okapi([toks(t) for t in texts])\n\n    def search(self, q, k=5, pool=20):\n        dense = np.argsort(-(self.vecs @ embed([q])[0]))[:pool]\n        bm = self.bm25.get_scores(toks(q))\n        sparse = [i for i in np.argsort(-bm)[:pool] if bm[i] > 0]\n        fused = {}\n        for ranking in (dense, sparse):\n            for rank, i in enumerate(ranking, 1):\n                fused[i] = fused.get(i, 0) + 1 / (60 + rank)\n        return sorted(fused, key=fused.get, reverse=True)[:k]\n\ndef build(docs, contextual):\n    ids, texts = [], []\n    for name, doc in docs.items():\n        for n, piece in enumerate(chunk(doc)):\n            if contextual:\n                piece = situate(doc, piece) + \"\\n\\n\" + piece\n            ids.append(f\"{name}#{n}\")\n            texts.append(piece)\n    return ids, Index(texts)\n\ndef recall(ids, index, qa, k=5):\n    def hit(item):\n        gold = set(item[\"gold\"])\n        top = {ids[i] for i in index.search(item[\"q\"], k)}\n        return len(top & gold) / len(gold)\n    return sum(hit(x) for x in qa) / len(qa)\n\nif __name__ == \"__main__\":\n    qa = json.load(open(sys.argv[1]))\n    docs = {p: open(p).read() for p in sys.argv[2:]}\n    for contextual in (False, True):\n        ids, index = build(docs, contextual)\n        r = recall(ids, index, qa)\n        print(f\"contextual={contextual} recall@5={r:.1%}\")\n    print(\"prompt tokens:\", usage)\n```\n\nRun it as `python ctx_eval.py qa.json docs/*.md`. Here is what each part is doing.\n\n`chunk` splits on fixed 500-word windows. Swap in your production chunker; both runs just have to use the same one, because the gold labels point at chunk positions.\n\n`situate` sends Anthropic's contextualizer prompt, copied from the [cookbook](https://github.com/anthropics/claude-cookbooks/blob/main/capabilities/contextual-embeddings/guide.ipynb). The document goes in the first content block with `cache_control`, and the chunk goes in a second block that is not cached. That ordering matters: the cache covers the prefix up to the marked block, so the document has to come first. The model is [Claude Haiku 5.5](https://platform.claude.com/docs/en/about-claude/models/overview), whose thinking is on by default. The [thinking docs](https://platform.claude.com/docs/en/build-with-claude/thinking) say Haiku 5.5 accepts `{\"type\": \"disabled\"}` at high effort or below, and the models overview lists its default effort as medium. Writing one sentence of context does not need reasoning tokens billed as output.\n\n`Index.search` runs both retrievers. The dense side is a plain dot product, which works as cosine similarity because [OpenAI embeddings are normalized to length 1](https://developers.openai.com/api/docs/guides/embeddings). The BM25 side lowercases and splits on word characters itself, since rank-bm25 does no preprocessing of its own. Chunks with a zero BM25 score are dropped so they don't sneak into the fusion. Anthropic's write-up says to merge the two result sets with \"rank fusion techniques\" without naming one, so this uses reciprocal rank fusion with the [k=60 constant Elasticsearch defaults to](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion).\n\n`build` is where the experiment lives. The `contextual` flag changes the text that gets embedded and indexed, and nothing else. Chunk IDs stay the same in both runs, so one label file scores both.\n\n`recall` averages, over all questions, the fraction of gold chunks that land in the top 5. For a question with one gold chunk, that is hit or miss.\n\nWithout caching, every chunk call re-sends the entire document. Ten chunks from one document means paying for that document ten times as input. With `cache_control` on the document block, the first call writes it to the cache and the next nine read it back.\n\nThe [prompt caching docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) price a 5-minute cache write at 1.25 times base input and a cache read at 0.1 times. On the [pricing page](https://platform.claude.com/docs/en/about-claude/pricing) as of October 2026, Claude Haiku 5.5 for prompts up to 100,000 tokens is $0.10 per million input tokens, $0.125 for 5-minute cache writes, $0.01 for cache hits and $0.50 for output. Prompts over 100,000 tokens move to a rate five times higher, so very long documents cost more per chunk.\n\nSome arithmetic from that table, using the shape Anthropic used for its own estimate: an 8,000-token document, ten 800-token chunks, about 50 tokens of instructions and 100 tokens of generated context per chunk. With caching, that is roughly $0.001 for the one cache write, $0.0007 for nine cache reads, $0.0009 for the uncached chunk text and $0.0005 for output. About $0.003 per document, or around $0.38 per million document tokens. Without caching, the same job comes to roughly $1.17 per million. That is my arithmetic from list prices, not a measured bill. Anthropic's 2024 figure for the same shape was [$1.02 per million document tokens](https://www.anthropic.com/news/contextual-retrieval), on Claude 3 Haiku pricing.\n\n*One write, many cheap reads: the document is cached and each chunk call reuses it.*\n\nThree details from the caching docs decide whether you actually get those reads:\n\nThe `prompt tokens` line at the end tells you how it went on your run. For comparison, the cookbook's own 737-chunk ingestion reported that 61.83% of all input tokens were read from cache, lower than the 70 to 80% its prose suggests. Read your counters before you trust anyone's estimate, including the one above.\n\nThe number this script prints is only as honest as `qa.json`. To label anything, you first need to see the chunk IDs. A helper that reuses the same chunker:\n\n``` python\nimport sys\nfrom ctx_eval import chunk\n\nfor path in sys.argv[1:]:\n    for n, piece in enumerate(chunk(open(path).read())):\n        print(f\"{path}#{n}\\t{piece[:90]}\")\n```\n\nImporting `ctx_eval` creates both API clients, so keep the environment variables set. Then write the label file:\n\n```\n[\n  {\"q\": \"How long do refunds take on annual plans?\",\n   \"gold\": [\"docs/billing.md#3\"]},\n  {\"q\": \"Which header carries the idempotency key?\",\n   \"gold\": [\"docs/api.md#7\", \"docs/api.md#8\"]}\n]\n```\n\nA few rules keep the set from flattering either run:\n\nWith 30 questions, each one is worth about 3.3 points of [recall@5](mailto:recall@5). A gap of one or two questions between runs is within noise. So look at the flips as well as the totals. Inside the `__main__` block, after each `build`, this prints a hit or miss per question:\n\n```\nfor item in qa:\n    top = {ids[i] for i in index.search(item[\"q\"])}\n    print(bool(top & set(item[\"gold\"])), item[\"q\"][:60])\n```\n\nQuestions that went from miss to hit tell you where the prefix helped. Questions that went from hit to miss deserve a closer read: when every chunk of one document gets a similar opening sentence, those chunks may start to look alike to the retriever.\n\nIf recall@5 barely moves, that is a real result. Your chunks may already be self-contained, or your baseline may already be high, and you just saved yourself an indexing step on every document update.\n\nIf it moves, [Anthropic's write-up](https://www.anthropic.com/news/contextual-retrieval) lists the next levers. They reranked by pulling the top 150 and narrowing to 20 with a [Cohere reranker](https://docs.cohere.com/docs/rerank), which is where the 67% figure came from. They tried passing 5, 10 and 20 chunks and found 20 worked best for them, while telling readers to test it on their own use case. They also suggest a contextualizer prompt tailored to your domain, such as one that includes a glossary of terms defined elsewhere in the knowledge base. Change one lever at a time and rerun the same frozen 30 questions.\n\nKeep the indexing cost in mind once this goes past an experiment. Contexts are generated at index time from the whole document, so when a document changes, any of its chunks' contexts can go stale. Re-contextualize the whole document, not only the edited chunk.\n\nThe script is the measuring stick; the decisions around it (how you chunk, when hybrid search beats dense alone, when a reranker earns its latency) are what the [RAG Pocket Guide](https://www.amazon.com/dp/B0GX2YDC5Z) covers, under its subtitle of retrieval, chunking and reranking patterns for production. Run the 30 questions first, then pick the chapter that matches the misses.", "url": "https://wpnews.pro/news/contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your", "canonical_source": "https://dev.to/gabrielanhaia/contextual-retrieval-in-80-lines-anthropics-chunk-prefix-trick-measured-on-your-docs-3350", "published_at": "2026-10-15 09:01:29+00:00", "updated_at": "2026-10-15 09:19:35.336125+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "natural-language-processing", "ai-search", "developer-tools"], "entities": ["Anthropic", "Claude 3 Haiku", "Gemini Text 004", "OpenAI", "rank-bm25", "NumPy", "Anthropic Python SDK", "OpenAI Python SDK"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your", "markdown": "https://wpnews.pro/news/contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your.md", "text": "https://wpnews.pro/news/contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your.txt", "jsonld": "https://wpnews.pro/news/contextual-retrieval-in-80-lines-anthropic-s-chunk-prefix-trick-measured-on-your.jsonld"}}