{"slug": "fix-the-query-before-the-reranker", "title": "Fix the Query Before the Reranker", "summary": "Trimming chat-context headers from 40 of 100 Hindsight recall queries lifted the MiniLM-L6 reranker from 0.092 to 0.437 nDCG@10 and Hindsight's default hs-rerank-1 from 0.352 to 0.546, according to a Vectorize.io blog post on Hindsight 1.2.1 run on a 16 GB M4 Mac mini. The gain from trimming roughly matches switching from MiniLM-L6 to hs-rerank-1, and the two improvements stack but not in full: after trimming, hs-rerank-1's lead over MiniLM-L6 is 0.110. In live traffic from 2026-11-12 to 11-19, 234 of 619 recall queries (38%), from 99 distinct queries, carried the chat-context header that the agent framework prepends to the user's latest message.", "body_md": "Article\n\n# Fix the Query Before the Reranker\n\nTrimming chat context from 40 Hindsight recall queries lifted the MiniLM-L6 reranker from 0.092 to 0.437 nDCG@10, about as much as switching to hs-rerank-1.\n\nWe run [Hindsight](https://github.com/vectorize-io/hindsight) 1.2.1 as the\nmemory of an AI assistant on a 16 GB M4 Mac mini. When recall quality\ndisappoints, the usual knobs are the cross-encoder reranker and how many\ncandidates it sees, and we measured both in\n[Going Back to Hindsight’s Default Reranker](https://feniix-hq.net/blog/back-to-the-default-reranker/).\nOn our queries that carried recent chat, the text the assistant sent as\nthe query mattered as much as the reranker, and more than its cap.\n\nForty of our 100 evaluation queries begin with a chat-context header: the agent framework pastes the recent conversation in front of the user’s message. Cutting each of them down to the user’s latest message lifted MiniLM-L6 (the smaller reranker we ran before) from 0.092 to 0.437 nDCG@10 and hs-rerank-1 (Hindsight’s default) from 0.352 to 0.546.\n\nThat is about as much as switching from MiniLM-L6 to hs-rerank-1, and the two gains stack, though not in full: after trimming, hs-rerank-1’s lead over MiniLM-L6 is 0.110. Trimming also reduced how deep in the candidate list the reranker had to look, though not by as much as we expected.\n\nHindsight 1.2.1, cap 300. Blind graded relevance (Claude Opus for every grading round after the first; Claude Sonnet confirms), paired over the 40 queries. The trim rule and the pass bar were fixed before measuring.\n\nTo try both levers yourself, the\n[reranker explainer](https://feniix-hq.net/demos/reranker/) lets you trim a query and move the\ncap, first on twelve invented memories and then on all 100 queries.\n\n## The chat header in recall queries\n\nA recall query is written by the agent framework, not typed by a person.\nWhen the assistant searches its memory while answering a chat message, the\nframework sends Hindsight the recent conversation, with the user’s latest\nmessage last. Context blocks are separated by a marker, and each chat line\nstarts with a header such as `#123 2026-11-18 20:44:32 UTC Ana (ana42):`.\n\nAn invented example, the same one the\n[explainer](https://feniix-hq.net/demos/reranker/#toy) uses:\n\n```\nAs sent:\n#4101 2026-11-18 20:41:07 UTC Ana (ana42): the staging banner\nbroke again after the Friday deploy <ctx> #4102 2026-11-18\n20:42:55 UTC Ben (benk): lunch is tacos today, and standup moved\nto 10 <ctx> #4103 2026-11-18 20:44:32 UTC Ana (ana42): Which\ndatabase did we pick for the billing service?\n\nTrimmed:\nWhich database did we pick for the billing service?\n```\n\nForty of the 100 queries in our evaluation set look like this. In a week of live traffic (2026-11-12 to 11-19), 234 of 619 recall queries did (38%), from 99 distinct queries. A cross-encoder reads the query and each memory together, so with the header in front, most of what it reads as “the query” is conversation the user did not ask about.\n\nWe fixed the trim rule and checked it on all 40 queries before measuring. Our version recognized a wrapped query by the fixed text the framework puts at its start; the version below looks for the marker instead. The two give the same output except on a query that contains the marker without that opening, which ours left alone. The marker here is a placeholder:\n\n``` python\nimport re\n\n# The marker your agent framework puts between blocks of chat context.\n# Ours is a short tag; replace it with whatever yours writes.\nCONTEXT_SEPARATOR = \"<ctx>\"\n\n# A chat-history line prefix such as\n# \"#123 2026-11-18 20:44:32 UTC Ana (ana42): \"\nCHAT_HEADER = re.compile(\n    r\"^#\\d+\\s+\\d{4}-\\d{2}-\\d{2}\\s+\\d{2}:\\d{2}:\\d{2}\"\n    r\"\\s+\\S+\\s+[^:()]*\\([^)]*\\):\\s*\"\n)\n\ndef trim(query: str) -> str:\n    \"\"\"Reduce a chat-wrapped recall query to the user's latest message.\"\"\"\n    # Plain queries pass through untouched.\n    if CONTEXT_SEPARATOR not in query:\n        return query\n    # The newest block is the last one.\n    latest = query.rsplit(CONTEXT_SEPARATOR, 1)[-1].strip()\n    # Drop who said it and when.\n    return CHAT_HEADER.sub(\"\", latest).strip()\n```\n\nOn the 40 queries the median length went from 190.5 to 67 characters, and\nnone came out empty. The rule is specific to one framework’s format; the\npoint is to keep the user’s message and drop the rest. The explainer shows\n[the same trim on a toy query](https://feniix-hq.net/demos/reranker/?form=trimmed#toy), with the\nscores hs-rerank-1 gives each memory before and after.\n\nWhy the rule looks like this:\n\n- It keeps the latest message because that is what the user asked; the earlier turns are context around it.\n- It drops the `#id date name:` tag because a cross-encoder reads the query\nand each memory as one input, so names, handles and timestamps become text\nit tries to match.\n- It is a fixed rule rather than a model rewrite of the query: that costs nothing at query time, gives the same output every time, and could be checked on all 40 queries before measuring. We did not test a model rewrite.\n- It runs before recall rather than only before the reranker, because retrieval reads the query too; lever 1 measures the two together.\n\n## Lever 1: the query\n\nThe chart shows nDCG@10 for the 40 headed queries, as sent and trimmed, for each model at cap 300. nDCG@10 scores the first ten results by their graded relevance, weighting the top positions most; 1.0 is the best possible order.\n\nBoth models improved, MiniLM-L6 the most: its share of queries with an answer at rank 1 went from 0.075 to 0.375. The result held with a second grader (Claude Sonnet: +0.395 and +0.215) and at cap 200 (+0.351 and +0.202).\n\nTrimmed MiniLM-L6 scored 0.085 above untrimmed hs-rerank-1, with an interval of [−0.011, +0.179]. The interval includes zero, so we cannot separate the two: on these queries, cleaning the query was worth about as much as the model switch.\n\nTrimming also changes what retrieval returns, so the gain is retrieval and reranking together; we did not separate them. Of 560 query–memory pairs in the new top 10s that had never been graded, 558 came from the trimmed queries’ candidate lists. We graded 559 of them (the missing one counts as not relevant) against the original query’s need, which is the user’s message either way.\n\nThese scores also use a larger set of graded memories than the default-reranker article’s, so compare them within this chart.\n\n## Lever 2: the candidate cap\n\nHindsight reranks only the top N of the fused candidate list, where N is\n`HINDSIGHT_API_RERANKER_MAX_CANDIDATES`. A smaller cap is faster, and it\ncosts quality only when relevant memories sit below N. On our pools,\nthat happened more often for headed queries than for plain ones.\n\nAt cap 300, hs-rerank-1 took 4.1 of each headed query’s top 10 from below rank 100 of the fused list. For the same queries trimmed it took 3.1, and for plain queries 1.4. Counting only relevant ones, the figures were 0.93, 0.72 and 0.38 per query. Trimming moved headed queries toward plain ones, not all the way. These counts were not part of a planned test.\n\nThe planned tests agree. Cutting the cap from 300 to 100 cost headed\nqueries as sent 0.063 nDCG@10, which failed our bar. On clean queries (the\n60 plain ones plus the 40 trimmed) it cost 0.006, which passed with both\ngraders. On the 40 trimmed queries alone it cost 0.019, and the graders\ndisagreed on whether that passed. So cut the header first, then test the\ncap on your own queries;\n[Going Back to Hindsight’s Default Reranker](https://feniix-hq.net/blog/back-to-the-default-reranker/#cost-the-cpu-and-the-apple-gpu)\nhas the cost side.\n\n## Lever 3: the model\n\nTrimming does not replace a better reranker. hs-rerank-1 still scored above MiniLM-L6 at cap 300 on trimmed queries, where it was better on 24 queries and worse on 9, and on plain ones:\n\n| Queries | hs-rerank-1 minus MiniLM-L6 | 95% interval | \n|---|---|---|\n| 40 headed, trimmed | +0.110 | [+0.044, +0.173] | \n| 60 plain (first test, all 100 queries, split after the fact) | +0.090 | [+0.041, +0.146] | \n\nMiniLM-L6 also got less out of depth: of the items each model pulled from below rank 100 into its top 10, 25% were relevant for hs-rerank-1 and 10% for MiniLM-L6 (first test). On headed queries the two levers together took nDCG@10 from 0.092 (MiniLM-L6, as sent) to 0.546 (hs-rerank-1, trimmed).\n\n## Long prompts\n\nRecall queries can also be long task prompts that end in a question. We tested six synthetic ones, cut to their final question, with hs-rerank-1 on the CPU, one at a time, on a warm machine:\n\n|  | As sent | Trimmed | \n|---|---|---|\n| Length | 3,125–4,649 characters | about 80 characters | \n| Median rerank time | 46.3 s | 11.1 s | \n| Peak memory | 3.82 GiB | 2.34 GiB | \n\nNone of our evaluation queries looked like that; a week of live traffic held one distinct long query (24 of 619 recalls), and it fitted neither trim pattern.\n\n## Check your own queries\n\n1. Sample a week of recall queries and count how many carry context the user did not type. Ours was 38%.\n2. Freeze each query’s candidate list once and rerank it offline for every setting, so the settings differ only in what you changed.\n3. Grade the top 10s blind, on a small scale such as 0 to 2, and use a second grader on a sample.\n4. Compare settings paired by query, and write the pass bar down before you look.\n\nIf Hindsight calls a TEI-compatible reranker, the recording proxy in\n[mps-rerank-server](https://github.com/feniix/mps-rerank-server) saves the\nexact requests it sends, which covers step 1 and the query text for step 2.\n\n## Limits\n\n- All of this is one assistant’s traffic: 100 evaluation queries, 40 of them headed.\n- The evaluation set kept only queries under 300 characters, so longer chat-wrapped queries were not tested.\n- The graders are language models (the first round’s model was not recorded; every later round used Claude Opus), and no human checked their labels.\n- The trim rule fits one framework’s format.\n- The long-prompt numbers come from synthetic prompts on the CPU, and the depth counts in lever 2 were computed after the fact.\n- The trim was measured offline, on replayed queries; it is not deployed in the assistant yet, so there is no live result.", "url": "https://wpnews.pro/news/fix-the-query-before-the-reranker", "canonical_source": "https://feniix-hq.net/blog/fix-the-query-before-the-reranker/", "published_at": "2026-11-20 00:00:00+00:00", "updated_at": "2026-11-20 13:41:58.070949+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools"], "entities": ["Hindsight", "Vectorize.io", "MiniLM-L6", "hs-rerank-1", "Claude Opus", "Claude Sonnet"], "also_reported_by": [], "takeaway": "Trimming chat-context headers from recall queries before reranking raised nDCG@10 from 0.092 to 0.437 for MiniLM-L6 and from 0.352 to 0.546 for hs-rerank-1 across 40 of 100 queries.", "recommendation": {"advice": "Strip the chat-context header and keep only the user's latest message before sending a recall query to the reranker; trimming lifted MiniLM-L6 from 0.092 to 0.437 nDCG@10.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/fix-the-query-before-the-reranker", "markdown": "https://wpnews.pro/news/fix-the-query-before-the-reranker.md", "text": "https://wpnews.pro/news/fix-the-query-before-the-reranker.txt", "jsonld": "https://wpnews.pro/news/fix-the-query-before-the-reranker.jsonld"}}