{"slug": "going-back-to-hindsight-s-default-reranker", "title": "Going Back to Hindsight's Default Reranker", "summary": "On 100 real recall queries, Hindsight's default reranker hs-rerank-1 at a candidate cap of 100 beat MiniLM-L6 at cap 300 by 0.090 nDCG@10, according to a benchmark run by the Hindsight team on Hindsight 1.2.1 on a 16 GB M4 Mac mini. Cutting hs-rerank-1's cap from 300 to 100 candidates cost nothing measurable on clean queries but lost quality on the 40 queries carrying a chat-context header, and on the Mac's CPU the model missed a 5-second budget under load even at cap 200 from a cool start, while on the GPU behind a small server it reranked 100 candidates in 0.70 s at p95. Reranking remains the slow part of a recall, with the median rerank at 80–88% of the median recall in the team's production log even with MiniLM-L6.", "body_md": "Article\n\n# Going Back to Hindsight's Default Reranker\n\nOn 100 clean recall queries, Hindsight's hs-rerank-1 at cap 100 beat MiniLM-L6 at cap 300 by 0.090 nDCG@10. On an Apple Silicon Mac it needs the GPU.\n\nWe run [Hindsight](https://github.com/vectorize-io/hindsight) 1.2.1 as the\nmemory of an AI assistant on a 16 GB M4 Mac mini. Every\n[recall](https://feniix-hq.net/blog/recall-reflect-and-the-knowledge-page/) gathers candidates\nfrom several retrievers, merges them with reciprocal rank fusion (RRF),\nand has a cross-encoder rerank the top of the merged list. How many\ncandidates it reranks is `HINDSIGHT_API_RERANKER_MAX_CANDIDATES`, 300 by\ndefault.\n\nHindsight’s default reranker is\n[hs-rerank-1](https://huggingface.co/hindsight-ai/hs-rerank-1), which\nreads up to 8,192 tokens per pair. Hindsight 1.2.0 made it the default;\nbefore that it was\n[`cross-encoder/ms-marco-MiniLM-L-6-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2),\na smaller model that reads at most 512 tokens per pair. When we upgraded\nto 1.2.1 we kept MiniLM-L6, because hs-rerank-1 was too slow on the Mac’s\nCPU. Reranking is still the slow part of a recall. Even with MiniLM-L6,\nthe median rerank is 80–88% of the median recall in our production log.\n\nWe measured whether going back to the default is worth it, at which cap, and what it takes to run. hs-rerank-1 ranked significantly better than MiniLM-L6 on 100 real recall queries. Cutting its cap from 300 to 100 candidates cost nothing measurable on clean queries (the question alone), but cost quality on queries that carried a chat-context header. On the Mac’s CPU it missed a 5-second budget under load even at cap 200, from a cool start. On the GPU, behind a small server, it reranked 100 candidates in 0.70 s at p95.\n\nHindsight 1.2.1 on a 16 GB M4 Mac mini. Quality from blind graded relevance (Claude Opus; Claude Sonnet as a second grader), paired over queries, pre-registered. All timings from a cool start; GPU one request at a time, CPU with four requests in flight.\n\n## Setup\n\nWe measured on 100 recall queries the assistant sent while working, taken from its log, and put every one through the same five steps.\n\n- **Queries.** Forty of the 100 begin with a chat-context header: the agent\nframework prefixes the user’s latest message with recent conversation.\n- **RRF merge.** We replayed each query once against the live engine\nwith tracing on and froze the top 300 of its merged list, which we call\nthe query’s pool. Those lists held 1,826 to 2,669 candidates, so even\ncap 300 reranks a small slice.\n- **Cap N.** A cap of N reranks the top N of the frozen list, which is what\nthe engine’s cap does.\n- **Reranker.** Both models ran through Hindsight’s own local\ncross-encoder class, at pinned revisions, on the CPU in FP32. The GPU\nserver in the cost section runs hs-rerank-1 in FP16, the format its\nweights ship in; on a second install of ours (described below), FP16\ngave the same rankings as FP32.\n- **Top 10.** Each candidate that reached any top 10 got a blind grade of 0,\n1 or 2 (not relevant, relevant, answers the query), in random order. We\nreused 2,581 grades from an earlier evaluation of the same queries;\nClaude Opus graded the rest, and Claude Sonnet graded the same new pairs\nas a check. On the 946 pairs of the main round they agreed exactly on 87%\n(quadratic weighted kappa 0.88), with Opus the stricter.\n\nThe numbers below use Opus grades; Sonnet’s lead to the same verdicts except where noted. The metric is nDCG@10: it scores the first ten results by their grades, weighting the top positions most, so 1.0 is the best possible order. Differences are paired by query, with 95% bootstrap intervals (10,000 resamples). Before each test we committed its hypotheses and pass bar. For the cap, a lower cap passes when its nDCG@10 is within 0.02 of cap 300’s and the lower end of the interval is above −0.05.\n\nEvery benchmark timing from the Mac mini starts from a cool machine. Under sustained load this Mac raised its thermal pressure and lowered its clocks. One afternoon the CPU load test ran 2.5–3.7 times slower than it did from a cool start, so we leave out every timing from that afternoon.\n\n## Quality: hs-rerank-1 against MiniLM-L6\n\nThe chart shows each model’s mean nDCG@10: on all 100 queries, split by whether they carry the header, and, in a later test, on clean queries. Clean queries are the 60 plain ones plus the 40 headed ones cut down to the user’s message.\n\nAcross the 100 queries as the assistant sent them, hs-rerank-1 scored 0.206 higher. Most of that gap comes from the 40 queries with a chat header, where MiniLM-L6 scores 0.119 against hs-rerank-1’s 0.500. On the 60 plain queries the gap is smaller, +0.090, but its interval is still well clear of zero.\n\nThat split was not planned, so we tested clean queries again on pools captured fresh a day later. There, hs-rerank-1 at cap 100 beat MiniLM-L6 at cap 300 by +0.090 [+0.052, +0.129] (+0.099 with Sonnet’s grades), better on 59 queries and worse on 24. On these clean queries hs-rerank-1 scored 0.625, 0.624 and 0.631 at caps 100, 200 and 300; MiniLM-L6 scored 0.535 at 300.\n\n## Choosing the reranker cap\n\nThe cap is a known cost. One Hindsight user reported recalls of more than\n30 seconds at the default of 300 with a local reranker, and asked for a\ndefault of 50\n([hindsight#2554](https://github.com/vectorize-io/hindsight/issues/2554)).\nWhat a lower cap costs in quality depends on the queries.\n\nThe first pre-registered test ran on the 100 queries as sent. For hs-rerank-1, cap 200 passed (−0.008 against cap 300), and caps 100 and 50 failed (−0.044 and −0.078). MiniLM-L6 barely changed with the cap: every cap was within 0.011 of cap 300. In cap 300’s top 10, both models placed about a quarter of their items from below RRF rank 100, but only 10% of MiniLM-L6’s deep picks were relevant, against 25% of hs-rerank-1’s, so MiniLM-L6 had little to lose.\n\nSplitting that result by query type, again after the fact, located the loss. On the 60 plain queries, cap 100 cost −0.003; on the 40 with a chat header, −0.106. The likely reason is that the header pushes relevant memories down the merged list: in cap 300’s top 10, hs-rerank-1 placed 4.2 items per headed query from below RRF rank 100, against 1.5 per plain query. A cap of 100 never shows it those candidates.\n\nThe second pre-registered test checked this on fresh pools, with the headed queries left as sent as a control. The table puts both tests side by side.\n\n| hs-rerank-1, query set | Cap 300 score | Cap 100 vs 300 | Cap 200 vs 300 | \n|---|---|---|---|\n| All 100, as sent first test | 0.615 | −0.044 fail [−0.082, −0.010] | −0.008 pass [−0.027, +0.009] | \n| Clean queries second test | 0.631 | −0.006 pass [−0.024, +0.010] | −0.007 pass [−0.020, +0.004] | \n| 40 headed, as sent control, second test | 0.350 | −0.063 fail [−0.110, −0.022] | not run | \n\nChange in hs-rerank-1’s nDCG@10 at a lower cap against cap 300, paired by query, with the 95% bootstrap interval below it. A lower cap passes when the change is −0.02 or better and the interval’s lower end is above −0.05, the bar set before each test. Clean queries are the 60 plain ones plus the 40 headed ones cut to the user’s message. Opus grades; Sonnet’s give the same verdicts. The second test ran on pools captured a day after the first. Compare scores within a test, not across tests: grading the trimmed queries found more relevant memories for the 40 headed queries, which raises the ideal ranking nDCG@10 is measured against. Rescored with those grades, the first test’s 0.500 for them becomes 0.351.\n\nWith clean queries, cap 100 passed with both graders: −0.006 with Opus, −0.002 with Sonnet. On the headed queries left as sent, cap 100 failed again, −0.063, so the first result replicated on new pools. On the GPU, cap 100 takes about half the time of cap 200 (next section) for the same quality, as long as the query holds only the question.\n\n```\nHINDSIGHT_API_RERANKER_MAX_CANDIDATES: \"100\"     # recall queries are just the question\n# HINDSIGHT_API_RERANKER_MAX_CANDIDATES: \"200\"   # queries carry conversation context\n```\n\nHindsight can also cap each recall budget separately with\n`HINDSIGHT_API_RERANKER_MAX_CANDIDATES_LOW`, `_MID` and `_HIGH`; our\nqueries ran at the `mid` budget. How we cut the header, and how much that\nalone changes, will be a separate post.\n\n## Cost: the CPU and the Apple GPU\n\nOn an Apple Silicon Mac, Hindsight’s built-in `local` reranker runs on the\nCPU whether Hindsight runs in Docker or natively. In Docker, containers\nrun in a Linux virtual machine that has no access to Metal, so PyTorch\ninside it sees no GPU. In a native install, Hindsight 1.2.1 never selects\nMPS, PyTorch’s backend for the Apple GPU, for the local reranker: MPS\nkeeps a cache for every input shape, which grows without bound on\nvariable-length inputs, and concurrent encodes abort the process\n([hindsight#4412](https://github.com/vectorize-io/hindsight/issues/4412)).\n\nThe CPU is too slow for hs-rerank-1. We replayed the load profile from our engine-upgrade test through the reranker alone, so the rest of a recall comes on top: 24 queries, four in flight, six of them long synthetic prompts. From a cool start, at cap 200, hs-rerank-1 took 5.95 s at p50 and 40.1 s at p95, and all 24 reranks went over 5 s. MiniLM-L6 at cap 300 took 6.2 s and 15.4 s on the same test, with 22 of 24 over 5 s; the long prompts set the p95. For comparison, the automatic recalls in our production log, with MiniLM-L6, take 3.3 s at p50 and 4.4 s at p95, whole recall included.\n\nOn the GPU, hs-rerank-1 stays well inside the 5-second budget. We\npublished the server we use as\n[mps-rerank-server](https://github.com/feniix/mps-rerank-server): a native\nmacOS process that keeps hs-rerank-1 on the GPU in FP16 and answers\nHindsight’s `tei` reranker provider over HTTP. Run on the same Mac mini\nfrom a cool start, one request at a time, with the 100 queries as sent:\n\n| GPU, 100 requests per cap | Mean | p50 | p95 | Max | \n|---|---|---|---|---|\n| Cap 100 | 0.52 s | 0.47 s | 0.70 s | 1.12 s | \n| Cap 200 | 0.93 s | 0.86 s | 1.26 s | 1.66 s | \n\nPer rerank, measured at the client. macOS thermal pressure stayed at Nominal throughout; no request failed. Cap 200 takes 1.8× as long as cap 100.\n\nTwo costs come with the GPU. The server runs one forward pass at a time, so concurrent recalls queue behind each other. And the MPS cache growth that keeps Hindsight off MPS applies here too: the server’s footprint went from 1,640 MB to 2,135 MB over the session’s 400 requests (the table’s 200 and 200 synthetic ones). The server releases its cached buffers every hour, and a restart resets the footprint.\n\nHindsight’s other Apple-GPU route is its in-process `jina-mlx` provider,\nwhich runs jina-reranker-v3 under a non-commercial licence (CC BY-NC 4.0).\nHindsight also serves hs-rerank-1 as a hosted service, Hindsight Inference\n(`HINDSIGHT_API_RERANKER_PROVIDER=hindsight`). We measured neither.\n\nPointing Hindsight at the server takes two settings, and a second, fallback reranker keeps recall working if the server is down:\n\n```\nHINDSIGHT_API_RERANKER_PROVIDER: tei\nHINDSIGHT_API_RERANKER_TEI_URL: http://host.docker.internal:8081   # 127.0.0.1 for a native install\nHINDSIGHT_API_RERANKER_1_PROVIDER: local\nHINDSIGHT_API_RERANKER_1_LOCAL_MODEL: cross-encoder/ms-marco-MiniLM-L-6-v2\n```\n\n## A second install\n\nOur personal Hindsight install, which holds coding-agent memories (the one\nbehind [our LLM comparison](https://feniix-hq.net/blog/hindsight-llm-comparison/)), ran the\nsame cap test on its own 106 queries, with the same server code on an M5\nMax. hs-rerank-1 scored 0.679, 0.693 and 0.695 at caps 100, 200 and 300.\nCap 100 was 0.017 below cap 300 [−0.032, −0.003], inside that install’s\npre-set tolerance of 0.02, so it kept cap 100, at 0.27 s per rerank\nagainst 0.96 s at cap 300. On a different corpus cap 100 again cost a\nlittle: more than on our clean queries (−0.006), less than on our queries\nas sent (−0.044).\n\n## What we plan, and what is left\n\nNone of this is deployed yet. The rollout ends with a soak that records latency together with the thermal pressure level, since throttling is what slowed the afternoon runs we left out.\n\nEach timing comes from one run on one Mac. The graders are language models, and no human checked their labels. On the new pairs of the second test, Opus and Sonnet agreed less than before (quadratic weighted kappa 0.66, Sonnet more lenient). On the 40 headed queries cut to the user’s message, taken alone, cap 100 was unresolved: it passed with Opus and failed with Sonnet on interval width. The 100 queries come from one assistant’s traffic.\n\nThe server’s repository includes the benchmark: a proxy that records the requests Hindsight sends, and a replay script that reports latency, memory and thermal pressure at any cap. It can also build synthetic requests from token lengths alone, but on our data those ran 10–14% faster at p50 than the real ones they copied, so for numbers that matter, replay real requests.", "url": "https://wpnews.pro/news/going-back-to-hindsight-s-default-reranker", "canonical_source": "https://feniix-hq.net/blog/back-to-the-default-reranker/", "published_at": "2026-11-19 00:00:00+00:00", "updated_at": "2026-11-19 21:12:15.627355+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning"], "entities": ["Hindsight", "hs-rerank-1", "MiniLM-L6", "Apple", "M4 Mac mini", "Claude Opus", "Claude Sonnet"], "also_reported_by": [], "takeaway": "hs-rerank-1 at a 100-candidate cap beat MiniLM-L6 at cap 300 by 0.090 nDCG@10 on 100 clean recall queries, but needs a GPU to rerank 100 candidates in 0.70 s at p95 versus missing a 5-second CPU budget under load.", "recommendation": {"advice": "Run the cross-encoder reranker on a GPU and cap candidates at 100 rather than 300; the source measured no nDCG@10 loss on clean queries and 0.70 s p95, but expect quality loss on chat-context-prefixed queries.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/going-back-to-hindsight-s-default-reranker", "markdown": "https://wpnews.pro/news/going-back-to-hindsight-s-default-reranker.md", "text": "https://wpnews.pro/news/going-back-to-hindsight-s-default-reranker.txt", "jsonld": "https://wpnews.pro/news/going-back-to-hindsight-s-default-reranker.jsonld"}}