cd /news/ai-infrastructure/going-back-to-hindsight-s-default-re… · home › topics › ai-infrastructure › article
[ARTICLE · art-172843] src=feniix-hq.net ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Going Back to Hindsight's Default Reranker

On 100 real recall queries, Hindsight's default reranker hs-rerank-1 at a candidate cap of 100 beat MiniLM-L6 at cap 300 by 0.090 nDCG@10, according to a benchmark run by the Hindsight team on Hindsight 1.2.1 on a 16 GB M4 Mac mini. Cutting hs-rerank-1's cap from 300 to 100 candidates cost nothing measurable on clean queries but lost quality on the 40 queries carrying a chat-context header, and on the Mac's CPU the model missed a 5-second budget under load even at cap 200 from a cool start, while on the GPU behind a small server it reranked 100 candidates in 0.70 s at p95. Reranking remains the slow part of a recall, with the median rerank at 80–88% of the median recall in the team's production log even with MiniLM-L6.

recommendation

Run the cross-encoder reranker on a GPU and cap candidates at 100 rather than 300; the source measured no nDCG@10 loss on clean queries and 0.70 s p95, but expect quality loss on chat-context-prefixed queries.

more in playbook: Retrieval and RAG
by read11 min views1 publishedNov 19, 2026
Going Back to Hindsight's Default Reranker
Image: Feniix-Hq (auto-discovered)

Article

On 100 clean recall queries, Hindsight's hs-rerank-1 at cap 100 beat MiniLM-L6 at cap 300 by 0.090 nDCG@10. On an Apple Silicon Mac it needs the GPU.

We run Hindsight 1.2.1 as the memory of an AI assistant on a 16 GB M4 Mac mini. Every recall gathers candidates from several retrievers, merges them with reciprocal rank fusion (RRF), and has a cross-encoder rerank the top of the merged list. How many candidates it reranks is HINDSIGHT_API_RERANKER_MAX_CANDIDATES, 300 by default.

Hindsight’s default reranker is hs-rerank-1, which reads up to 8,192 tokens per pair. Hindsight 1.2.0 made it the default; before that it was cross-encoder/ms-marco-MiniLM-L-6-v2, a smaller model that reads at most 512 tokens per pair. When we upgraded to 1.2.1 we kept MiniLM-L6, because hs-rerank-1 was too slow on the Mac’s CPU. Reranking is still the slow part of a recall. Even with MiniLM-L6, the median rerank is 80–88% of the median recall in our production log.

We measured whether going back to the default is worth it, at which cap, and what it takes to run. hs-rerank-1 ranked significantly better than MiniLM-L6 on 100 real recall queries. Cutting its cap from 300 to 100 candidates cost nothing measurable on clean queries (the question alone), but cost quality on queries that carried a chat-context header. On the Mac’s CPU it missed a 5-second budget under load even at cap 200, from a cool start. On the GPU, behind a small server, it reranked 100 candidates in 0.70 s at p95.

Hindsight 1.2.1 on a 16 GB M4 Mac mini. Quality from blind graded relevance (Claude Opus; Claude Sonnet as a second grader), paired over queries, pre-registered. All timings from a cool start; GPU one request at a time, CPU with four requests in flight.

Setup #

We measured on 100 recall queries the assistant sent while working, taken from its log, and put every one through the same five steps.

  • Queries. Forty of the 100 begin with a chat-context header: the agent framework prefixes the user’s latest message with recent conversation.
  • RRF merge. We replayed each query once against the live engine with tracing on and froze the top 300 of its merged list, which we call the query’s pool. Those lists held 1,826 to 2,669 candidates, so even cap 300 reranks a small slice.
  • Cap N. A cap of N reranks the top N of the frozen list, which is what the engine’s cap does.
  • Reranker. Both models ran through Hindsight’s own local cross-encoder class, at pinned revisions, on the CPU in FP32. The GPU server in the cost section runs hs-rerank-1 in FP16, the format its weights ship in; on a second install of ours (described below), FP16 gave the same rankings as FP32.
  • Top 10. Each candidate that reached any top 10 got a blind grade of 0, 1 or 2 (not relevant, relevant, answers the query), in random order. We reused 2,581 grades from an earlier evaluation of the same queries; Claude Opus graded the rest, and Claude Sonnet graded the same new pairs as a check. On the 946 pairs of the main round they agreed exactly on 87% (quadratic weighted kappa 0.88), with Opus the stricter.

The numbers below use Opus grades; Sonnet’s lead to the same verdicts except where noted. The metric is nDCG@10: it scores the first ten results by their grades, weighting the top positions most, so 1.0 is the best possible order. Differences are paired by query, with 95% bootstrap intervals (10,000 resamples). Before each test we committed its hypotheses and pass bar. For the cap, a lower cap passes when its nDCG@10 is within 0.02 of cap 300’s and the lower end of the interval is above −0.05.

Every benchmark timing from the Mac mini starts from a cool machine. Under sustained load this Mac raised its thermal pressure and lowered its clocks. One afternoon the CPU load test ran 2.5–3.7 times slower than it did from a cool start, so we leave out every timing from that afternoon.

Quality: hs-rerank-1 against MiniLM-L6 #

The chart shows each model’s mean nDCG@10: on all 100 queries, split by whether they carry the header, and, in a later test, on clean queries. Clean queries are the 60 plain ones plus the 40 headed ones cut down to the user’s message.

Across the 100 queries as the assistant sent them, hs-rerank-1 scored 0.206 higher. Most of that gap comes from the 40 queries with a chat header, where MiniLM-L6 scores 0.119 against hs-rerank-1’s 0.500. On the 60 plain queries the gap is smaller, +0.090, but its interval is still well clear of zero.

That split was not planned, so we tested clean queries again on pools captured fresh a day later. There, hs-rerank-1 at cap 100 beat MiniLM-L6 at cap 300 by +0.090 [+0.052, +0.129] (+0.099 with Sonnet’s grades), better on 59 queries and worse on 24. On these clean queries hs-rerank-1 scored 0.625, 0.624 and 0.631 at caps 100, 200 and 300; MiniLM-L6 scored 0.535 at 300.

Choosing the reranker cap #

The cap is a known cost. One Hindsight user reported recalls of more than 30 seconds at the default of 300 with a local reranker, and asked for a default of 50 (hindsight#2554). What a lower cap costs in quality depends on the queries.

The first pre-registered test ran on the 100 queries as sent. For hs-rerank-1, cap 200 passed (−0.008 against cap 300), and caps 100 and 50 failed (−0.044 and −0.078). MiniLM-L6 barely changed with the cap: every cap was within 0.011 of cap 300. In cap 300’s top 10, both models placed about a quarter of their items from below RRF rank 100, but only 10% of MiniLM-L6’s deep picks were relevant, against 25% of hs-rerank-1’s, so MiniLM-L6 had little to lose.

Splitting that result by query type, again after the fact, located the loss. On the 60 plain queries, cap 100 cost −0.003; on the 40 with a chat header, −0.106. The likely reason is that the header pushes relevant memories down the merged list: in cap 300’s top 10, hs-rerank-1 placed 4.2 items per headed query from below RRF rank 100, against 1.5 per plain query. A cap of 100 never shows it those candidates.

The second pre-registered test checked this on fresh pools, with the headed queries left as sent as a control. The table puts both tests side by side.

hs-rerank-1, query set Cap 300 score Cap 100 vs 300 Cap 200 vs 300
All 100, as sent first test 0.615 −0.044 fail [−0.082, −0.010] −0.008 pass [−0.027, +0.009]
Clean queries second test 0.631 −0.006 pass [−0.024, +0.010] −0.007 pass [−0.020, +0.004]
40 headed, as sent control, second test 0.350 −0.063 fail [−0.110, −0.022] not run

Change in hs-rerank-1’s nDCG@10 at a lower cap against cap 300, paired by query, with the 95% bootstrap interval below it. A lower cap passes when the change is −0.02 or better and the interval’s lower end is above −0.05, the bar set before each test. Clean queries are the 60 plain ones plus the 40 headed ones cut to the user’s message. Opus grades; Sonnet’s give the same verdicts. The second test ran on pools captured a day after the first. Compare scores within a test, not across tests: grading the trimmed queries found more relevant memories for the 40 headed queries, which raises the ideal ranking nDCG@10 is measured against. Rescored with those grades, the first test’s 0.500 for them becomes 0.351.

With clean queries, cap 100 passed with both graders: −0.006 with Opus, −0.002 with Sonnet. On the headed queries left as sent, cap 100 failed again, −0.063, so the first result replicated on new pools. On the GPU, cap 100 takes about half the time of cap 200 (next section) for the same quality, as long as the query holds only the question.

HINDSIGHT_API_RERANKER_MAX_CANDIDATES: "100"     # recall queries are just the question

Hindsight can also cap each recall budget separately with HINDSIGHT_API_RERANKER_MAX_CANDIDATES_LOW, _MID and _HIGH; our queries ran at the mid budget. How we cut the header, and how much that alone changes, will be a separate post.

Cost: the CPU and the Apple GPU #

On an Apple Silicon Mac, Hindsight’s built-in local reranker runs on the CPU whether Hindsight runs in Docker or natively. In Docker, containers run in a Linux virtual machine that has no access to Metal, so PyTorch inside it sees no GPU. In a native install, Hindsight 1.2.1 never selects MPS, PyTorch’s backend for the Apple GPU, for the local reranker: MPS keeps a cache for every input shape, which grows without bound on variable-length inputs, and concurrent encodes abort the process (hindsight#4412).

The CPU is too slow for hs-rerank-1. We replayed the load profile from our engine-upgrade test through the reranker alone, so the rest of a recall comes on top: 24 queries, four in flight, six of them long synthetic prompts. From a cool start, at cap 200, hs-rerank-1 took 5.95 s at p50 and 40.1 s at p95, and all 24 reranks went over 5 s. MiniLM-L6 at cap 300 took 6.2 s and 15.4 s on the same test, with 22 of 24 over 5 s; the long prompts set the p95. For comparison, the automatic recalls in our production log, with MiniLM-L6, take 3.3 s at p50 and 4.4 s at p95, whole recall included.

On the GPU, hs-rerank-1 stays well inside the 5-second budget. We published the server we use as mps-rerank-server: a native macOS process that keeps hs-rerank-1 on the GPU in FP16 and answers Hindsight’s tei reranker provider over HTTP. Run on the same Mac mini from a cool start, one request at a time, with the 100 queries as sent:

GPU, 100 requests per cap Mean p50 p95 Max
Cap 100 0.52 s 0.47 s 0.70 s 1.12 s
Cap 200 0.93 s 0.86 s 1.26 s 1.66 s

Per rerank, measured at the client. macOS thermal pressure stayed at Nominal throughout; no request failed. Cap 200 takes 1.8× as long as cap 100.

Two costs come with the GPU. The server runs one forward pass at a time, so concurrent recalls queue behind each other. And the MPS cache growth that keeps Hindsight off MPS applies here too: the server’s footprint went from 1,640 MB to 2,135 MB over the session’s 400 requests (the table’s 200 and 200 synthetic ones). The server releases its cached buffers every hour, and a restart resets the footprint.

Hindsight’s other Apple-GPU route is its in-process jina-mlx provider, which runs jina-reranker-v3 under a non-commercial licence (CC BY-NC 4.0). Hindsight also serves hs-rerank-1 as a hosted service, Hindsight Inference (HINDSIGHT_API_RERANKER_PROVIDER=hindsight). We measured neither.

Pointing Hindsight at the server takes two settings, and a second, fallback reranker keeps recall working if the server is down:

HINDSIGHT_API_RERANKER_PROVIDER: tei
HINDSIGHT_API_RERANKER_TEI_URL: http://host.docker.internal:8081   # 127.0.0.1 for a native install
HINDSIGHT_API_RERANKER_1_PROVIDER: local
HINDSIGHT_API_RERANKER_1_LOCAL_MODEL: cross-encoder/ms-marco-MiniLM-L-6-v2

A second install #

Our personal Hindsight install, which holds coding-agent memories (the one behind our LLM comparison), ran the same cap test on its own 106 queries, with the same server code on an M5 Max. hs-rerank-1 scored 0.679, 0.693 and 0.695 at caps 100, 200 and 300. Cap 100 was 0.017 below cap 300 [−0.032, −0.003], inside that install’s pre-set tolerance of 0.02, so it kept cap 100, at 0.27 s per rerank against 0.96 s at cap 300. On a different corpus cap 100 again cost a little: more than on our clean queries (−0.006), less than on our queries as sent (−0.044).

What we plan, and what is left #

None of this is deployed yet. The rollout ends with a soak that records latency together with the thermal pressure level, since throttling is what slowed the afternoon runs we left out.

Each timing comes from one run on one Mac. The graders are language models, and no human checked their labels. On the new pairs of the second test, Opus and Sonnet agreed less than before (quadratic weighted kappa 0.66, Sonnet more lenient). On the 40 headed queries cut to the user’s message, taken alone, cap 100 was unresolved: it passed with Opus and failed with Sonnet on interval width. The 100 queries come from one assistant’s traffic.

The server’s repository includes the benchmark: a proxy that records the requests Hindsight sends, and a replay script that reports latency, memory and thermal pressure at any cap. It can also build synthetic requests from token lengths alone, but on our data those ran 10–14% faster at p50 than the real ones they copied, so for numbers that matter, replay real requests.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @hindsight 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/going-back-to-hindsi…] indexed:0 read:11min 2026-11-19 · —