Local decision model as memory search reranker

Behind many modern search systems is a two-step process. The first step is fast and rough: it fetches a pool of maybe 20 candidate passages. The second step reorders them to put the best ones at the top. That second step is called a reranker, and it is usually the most expensive part of the pipeline. The question we had was: can we make it cheaper without losing quality? We tested five methods on 800 questions from the public ConditionalQA benchmark. Every model in the test runs locally through Ollama; nothing calls a cloud API. The headline: a 9B typed-answer decision model (nimble) matched the shipped 27B general language model (qwen3.8) at recall@5 (both put a good answer in the top 5 about 94% of the time) and ran three times faster (about 1.5 seconds vs 4.4 seconds per question on the same hardware). Where the smaller model loses: at recall@1 (putting the single best answer first), 78% vs 81%. Why: a decision call returns a softmax that separates the winner from the field but does not separate second from third very well. We also hoped the probabilities could serve as an abstention gate (none of these answer, say I do not know). They cannot. The pool with the correct passage scored higher than the pool without it only 75% of the time; we needed over 85%. The probabilities are overconfident: a 97% claimed confidence meant 42% actually correct; 50 to 90% claimed meant under 15% correct. Follow-on on two newer local models (clef 27B multimodal and clef-flash 9.1B) found neither beats nimble on LoCoMo (0.750 vs 0.739 vs 0.711). Practical: if the top handful is what matters, swap in the smaller typed-answer model and save two-thirds of the reranker compute; if you need the single best result and act on it, keep the bigger general model; if you want an abstention signal, these probabilities are not it.