# Failure Recovery Patterns for Low-Latency Candidate Retrieval and Reranking

> Source: <https://dev.to/thatchercole8235/failure-recovery-patterns-for-low-latency-candidate-retrieval-and-reranking-9b0>
> Published: 2026-11-19 19:40:03+00:00

**Recommendation:** Bound vector and reranker stages with independent deadlines inside your latency budget and serve lexical hits when they time out; measure recall@50 and nDCG@10 to confirm the fallback doesn't hide a recall loss.
More in the playbook "Retrieval and RAG": <https://wpnews.pro/playbooks/rag-retrieval>

Short answer: keep candidate retrieval useful when a reranker, index, or embedding service fails by making lexical search the authoritative fallback, bounding every optional stage with a deadline, and logging which path actually served the request. The decision is retrieval quality versus latency, and recovery has to preserve that contract.

I learned to treat failure recovery as part of ranking design, not an exception handler added after launch. A recruiter searching for “senior iOS engineer, payments” can tolerate a less polished ordering for a few seconds; an empty result page is much harder to explain. The same pattern applies to an e-commerce catalog reranker: the system must return plausible items while an expensive model is unavailable.

The simple approach is one request that calls vector retrieval, metadata filters, a cross-encoder reranker, and an LLM explanation in sequence. It looks tidy on a diagram. It also lets the slowest dependency decide whether a recruiter sees anything.

My test harness uses a 250 ms end-to-end budget. Stage one runs a lexical query against the candidate index; stage two may add vector results; stage three reranks only the merged top 100. Explanations are asynchronous. If stage two misses its 70 ms budget, the merge continues with lexical hits. If stage three misses 110 ms, the original order remains valid.

Three words matter: return something useful.

Before copying this shape, measure recall@50, nDCG@10, p95 latency, timeout rate, and the percentage of requests served by each path. A faster fallback that hides a 20% recall loss is not a recovery win. Your mileage may vary because corpus size, language, and filter selectivity change those numbers.

Use independent budgets and an explicit result envelope. A timeout should become a typed event, not an unstructured log line. This TypeScript sketch keeps the policy in one place so a UI can show a result set without guessing why a stage was skipped.

```
type RetrievalPath = "lexical" | "hybrid" | "reranked";

type CandidateHit = {
  id: string;
  score: number;
  source: "lexical" | "vector" | "reranker";
};

type SearchResponse = {
  hits: CandidateHit[];
  path: RetrievalPath;
  timedOutStages: string[];
};

async function searchCandidates(query: string, deadlineMs = 250): Promise<SearchResponse> {
  const started = performance.now();
  const timedOutStages: string[] = [];
  const lexical = await lexicalSearch(query);

  const remaining = () => deadlineMs - (performance.now() - started);
  if (remaining() <= 70) return { hits: lexical, path: "lexical", timedOutStages };

  const vector = await withTimeout(vectorSearch(query), Math.min(70, remaining()));
  if (!vector.ok) timedOutStages.push("vector");
  const merged = mergeByCandidateId(lexical, vector.ok ? vector.value : []);

  if (remaining() <= 110) return { hits: merged, path: "hybrid", timedOutStages };
  const ranked = await withTimeout(rerank(query, merged.slice(0, 100)), Math.min(110, remaining()));
  if (!ranked.ok) {
    timedOutStages.push("reranker");
    return { hits: merged, path: "hybrid", timedOutStages };
  }
  return { hits: ranked.value, path: "reranked", timedOutStages };
}
```

The important detail is that lexical hits are valid on their own. They carry stable IDs and explainable fields such as title, skills, location, and availability. A vector or reranker adds recall and ordering quality; it does not own correctness. Never retry a timed-out stage inside the user request. Put retries on an asynchronous queue, with a request ID for correlation.

An HTTP 200 with zero hits can mean a bad filter, an empty shard, an expired index alias, or a genuinely narrow query. Track those cases separately. Return a reason code such as `filter_conflict`, `dependency_timeout`, or `no_match`; do not make the client infer it from an empty array.

Stale embeddings are quieter. A new skill taxonomy can make vector scores incomparable while lexical matching still appears healthy. Store the embedding model version beside each vector, sample score distributions, and quarantine a version whose drift crosses a pre-set threshold. For candidate data, also enforce authorization filters before blending scores; relevance never overrides access control.

Cancellation is another sharp edge. When a recruiter closes the page, propagate an `AbortSignal` to downstream calls. Otherwise abandoned reranks consume the same CPU and token budget needed for live searches. I once found a queue full of work for requests that no longer had a browser waiting; the fix was cancellation plus a per-tenant concurrency limit, not a larger cluster.

Elasticsearch, OpenSearch, and Vespa all expose different combinations of lexical, vector, and ranking primitives. The engineering question is not which name wins; it is whether the system supports your required timeout controls, filtering semantics, index versioning, and explainability. A self-hosted stack may give deeper tuning at the cost of operating more moving parts. A managed service can reduce maintenance while constraining query execution or deployment choices.

Keep the interface narrow: `search(query, filters, limit, deadline)` returns hits plus path metadata. Behind it, adapters can call BM25, approximate nearest-neighbor search, or a learned ranker. This keeps an index migration from changing recruiter-facing behavior and makes shadow traffic possible.

The catch is that a lexical fallback is not suitable when meaning depends on implicit context, such as an uncommon abbreviation or multilingual skill synonym. In that case, invest in synonym maps, curated aliases, and offline evaluation before raising the online deadline. Stick with a lexical-only path when traffic is small, filters dominate, or an extra 100 ms would violate the product promise.

Ship the fallback.

Build a failure matrix before production: vector timeout, reranker timeout, partial shard response, stale model version, malformed filter, and cancellation. For each row, specify the returned path, user-visible status, metric, and alert threshold. Replay anonymized queries against a fixed index, then inject faults during canary traffic.

A useful dashboard has latency and quality side by side. Watch p50 and p95 by path, recall@50 on judged queries, empty-result rate, and the share of traffic using a fallback. Sample explanations only after the result set is stable; generated prose must never block retrieval.

During a drill, I force the vector call to sleep for 90 ms, then send a query with a location filter and a skill synonym. The expected trace has a lexical result set, a `dependency_timeout` event, and a hybrid response before the 250 ms deadline. I also replay the same query with a stale embedding version. Its vector scores are excluded from the merge, while the lexical path remains visible in the response metadata. That trace is more useful than a green uptime chart: it proves the user received candidates, the filter was enforced, and the team can tell which dependency caused the downgrade. Keep the drill fixture in CI so a refactor cannot silently turn a bounded fallback into a blocking call.

| Approach | Integration shape | Good fit | Main limitation | 
|---|---|---|---|
| Lexical first | Standard query API | Exact skills, filters, strict latency | Misses implicit synonyms | 
| Hybrid retrieval | Lexical plus vector adapter | Better recall on varied phrasing | More index and model state to operate | 
| Learned reranker | Batch top-k scoring | Fine ordering for judged queries | Highest tail latency and compute | 

I am not sure there is one universal timeout split. Start with the product deadline, subtract network and serialization overhead, and allocate the remainder from measured stage distributions. Revisit it when the corpus, filters, or model changes.
