Short answer: combine lexical and embedding candidates, fuse ranks rather than raw scores, rerank only a small merged set, and refuse to produce a code-review finding unless the final evidence still points to an accessible, current policy passage. This costs more latency than a single search, but it protects the exact identifiers that semantic retrieval tends to blur while keeping the retrieval contract portable.
The incident lesson is blunt: retrieval success is not the same as review success. In a B2B SaaS review pipeline, a missed policy can let a risky change pass, while a duplicated delivery can post the same finding twice. I've been paged for both missed jobs and duplicate deliveries in cron and queue infrastructure. The useful invariant carries over: every stage needs an identity, a bounded retry policy, and evidence that survives replay. A high similarity score is not evidence.
This example assumes a chatbot that searches internal Node.js engineering docs, examines a proposed code change, and returns structured findings. The retrieval service happens to be written in Go because the boundary matters more than the client language: a Node.js worker can call the same HTTP contract, and the index or reranker can change without rewriting review logic.
Start with two independent candidate generators. Keyword search should carry exact strings such as AbortSignal
, package-lock.json
, rule IDs, error codes, and configuration keys. Embedding search should carry paraphrases, such as matching "stop work after the caller disconnects" with a policy about cancellation propagation. Neither branch gets the last word.
Each branch returns document identity, chunk identity, revision, access scope, rank, and a short excerpt. Do not merge raw relevance scores. A lexical score and a vector distance have different meanings, ranges, and calibration drift. Merge by rank with reciprocal rank fusion (RRF), deduplicate by stable chunk identity, then rerank the leading candidates against the proposed change and the review question.
A compact starting policy is 40 candidates from each branch, a fused pool of at most 60, and 12 inputs to the reranker. Those are configuration values, not universal benchmarks. Your mileage may vary: resolve them with an offline evaluation set containing exact-token cases, paraphrases, stale revisions, forbidden documents, and changes that should produce no finding. I am not sure which cutoff fits a given corpus until that evaluation includes the team's real policy language.
The order is deliberate:
That's the whole retrieval path.
The tempting design returns a list of strings. It works in a demo and fails in a runbook because an excerpt cannot answer basic operational questions: Which document revision produced this finding? Was the caller allowed to read it? Did two chunks come from the same policy paragraph? Can a retry reproduce the same result? The result type must carry those answers forward.
For code review, I treat a finding as a derived record with a deterministic identity. Hash the repository, commit SHA, policy revision, rule ID, and normalized code location. A delivery retry may regenerate the record, but the sink performs an upsert on that key. Do not use a newly generated request ID as the deduplication key; it identifies an attempt, not the work.
The same discipline catches missed findings. Record candidate counts for both branches, fused and reranked counts, rejected-evidence reasons, index revision, policy revision, and end-to-end duration. An empty lexical branch is different from an authorized query with no lexical match. Likewise, a generator that returns no finding is different from a retrieval stage that silently lost its candidates. Keep those states separate.
One long paragraph is warranted here because the common failure spans several boundaries. Suppose a change replaces a cancellation-aware Node.js call with one that no longer accepts an AbortSignal
. Semantic search finds a general document about resource cleanup; keyword search finds the exact API policy; fusion retains both; reranking prefers the exact rule in the context of the diff; the evidence gate verifies that the rule revision is current and readable; generation emits a structured finding with the rule ID and source chunk. If delivery is retried, the deterministic finding key prevents a duplicate comment. If the exact policy is absent or inaccessible, the system abstains rather than turning the general cleanup passage into a confident claim. The lesson from queue incidents applies cleanly — ambiguity must become an explicit state, not an invented success.
Portability lives in the types and test fixtures, not in a promise that every provider behaves alike. Keep provider-specific scores behind two ranked interfaces, preserve stable source metadata, and make fusion deterministic. The following core omits storage and model calls on purpose; those belong in adapters.
package retrieval
import (
"context"
"sort"
)
type Query struct {
TenantID string
RepoID string
Text string
Diff string
Revision string
}
type Candidate struct {
ChunkID string
Document string
Revision string
Excerpt string
Rank int
Fused float64
}
type Searcher interface {
Search(context.Context, Query, int) ([]Candidate, error)
}
type Reranker interface {
Rerank(context.Context, Query, []Candidate, int) ([]Candidate, error)
}
func Fuse(lists [][]Candidate, limit int) []Candidate {
const k = 60.0
byID := make(map[string]Candidate)
for _, list := range lists {
for i, candidate := range list {
candidate.Fused += 1.0 / (k + float64(i+1))
if current, ok := byID[candidate.ChunkID]; ok {
candidate.Fused += current.Fused
}
byID[candidate.ChunkID] = candidate
}
}
merged := make([]Candidate, 0, len(byID))
for _, candidate := range byID {
merged = append(merged, candidate)
}
sort.SliceStable(merged, func(i, j int) bool {
if merged[i].Fused == merged[j].Fused {
return merged[i].ChunkID < merged[j].ChunkID
}
return merged[i].Fused > merged[j].Fused
})
if len(merged) > limit {
merged = merged[:limit]
}
return merged
}
The tie-break on ChunkID
matters. Without it, equal fused scores can move between runs because map iteration order is not a ranking policy. Deterministic ordering makes golden tests useful and keeps provider migrations diagnosable.
The orchestration layer should run the two searches under one deadline, but it must classify outcomes. A timeout is not "zero matches." Depending on the review policy, one unavailable branch should either fail the review or produce a clearly degraded, non-blocking result; it should never quietly masquerade as complete retrieval. For a security gate, fail closed. For an advisory style check, degraded output may be acceptable if the response exposes that state to the caller and does not claim full coverage.
Build the evaluation set from review decisions, not from pretty chatbot answers. Each case needs a code diff, authorized corpus snapshot, expected source chunks, allowed rule IDs, and whether abstention is correct. Measure lexical recall, semantic recall, fused recall, reranker recall, unsupported-finding rate, duplicate-finding rate, and latency by stage. A single aggregate "accuracy" number hides the failure modes that wake someone up.
Include adversarial documents. Retrieved text is untrusted input, and the OWASP guidance for LLM applications treats prompt injection and sensitive information disclosure as material risks. A document sentence that instructs the model to ignore the review policy remains document content; it does not become a system instruction. Delimit evidence, constrain the output schema, and verify every emitted rule ID against the authorized candidate set after generation.
Access control must happen before retrieval and again before evidence leaves the service. Post-filtering a global nearest-neighbor result can leak existence through ranks or exhaust the candidate budget with forbidden chunks. Tenant, repository, and document scope belong in both search adapters. Log identities and decisions, but don't put raw diffs or sensitive excerpts into routine metrics.
Deployment is a dual-read problem. Build the new index revision, replay the fixed evaluation set, shadow representative queries without emitting findings, and compare evidence identities rather than prose. Then move traffic gradually while retaining the previous revision for rollback. Index revision and policy revision must appear in the result so an incident timeline can explain which knowledge was active.
No magic here.
The catch is extra machinery: two indexes, a fusion step, a reranker, more telemetry, and another latency budget. Hybrid retrieval is not suitable when the corpus is small enough for deterministic rules, when every policy is already keyed by an exact identifier, or when the review decision must be proven by a static analyzer. In those cases, stick with direct lookup or compiler-backed analysis. Search should not imitate a type checker.
Skip reranking when fusion already leaves only a few unambiguous, exact matches, or when the added model call cannot fit the review service's deadline. Use lexical-only retrieval for identifier-heavy policies with controlled vocabulary. Use semantic-only retrieval only when exact token misses are demonstrably harmless and the evaluation set supports that decision. The right fallback is workload-specific; making it explicit is more important than picking the fashionable branch.
Provider portability also has a limit. Interfaces make replacement testable, but embeddings from different models do not share a vector space, rerank scores are not interchangeable, and tokenization can change chunk boundaries. A migration therefore means rebuilding the index and rerunning evidence-level evaluations. The contract prevents application churn; it does not erase semantic change.
For a code-review assistant, the release criterion is simple: the new stack must preserve or improve source-chunk recall on the frozen evaluation set, must not increase unsupported findings, and must keep duplicate identities stable. If it cannot, don't ship the migration.