Building a multi-agent document-search copilot — Part 2: adaptive Hybrid, and a permission gate after the rank

A developer building a multi-agent document-search copilot implemented an adaptive hybrid retrieval strategy that branches based on filter selectivity. When filters match few documents, the system scopes semantic search to the filtered ID set for authoritative enforcement; when filters match many documents, it ranks first and filters locally, avoiding recall loss. The approach resolves the tradeoff between parallel and sequential hybrid retrieval by routing on selectivity, ensuring no off-filter documents rank while maintaining performance.

This is Part 2 of two. Part 1 opened on a document-search copilot whose v1 produced muddy results — two retrieval lanes fused into one rerank, where metadata rows that carry no text corrupted the relevance scores. The fix was two reframes: collapse the router into one structured Bedrock call with a deterministic fallback , and pick exactly one retrieval strategy per query — MetadataOnly, ContentOnly, or Hybrid — instead of fusing the lanes. We left off at the one strategy that refuses to be tidy: Hybrid, where the user wants a topic and filters at the same time. Picking up where Part 1 stopped, we're now in the back half of the same pipeline — the direct search → rerank → permission filter → finalize results tail of the search graph. Two decisions left: how Hybrid actually retrieves, and where permissions get enforced. Hybrid is the interesting one, because "topic + filters" is genuinely a tradeoff and not a single right answer. There was a real debate about it — the kind that goes a few rounds on a whiteboard: run the filter and the rank in parallel fast, but you have to reconcile two sets , or in sequence precise, but slower ? We resolved it by refusing to pick — and routing on selectivity instead. The trick is one cheap question asked once: how many documents match the filters? direct search peeks the filtered-universe size from the Documents service a single time, with a threshold default 1000 , and branches: elif strategy == "Hybrid": threshold = cfg.get "filter first threshold", 1000 id page = cfg.get "filter id page size", 100 scope ids, total records = await fetch filtered ids metadata filters, threshold, id page if scope ids is not None: candidates = await content hits scope ids=scope ids filter-first filters enforced upstream = True else: candidates = await content hits rank-then-filter terms id-scope over the version field on chunks, the parent field on attachment chunks . Every filter is enforced authoritatively at the source — nothing off-filter can possibly rank, and nothing on-filter is missed. It sets a flag so the later permission gate knows not to re-filter: filter-first scoped the topic ranking to the service-filtered id set, so the filters are already enforced authoritatively; permission filter must NOT re-enforce them. filters enforced upstream = True row passes filters — a faithful mirror of the Documents service's operation semantics across every searchable column string / number / date / boolean / user list, with contains / equal / greaterThan / the rest : php def row passes filters row: dict, filters: list dict - bool: """Faithful local mirror of the Documents service filter semantics ... any hit whose row fails a hard filter is dropped, fail-closed.""" return all row passes one row, f for f in filters The thing that makes this safe rather than a sloppy approximation: when the filtered set is huge, the filter is nearly a no-op anyway it matches most of the library , so the recall ceiling from ranking-first-then-dropping is harmless. The case where dropping would lose good matches — a selective filter — is exactly the case that takes the filter-first path, where nothing is dropped. The branch picks the strategy where its own weakness doesn't apply. I'll be honest: it took me a beat to trust this — it feels like a cheat until you sit with it — but the two failure modes genuinely cancel out. | filter-first selective | rank-then-filter broad | | |---|---|---| | Filtered universe | ≤ threshold default 1000 | threshold | | Semantic query | scoped to the filtered id set | unscoped | | Filters enforced | at the Documents service authoritative | locally, in permission filter | | Recall risk | none — nothing off-filter can rank | harmless — filter matches most docs | | Round trips for the peek | one read the id set | one just the count, then bail | One peek, no extra round trips, and the precision/latency tradeoff is made explicit and data-driven instead of being a coin flip baked into the architecture. 💡 The pattern, generalized: when you can't decide between two retrieval strategies, measure the one property that distinguishes them here, filter selectivity with a single cheap probe, and route on it. Each branch is chosen precisely where its failure mode is benign. The content path is ranked by Cohere over the top rerank candidates , default 30 , with the OpenSearch fused score / RRF as the fallback when the reranker is unreachable: results = await bedrock client.rerank query=query text, documents=documents, model id=model id ... on any failure or empty result - fallback sorts by the OpenSearch raw score Metadata results skip the reranker entirely — there's nothing to score — and keep the Documents service order: if all c.get "match kind" == "metadata" for c in candidates : return {"reranked": candidates, "cached top 30": candidates} Then comes the part that makes security people lean forward in their chair: the per-document view-permission gate runs after rerank , not before retrieval. Stay with me — it's safe, and the reason it's safe is the whole point. permission filter takes the reranked content hits, confirms each one is viewable via a single bulk export by ids call against the system of record, and drops the ones that aren't — fail-closed, so if the service is unreachable, every content hit is dropped rather than leaked: rows = await documents service.export by ids content ids, jwt for row in rows: dvid = row.get "document version id" if dvid: viewable.add dvid ... survivors = metadata rows trusted + content hits whose id is in viewable Why is gating after rerank safe? Because there are two different boundaries doing two different jobs: member id , and that id comes from the JWT — never from the request body, never from the index. A user physically cannot retrieve another tenant's vectors. That boundary is upstream and absolute.Rerank only orders the candidates. It exposes nothing — every candidate it sees already passed the tenant wall, and the set it's ordering is bounded the reranked top set , so the view gate is one bounded bulk call, not a fan-out. The metadata lane skips the gate entirely because its rows came back already view-filtered by the service's own SQL — re-checking trusted rows would be wasted work. Identity from the token, tenant at retrieval, view-visibility after, fail-closed throughout. Here's a small but real consequence of "one strategy per query" that landed squarely on my side of the stack — and nearly slipped past me. Content hits carry a relevance score. Metadata hits don't — a satisfied filter is not a relevance measurement, and showing the OpenSearch fused score which for an exact field match is ~0 would read as "barely relevant" for what is actually a perfect match. So a metadata hit emits a null score in the SSE payload, and the UI has to render that as "matched your filter," not "0% relevant": php def shape source for emit s: dict - dict: A metadata match is a satisfied filter, not a relevance measure, so its score stays null the card shows just the match reason ; content matches keep their fused/rerank score. score = None if s.get "score signal" == "metadata" else s.get "aggregate score" or s.get "score" or 0.0 ... It's a tiny detail, but it's the kind of thing that breaks if the frontend assumes "every result has a number." A null score is a first-class state — "this matched a filter, there's no relevance to show" — not a missing value to default to zero. Defaulting it to 0.0 would have quietly buried every exact metadata match at the bottom of the list, sorted below loosely-relevant content — a perfect match dead last, and no error anywhere to tell you why. The contract had to carry the absence on purpose. The throughline across all four decisions — the two in Part 1 and the two here — is the same instinct: make the system commit to one clear shape per turn, and make the tradeoffs explicit instead of averaging them away. One router call instead of three. One retrieval strategy instead of a fused blend. One selectivity probe instead of a parallel-vs-sequential guess. One bounded permission check after a bounded rerank. The muddy-results version tried to do everything at once and let the reranker sort it out. It couldn't. Picking, on purpose, is what made the rank mean something again. 👋 Thanks for sticking through both parts. If you've fought the same muddy-rank fight, I'd genuinely love to hear how you cut yours — the failure modes are weirdly universal. Missed the setup? Part 1 covers the muddy-results problem, the single-call router, and the reframe to one strategy per query. Thanks for reading all the way through 🙌 If you're building agents and fighting the same "is this finding even real?" problem, I'd genuinely like to compare notes — come say hi on LinkedIn https://www.linkedin.com/in/rodrigo-diego-67867185/ .