Building a multi-agent document-search copilot — Part 2: adaptive Hybrid, and a permission gate after the rank

This is Part 2 of two. Part 1 opened on a document-search copilot whose v1 produced muddy results — two retrieval lanes fused into one rerank, where metadata rows that carry no text corrupted the relevance scores. The fix was two reframes: collapse the router into one structured Bedrock call (with a deterministic fallback), and pick exactly one retrieval strategy per query — MetadataOnly, ContentOnly, or Hybrid — instead of fusing the lanes. We left off at the one strategy that refuses to be tidy: Hybrid, where the user wants a topic and filters at the same time.

Picking up where Part 1 stopped, we're now in the back half of the same pipeline — the direct_search → rerank → permission_filter → finalize_results

tail of the search graph. Two decisions left: how Hybrid

actually retrieves, and where permissions get enforced.

Hybrid

is the interesting one, because "topic + filters" is genuinely a tradeoff and not a single right answer. There was a real debate about it — the kind that goes a few rounds on a whiteboard: run the filter and the rank in parallel (fast, but you have to reconcile two sets), or in sequence (precise, but slower)? We resolved it by refusing to pick — and routing on selectivity instead.

The trick is one cheap question asked once: how many documents match the filters? direct_search

peeks the filtered-universe size from the Documents service a single time, with a threshold (default 1000

), and branches:

elif strategy == "Hybrid":
    threshold = cfg.get("filter_first_threshold", 1000)
    id_page = cfg.get("filter_id_page_size", 100)
    scope_ids, total_records = await _fetch_filtered_ids(metadata_filters, threshold, id_page)
    if scope_ids is not None:
        candidates = await _content_hits(scope_ids=scope_ids)   # filter-first
        filters_enforced_upstream = True
    else:
        candidates = await _content_hits()                       # rank-then-filter

terms

id-scope (over the version field on chunks, the parent field on attachment chunks). Every filter is enforced authoritatively at the source — nothing off-filter can possibly rank, and nothing on-filter is missed. It sets a flag so the later permission gate knows not to re-filter:

  filters_enforced_upstream = True

_row_passes_filters

— a faithful mirror of the Documents service's operation semantics across every searchable column (string / number / date / boolean / user_list, with contains

/ equal

/ greaterThan

/ the rest):

  def _row_passes_filters(row: dict, filters: list[dict]) -> bool:
      """Faithful local mirror of the Documents service filter semantics ...
      any hit whose row fails a hard filter is dropped, fail-closed."""
      return all(_row_passes_one(row, f) for f in filters)

The thing that makes this safe rather than a sloppy approximation: when the filtered set is huge, the filter is nearly a no-op anyway (it matches most of the library), so the recall ceiling from ranking-first-then-dropping is harmless. The case where dropping would lose good matches — a selective filter — is exactly the case that takes the filter-first path, where nothing is dropped. The branch picks the strategy where its own weakness doesn't apply. I'll be honest: it took me a beat to trust this — it feels like a cheat until you sit with it — but the two failure modes genuinely cancel out.

One peek, no extra round trips, and the precision/latency tradeoff is made explicit and data-driven instead of being a coin flip baked into the architecture.

💡 The pattern, generalized: when you can't decide between two retrieval strategies, measure the one property that distinguishes them (here, filter selectivity) with a single cheap probe, and route on it. Each branch is chosen precisely where its failure mode is benign.

The content path is ranked by Cohere (over the top rerank_candidates

, default 30

), with the OpenSearch fused score / RRF as the fallback when the reranker is unreachable:

results = await bedrock_client.rerank(query=query_text, documents=documents, model_id=model_id)

Metadata results skip the reranker entirely — there's nothing to score — and keep the Documents service order:

if all(c.get("match_kind") == "metadata" for c in candidates):
    return {"reranked": candidates, "cached_top_30": candidates}

Then comes the part that makes security people lean forward in their chair: the per-document view-permission gate runs after rerank, not before retrieval. Stay with me — it's safe, and the reason it's safe is the whole point.

permission_filter

takes the reranked content hits, confirms each one is viewable via a single bulk export_by_ids

call against the system of record, and drops the ones that aren't — fail-closed, so if the service is unreachable, every content hit is dropped rather than leaked:

rows = await documents_service.export_by_ids(content_ids, jwt)
for row in rows:
    dvid = row.get("document_version_id")
    if dvid:
        viewable.add(dvid)

Why is gating after rerank safe? Because there are two different boundaries doing two different jobs:

member_id

, and that id comes from the JWT — never from the request body, never from the index. A user physically cannot retrieve another tenant's vectors. That boundary is upstream and absolute.Rerank only orders the candidates. It exposes nothing — every candidate it sees already passed the tenant wall, and the set it's ordering is bounded (the reranked top set), so the view gate is one bounded bulk call, not a fan-out. The metadata lane skips the gate entirely because its rows came back already view-filtered by the service's own SQL — re-checking trusted rows would be wasted work. Identity from the token, tenant at retrieval, view-visibility after, fail-closed throughout.

Here's a small but real consequence of "one strategy per query" that landed squarely on my side of the stack — and nearly slipped past me. Content hits carry a relevance score. Metadata hits don't — a satisfied filter is not a relevance measurement, and showing the OpenSearch fused score (which for an exact field match is ~0) would read as "barely relevant" for what is actually a perfect match. So a metadata hit emits a ** null** score in the SSE payload, and the UI has to render that as "matched your filter," not "0% relevant":

def _shape_source_for_emit(s: dict) -> dict:
    score = None if s.get("score_signal") == "metadata" else (
        s.get("aggregate_score") or s.get("score") or 0.0
    )
    ...

It's a tiny detail, but it's the kind of thing that breaks if the frontend assumes "every result has a number." A null

score is a first-class state — "this matched a filter, there's no relevance to show" — not a missing value to default to zero. Defaulting it to 0.0

would have quietly buried every exact metadata match at the bottom of the list, sorted below loosely-relevant content — a perfect match dead last, and no error anywhere to tell you why. The contract had to carry the absence on purpose.

The throughline across all four decisions — the two in Part 1 and the two here — is the same instinct: make the system commit to one clear shape per turn, and make the tradeoffs explicit instead of averaging them away. One router call instead of three. One retrieval strategy instead of a fused blend. One selectivity probe instead of a parallel-vs-sequential guess. One bounded permission check after a bounded rerank. The muddy-results version tried to do everything at once and let the reranker sort it out. It couldn't. Picking, on purpose, is what made the rank mean something again.

👋 Thanks for sticking through both parts. If you've fought the same muddy-rank fight, I'd genuinely love to hear how you cut yours — the failure modes are weirdly universal.

Missed the setup? Part 1 covers the muddy-results problem, the single-call router, and the reframe to one strategy per query.

Thanks for reading all the way through 🙌 If you're building agents and fighting the same "is this finding even real?" problem, I'd genuinely like to compare notes — come say hi on LinkedIn.