RAG Permissions at Retrieval Time: Why \\\\\\\"Don't Show It\\\\\\\" Fails as Agent Access Control

Tutorial RAG systems embed a folder and return the top five chunks. Production RAG systems embed a company with HR files, board decks, customer contracts, and incident reports where half the pages are restricted to specific teams. The moment multiple users with different permissions query the same index, you have an authorization problem that prompt instructions cannot solve.

The most common mistake is trying to enforce access control in the prompt:

You may only use documents the user is permitted to see. 
Do not reveal confidential information.

By the time the model reads that instruction, the confidential chunk is already in its context window. You are asking a probabilistic text generator to un-know something you just handed it. Sometimes it complies. Sometimes a well-phrased follow-up or a prompt injection inside another document extracts it anyway.

A pattern that looks reasonable:

chunks = vector_store.similarity_search(query, k=20)
allowed = [c for c in chunks if user_can_read(user, c.metadata["doc_id"])]
context = allowed[:5]

This has two problems. First, if the filter has a bug or a missing branch, the fallback is "everything." Second, top-k is computed over documents the user cannot see. A restricted folder full of near matches can push every allowed result out of the window, leaving the user with an empty or degraded answer.

Hybrid search systems merge vector similarity results with keyword BM25 results. If the vector store filters by ACL but the keyword index does not, the merge step can leak document titles, folder names, or snippet previews from forbidden documents. The LLM never sees the full text, but it sees enough metadata to infer structure or content.

Many RAG pipelines retrieve 50 chunks, then use a cross-encoder or LLM to re-rank them down to 5. If the re-ranker runs after the ACL filter, you are fine. If it runs before, the re-ranker sees forbidden documents and may promote them based on relevance. The final filter then removes them, but the ranking signal has already been contaminated.

Production systems enforce permissions during the similarity search, not after. This requires injecting user context into the query itself.

Most vector databases support metadata filters that run alongside the similarity search:

chunks = vector_store.similarity_search(
    query,
    k=5,
    filter={"$or": [
        {"owner_id": user.id},
        {"shared_with": {"$contains": user.id}},
        {"visibility": "public"}
    ]}
)

This works when ACLs are simple (owner, shared list, public flag). It breaks down when permissions are computed dynamically (group membership, role hierarchies, time-based grants). The vector store has no way to evaluate complex authorization logic.

For multi-tenant SaaS, some teams maintain one vector index per customer. Retrieval is scoped to the tenant's index by default. This eliminates cross-tenant leakage but creates operational overhead. You need to manage index lifecycle (create, update, delete), handle schema migrations across thousands of indexes, and route queries to the correct index based on authentication context.

Embed the document along with its access control metadata. Store (embedding, doc_id, acl_hash) in the vector store. At query time, compute the user's effective ACL hash and filter results to matching hashes. This works when ACLs are stable and coarse-grained (department-level, project-level). It fails when permissions change frequently or are user-specific, because you would need to re-embed and re-index every affected document.

Place an authorization service between the RAG pipeline and the vector store. The proxy intercepts every retrieval request, fetches the user's effective permissions from an external policy engine (OPA, Zanzibar, custom RBAC), and rewrites the vector query to include the appropriate filters. This decouples permission logic from the vector store and the LLM orchestration layer, but it adds latency and a new failure mode.

When permissions change, the retrieval layer needs to know immediately. If a user is removed from a project, their next query must not return documents from that project. This requires either:

Most production systems use a hybrid: cache permissions for 30 seconds, but subscribe to a permission-change event stream to invalidate specific user caches immediately.

Standard RAG observability tracks query latency, chunk relevance, and LLM token usage. It does not track:

You need custom metrics:

metrics.record("rag.chunks.retrieved", len(all_chunks))
metrics.record("rag.chunks.allowed", len(allowed_chunks))
metrics.record("rag.chunks.filtered", len(all_chunks) - len(allowed_chunks))

If chunks.filtered is consistently high for a user or query pattern, either the ACLs are too restrictive or the index contains too many documents the user cannot see. Both are actionable signals.

Silent degradation. The user asks a question. The answer is technically correct but incomplete because the most relevant chunks were filtered out. The LLM has no way to know it is missing context, so it does not tell the user. The user assumes the answer is complete.

ACL drift. Permissions are updated in the source system (Google Drive, Notion, Confluence) but not propagated to the vector store metadata. The RAG system continues to enforce stale permissions, either over-restricting (user cannot see documents they should) or under-restricting (user sees documents they should not).

Metadata injection. An attacker uploads a document with crafted metadata ({"shared_with": ["*"]}). If the ingestion pipeline does not validate metadata against the source system's actual ACLs, the document becomes visible to everyone.

Use retrieval-time filtering (metadata filters or separate indexes) when:

Avoid post-hoc filtering (prompt instructions or post-retrieval ACL checks) when:

Use an authorization proxy when:

Do not rely on the LLM to enforce access control. By the time the model sees the instruction, the access decision has already been made.