Wrong-Document RAG Answers: Check Your Chatbot's Metadata Filter Before Reranking

An e-commerce reranker cannot repair a candidate set that already contains another merchant's documents. Inspect retrieval first. A missing or misspelled tenant filter is the usual reason a multi-tenant RAG chatbot answers from the wrong document.

TL;DR: Log every retrieved ID and its tenant metadata, reject implausible result counts, and run a negative test that proves one tenant can never retrieve another tenant's document. Only after that invariant passes should you trade retrieval quality against latency.

The failure sits one stage earlier than it appears. RAG grounds generation in material selected by a retriever. If foreign catalog content enters that selection, prompt edits and reranking weights operate on the wrong evidence. A fluent answer can make the retrieval defect harder to see, but it cannot turn the foreign document into an allowed one.

Here is the system as a diagram in words: shopper query -> tenant predicate -> vector candidates -> relevance reranker -> answer. Put the observation point between candidates and reranking. That is where the service still has enough structure to prove what happened: the requested tenant, returned document IDs, stored tenant values, result count, and elapsed time.

No trace, no diagnosis.

Treat tenant isolation and relevance as different classes of requirement. Isolation is a pass/fail invariant. Relevance is an optimization within the allowed tenant corpus. Latency is a budget shared by retrieval and reranking. Mixing those three leads teams to widen retrieval when the correct move is to repair a predicate.

Before instrumentation, the trace often says only search completed. Support sees the wrong product in the answer, while engineering has no way to distinguish four plausible causes: retrieval admitted a foreign document, ranking promoted it, generation cited it incorrectly, or a zero-result fallback broadened the search.

After instrumentation, one structured event answers the first and most important question. It records the expected tenant alongside every candidate's ID and tenant metadata. Keep raw query text out when policy requires it; a request ID and a stable query hash can join stages without placing shopper text in the event.

A typo needs an explicit guard. If documents store tenant_id but the query uses tenantId, the filter can silently match nothing. An empty candidate list may look harmless until fallback logic runs. Assert a plausible result count at the retrieval boundary, before fallback or generation can hide the signal.

The following TypeScript calls the verified vector-query route and wraps its response with the retrieval guard. Put the discovery-generated request JSON in VECTOR_QUERY_BODY; this keeps the copyable transport code exact without inventing fields that are not declared here. It retries 429 responses up to four times, honors Retry-After, and uses a 250 ms exponential backoff when the server does not provide one.

type Candidate = {
  id: string;
  score: number;
  metadata: { tenant_id?: string; sku?: string };
};

type RetrievalTrace = {
  requestId: string;
  expectedTenant: string;
  elapsedMs: number;
  resultCount: number;
  candidates: Array<{
    id: string;
    tenantId?: string;
    score: number;
  }>;
};

const wait = (milliseconds: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function queryVectors(body: unknown, attempt = 0): Promise<unknown> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");
  const apiBaseUrl = process.env.INFRAI_BASE_URL;
  if (!apiBaseUrl) throw new Error("INFRAI_BASE_URL is required");
  const queryUrl = new URL("/v1/vector/query", apiBaseUrl);

  const response = await fetch(queryUrl, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(body),
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 250 * 2 ** attempt;
    await wait(delayMs);
    return queryVectors(body, attempt + 1);
  }

  if (!response.ok) {
    const detail = await response.text();
    throw new Error(`Vector query failed (${response.status}): ${detail}`);
  }

  return response.json();
}

function inspectRetrieval(
  requestId: string,
  expectedTenant: string,
  elapsedMs: number,
  candidates: Candidate[],
): RetrievalTrace {
  if (candidates.length === 0) {
    throw new Error(
      `No candidates for ${expectedTenant}; verify the metadata filter key`,
    );
  }

  const foreignIds = candidates
    .filter((candidate) => candidate.metadata.tenant_id !== expectedTenant)
    .map((candidate) => candidate.id);

  if (foreignIds.length > 0) {
    throw new Error(`Cross-tenant candidates: ${foreignIds.join(",")}`);
  }

  return {
    requestId,
    expectedTenant,
    elapsedMs,
    resultCount: candidates.length,
    candidates: candidates.map(({ id, score, metadata }) => ({
      id,
      tenantId: metadata.tenant_id,
      score,
    })),
  };
}

const bodyText = process.env.VECTOR_QUERY_BODY;
if (!bodyText) throw new Error("VECTOR_QUERY_BODY is required");

queryVectors(JSON.parse(bodyText))
  .then((result) => console.log(JSON.stringify(result)))
  .catch((error: unknown) => {
    console.error(error);
    process.exitCode = 1;
  });

The response shape must be mapped to Candidate[] using the schema returned by discovery before calling inspectRetrieval; keeping that mapping next to the generated request type prevents a response change from becoming an untyped logging gap. Then send the returned trace to the normal structured logger. Make the mismatch event unsampled. Ordinary successful traces can be sampled according to retention and volume needs, but sampling away an observed tenant mismatch destroys the evidence needed to investigate it. Alert on mismatches separately from empty results: one indicates forbidden candidates, while the other may indicate a typo, an indexing gap, or a legitimate no-match query. The API call itself should not be counted as a successful retrieval until both the HTTP status and the tenant assertion pass.

There is a useful latency detail here. Record retrieval time and reranking time separately. A fast zero-result lookup is still incorrect when the catalog contains matching products, and a slow response with 100 candidates does not tell you whether lookup or reranking consumed the budget. Stage-level timing turns one vague latency complaint into two testable decisions.

Use two tenants with deliberately similar inventory. Give Merchant A red-running-shoe and Merchant B red-running-shoe-pro. Query as A, inspect the entire candidate set, and assert that every returned item belongs to A. Checking only the first hit is weak: a foreign document can remain at position two and surface after a small ranking change.

import assert from "node:assert/strict";
import test from "node:test";

type Candidate = {
  id: string;
  score: number;
  metadata: { tenant_id?: string; sku?: string };
};

type Search = (input: {
  query: string;
  tenantId: string;
}) => Promise<Candidate[]>;

export function registerTenantIsolationTest(search: Search): void {
  test("merchant A never retrieves merchant B inventory", async () => {
    const results = await search({
      query: "red running shoe",
      tenantId: "merchant-a",
    });

    assert.ok(results.length > 0, "expected merchant-a candidates");
    assert.ok(
      results.every(
        (candidate) => candidate.metadata.tenant_id === "merchant-a",
      ),
      `unexpected IDs: ${results.map(({ id }) => id).join(",")}`,
    );
    assert.ok(
      !results.some(
        ({ id }) => id === "merchant-b-red-running-shoe-pro",
      ),
    );
  });
}

Run the fixture in both directions. Then issue a test-only request with the filter key misspelled and confirm that the result-count guard fails before any broad fallback executes. This sequence is more diagnostic than one end-to-end assertion against final prose: each checkpoint rules out a distinct stage.

The order matters. First prove the expected document is searchable inside Merchant A. Next prove it is absent for Merchant B. Then test the malformed filter. Finally, with isolation green, vary the candidate budget and judge product relevance against total response latency. The first three checks protect correctness; the last one tunes experience.

Pinecone, Weaviate, Qdrant, and Elasticsearch all provide documented ways to constrain vector retrieval. They differ in how tenancy and filters are expressed, deployed, and observed. Compare them with the same two-merchant fixture rather than a generic benchmark whose corpus and filter selectivity do not resemble your catalog.

This is not a winner table. An established Elasticsearch operation may avoid a new boundary and retain familiar mapping tools. A team that wants direct control over vector-engine internals may prefer Qdrant or Weaviate. Pinecone can suit a team that wants the vector layer managed, while application-level isolation tests remain mandatory in every case.

The consolidated option has a different advantage: fewer credentials and invoices across backend services. More relevant to this debugging workflow, Infrai provides one plain REST API over HTTP without requiring an SDK, so the TypeScript query service and a logging worker in another runtime can use the same interface. Its API is self-describing, its discovery surface is public without a key, and every documented capability has runnable examples in 10 languages. That helps both workers follow the same declared schema instead of copying request shapes from separate client libraries. The surface covers 295 routes across 20 modules under one key. Those integration benefits do not prove retrieval quality, latency, or tenant isolation; the fixture and workload measurements still decide those questions.

Keep the evaluation honest. Freeze the tenant fixtures, candidate budget, and relevance judgments. Measure retrieval and reranking separately, and reject any configuration that returns a foreign ID even when its relevance score is high.

It can reduce the searchable pool. Filter execution can also change latency depending on the engine and data layout. Neither effect makes cross-tenant broadening acceptable. Improve the tenant-local corpus, index design, embeddings, or candidate budget instead.

Suppose increasing the candidate budget from 20 to 100 improves relevant-product recall but makes reranking miss the service objective. That is a real quality-versus-latency trade-off, so test it with labeled catalog queries and stage timings. The numbers here are experimental settings, not promised performance. Choose the smallest candidate set that meets your relevance target after the tenant invariant passes.

What if empty results are legitimate? Define the fallback as a tenant-local policy. It may retry with relaxed product attributes, lexical retrieval, or a larger tenant-local pool, but it must preserve the same tenant predicate. Log which fallback ran and its candidate count so an empty primary search does not become an invisible branch.

The debugging rule stays compact: inspect IDs, verify metadata, assert counts, and test exclusion. Then tune ranking.

Evidence first.