RAG Retrieval Optimization: Reduce Vector Search Before Ranking

KoutenDB, an open-source embedded document and vector database written in Nim, reduces RAG latency and memory use by applying pre-ranking locality boundaries such as tenant, product, or version before vector search. The project demonstrates that filtering metadata before similarity ranking can shrink the working set and improve efficiency, while cautioning that partitions must retain relevant documents.

Most RAG performance advice begins at the ranking stage: choose a faster embedding model, tune an ANN index, reduce the result count, add a reranker, or cache common queries. Those are useful techniques. But RAG retrieval optimization should start one step earlier: Why is this query considering these vectors at all? A request often already contains a reliable boundary: tenant, repository, product, language, document type, version, date range, permission scope, or the object currently open in the application. Applying that knowledge before vector ranking can reduce RAG latency, vector-search memory use, and unrelated context. This is the difference between searching a whole corpus for similar documents and searching the authorized, relevant part of that corpus. It is also the specific problem that a locality-aware RAG database such as KoutenDB https://github.com/puffball1567/koutendb explores. Vector search metadata filtering commonly restricts a query by fields such as: tenant id = "acme" AND product = "billing" AND language = "en" AND published = true AND version = "2026.2" Those constraints are necessary for both relevance and security. Their placement in the read path, however, changes the cost: | Strategy | What happens | |---|---| | Filter after broad retrieval | Rank widely, then discard ineligible candidates. | | Filter-aware vector index | Use indexed metadata while selecting vector candidates. | | Namespace or partition selection | Select an eligible subset, then search it. | | Application-local routing | Route directly to a known tenant, source, or related-data neighborhood. | The first approach can still enforce correct access rules, but it may score vectors that should never have been candidates. The latter two are valuable when the application already knows a stable boundary. For a multi-tenant support assistant, the tenant boundary is not merely a relevance hint. It is an authorization rule and a natural first search scope. For a code assistant, the active repository and branch often serve the same purpose. For a product assistant, version and language can rule out most of the corpus before similarity becomes useful. The working set is everything a request touches: candidate vectors, metadata,payload bytes, reranker inputs, and finally model context. Reducing it changes several costs at once: This is not an argument for creating a partition for every tag. A pre-ranking boundary should be stable, known at request time, meaningful to the application,and strong enough to exclude large amounts of unrelated data. Tenant, source,product, and document-version boundaries often qualify; free-form tags usually do not. There is also an essential quality check: a smaller scope is only an optimization if it retains the documents the user needs. A wrong partition can be fast because it is wrong. KoutenDB is an open-source, embedded-capable document and vector database written in Nim. It is not a general replacement for PostgreSQL, a mature ANN vector database, or a global secondary-index engine. Its vector path makes a focused trade-off: For example, a RAG application that already knows the tenant and product can place and query documentation in a corresponding ring: python import koutendb var db = koutendb.open dataDir = "data" let hits = db.retrieve @ 1.0'f32, 0.0'f32 , ring = "tenant/acme/product/billing", budget = 8 The ring is chosen by application logic, policy, or an import rule; KoutenDB does not claim to infer the right security or business boundary from embedding similarity. JSONL ingestion can derive rings from a field, for example with a tenant field and a tenant prefix. Filters and projections still narrow results, but only after the local ring has been selected. This boundary is important: KoutenDB is a good fit when an application can name a meaningful pre-ranking scope. It is not the right primary tool when every query must perform global, cross-corpus discovery. Returning fewer hits does not prove that a system did less search work. A useful RAG database should expose whether unrelated candidates were skipped before ranking. KoutenDB reports total vectors, scanned candidates, skipped vectors, rings touched, candidate reduction, payload bytes, and estimated tokens. Its included working-set benchmark uses 10,000 vectors across 100 rings. In one local run, global retrieval scanned 10,000 vectors per query, while routed retrieval scanned 100: a 99% reduction. Measured latency in that run was 1,954.9 microseconds for the global path and 31.4 microseconds for the routed path. An included RAG-style test kept recall at 1.000 for the correctly routed ring while reducing scanned candidates from 400 to 40 and estimated tokens from 615.2 to 231.6. It also demonstrates the failure case: selecting the wrong ring keeps the small scan but produces zero recall. These are local synthetic measurements, not a universal latency claim or a claim that KoutenDB beats every vector database. They demonstrate a narrower invariant: candidates in unrelated rings are skipped before exact vector scoring, rather than discarded after a broad search. This approach is particularly relevant for: It is less suitable when global discovery is the core job, or when the application cannot identify a trustworthy boundary before retrieval. In that case, a broad ANN-oriented vector database with filter-aware indexing is often the better primary retrieval layer. Before tuning a model or index, ask: The answer may lead to namespaces, metadata indexes, database partitions, or a locality-aware database. The common requirement is to make the pre-ranking boundary explicit and measurable. The best way to reduce RAG cost or RAG latency is not always a more complicated ranking algorithm. Often it is avoiding a ranking problem that the application already knows is irrelevant. Use metadata filtering for correctness. Use tenant, source, version, and task locality to reduce the eligible search space when those boundaries are trustworthy. Then measure scanned candidates, recall, latency, and context size together. That is the design space KoutenDB explores: a database for RAG where application-level locality becomes part of the retrieval path before exact vector ranking begins.