Vector Search Fundamentals for RAG Design: ANN (HNSW), Distance Metrics, Metadata Filtering, and BM25

When designing a RAG (Retrieval-Augmented Generation) system, understanding what's happening inside vector search lets you tune the trade-offs between accuracy, speed, and cost yourself.

This article covers the fundamentals of vector search:

and then summarizes practical rules of thumb for choosing between them when designing a RAG system.

In vector search, you look for the vectors in a corpus that are closest to a query embedding. Exact nearest neighbor search (kNN), which computes distances against every vector via brute force, gets linearly slower as the dataset grows.

ANN (Approximate Nearest Neighbor) search trades a small amount of accuracy (recall) for the ability to return "close enough" vectors quickly, even at large scale.

HNSW (Hierarchical Navigable Small World) is one of the most widely used ANN algorithms. It treats vectors as nodes in a graph, connecting nearby vectors with edges. Its defining feature is a hierarchical structure: upper layers are sparse and used for long jumps, while lower layers are dense and used for fine-grained search.

Search starts at the top layer and descends one layer at a time as it gets closer to the query vector. Think of it like flying between continents by plane, taking a train into a city, then walking the last stretch on foot. This reaches the neighborhood of the target vector using far fewer distance calculations than brute force.

Accuracy and speed are tuned mainly through two parameters.

Three common metrics measure how "close" two vectors are.

Text embeddings usually encode meaning in the vector's direction, with magnitude carrying little meaning, so cosine similarity — or dot product on normalized vectors — is the common choice. L2 distance shows up more often for embeddings such as image features, where magnitude itself is meaningful.

Vectors can carry metadata beyond the raw content itself — category, date, tags, access permissions, and so on. Metadata filtering means narrowing candidates by these conditions, either before or while running the vector search.

Pre-filtering narrows candidates to those matching the conditions first, then runs vector search within that subset. It's more accurate, but the narrowing step itself has a cost.

Post-filtering runs ANN search first to over-fetch neighbor candidates, then excludes ones that don't match the metadata conditions. It reuses the ANN graph structure as-is, so it's fast, but if the filter rejects a large fraction of candidates, you can end up with too few results.

Newer vector databases such as Pinecone, Weaviate, and Qdrant often implement "filter-aware ANN," which factors filter conditions into the graph traversal itself — getting the best of both approaches in many cases.

The "keyword search" side of hybrid search in RAG is, in most implementations, an algorithm called BM25 (Best Matching 25). It's a refinement of TF-IDF and is the default scoring function in many full-text search engines, including Elasticsearch. Its score is driven mainly by three factors.

Where vector search captures semantic closeness, BM25 captures lexical, word-level matches. BM25 tends to win on queries where exact matches matter — model numbers, proper nouns — while vector search tends to win on paraphrased or ambiguous queries. Combining the two in hybrid search lets each compensate for the other's weaknesses.

Here's how to combine everything above when actually designing a RAG system.

Rather than choosing a metric yourself, match whatever metric your embedding model assumes at training time. Major text embedding models from OpenAI, Cohere, and others are trained assuming cosine similarity (or dot product on normalized vectors), so the practical default is cosine similarity — or dot product on normalized embeddings if you want to shave off compute. L2 distance rarely shows up in text RAG.

In RAG, missing even one relevant neighbor means the LLM never sees information it needed. So prioritize recall over query speed: set ef_search

relatively high, and over-fetch more candidates than your final top-k (e.g., fetch 20-50 candidates if you'll ultimately use 5) before narrowing down.

For corpora up to roughly tens of thousands of vectors, sticking with exact kNN instead of ANN is a perfectly reasonable choice. It eliminates any recall loss from approximation, and brute-force search is often fast enough at that scale. ANN becomes genuinely necessary somewhere in the hundreds of thousands to millions of vectors. Requirements like access control and tenant isolation — where a document must never leak into results it shouldn't — always call for pre-filtering (or filter-aware ANN). With post-filtering, if only a few permitted documents happen to land in the ANN candidate set, you can end up with empty results, or worse, a bug that leaks unauthorized documents.

Filters meant purely to improve precision — date ranges, document type, and so on — are often fine with post-filtering (over-fetch candidates, then narrow), which has the advantage of a simpler implementation.

RAG retrieval layers tend to combine more than plain vector search alone.

Vector search isn't just about fetching neighbors quickly with ANN. Designing the metric choice, filter strategy, and downstream reranking together is what meaningfully moves the needle on a RAG system's answer quality.