# OpenSearch Optimizations for Production RAG > Source: > Published: 2026-07-14 16:31:01+00:00 *This is Part 1 of a series on optimizing OpenSearch for production RAG. Part 1 covers semantic retrieval, meaning vector search with Approximate and exact Nearest Neighbor methods. Part 2 will cover lexical retrieval, meaning text-based search and how it complements the semantic path.* Retrieval Augmented Generation (RAG) systems retrieve a set of relevant documents and pass them as context to a Large Language Model (LLM). The quality of the generated answer depends heavily on the quality of this retrieval step. If the right documents are not retrieved, no amount of prompting will recover the missing information. Hence, the retrieval layer is often the real bottleneck in a production RAG system, and it is where most of the tuning effort is spent. Most RAG systems use vector search. Documents are split into chunks, each chunk is converted into an embedding vector by an embedding model, and the vectors are stored in a vector index. At query time, the user question is converted into a query vector, and the system retrieves the nearest vectors to it. OpenSearch is a common choice for this index because it combines vector search with the lexical search, metadata filtering, and operational tooling that production systems need. There are two axes that matter when tuning vector retrieval, and they pull against each other: The rest of this article discusses how vector search works in OpenSearch, the problems one typically finds in production around recall and latency, and a set of optimizations that improve latency without regressing recall. The techniques are described in a general way so that they apply to any RAG system, not a specific codebase. This article assumes familiarity with OpenSearch basics such as indices, shards, and mappings; for a refresher on those fundamentals, see the links in the [References](https://pub.towardsai.net/feed#35cd) section at the bottom of the article. Finding the true nearest vectors to a query out of hundreds of millions of documents by comparing against every one of them is too slow for a user-facing system. Approximate Nearest Neighbor (ANN) search solves this by building a data structure that can be navigated toward the answer without a full scan. The most widely used ANN algorithm is Hierarchical Navigable Small World (HNSW). HNSW builds a layered graph. Each vector is a node connected to its nearest neighbors. The bottom layer contains every node with dense local links. The upper layers are sparse, containing progressively fewer nodes with long-range links that act as shortcuts across the vector space. A search enters at the top layer, greedily hops toward the query vector using the long-range links to cover large distances quickly, and then descends layer by layer, refining the result until it reaches the dense bottom layer. Because the upper layers zoom the search into the right neighborhood in a few hops, the total work is roughly logarithmic in the number of vectors rather than linear. Before listing the parameters, it helps to separate two moments in the life of the index, because each parameter belongs to one of them. *Index time* is when documents are added and the graph is built. Work done here is paid once per document, offline, and does not affect how fast a user’s query runs. It determines the *quality* of the graph, which in turn sets the ceiling on the recall a query can reach. Building a better graph costs more time and memory up front. *Query time* is when a user’s query searches the graph. Work done here is paid on every query and sits directly in the user-facing response path. Parameters that act at query time are the ones that move latency for the live system. The distinction matters because a parameter that only costs more at index time is “free” from the user’s latency perspective, since you pay it once during ingestion, whereas a parameter that costs more at query time is paid on every request. HNSW has parameters of both kinds: A useful way to hold these together: ef_construction and m shape how good the graph *can* be (set once, at index time), while ef_search decides how hard each query *works* to realize that potential (paid every query). Raising ef_construction costs you slower ingestion; raising ef_search costs you slower queries; raising m costs you both, plus memory. A basic ANN query asks for the k nearest vectors to the query embedding: ``` GET /documents/_search{ "size": 10, "query": { "knn": { "embedding": { "vector": [0.12, -0.03, 0.88, ...], "k": 10 } } }} ``` The graph parameters are configured on the index. m and ef_construction are set on the vector field at creation time and are fixed for the life of the index, so changing them means reindexing. ef_search is a query-time setting: it has an index-level default (shown here) but can also be overridden per query, so it can be tuned without a rebuild. ``` PUT /documents{ "settings": { "index": { "knn": true, "knn.algo_param.ef_search": 512 } }, "mappings": { "properties": { "embedding": { "type": "knn_vector", "dimension": 1024, "method": { "name": "hnsw", "engine": "faiss", "space_type": "innerproduct", "parameters": { "ef_construction": 512, "m": 16 } } } } }} ``` Because ef_search is a query-time knob, it can be raised for a specific query that needs higher recall without touching the index: ``` GET /documents/_search{ "query": { "knn": { "embedding": { "vector": [...], "k": 10, "method_parameters": { "ef_search": 1024 } } } }} ``` The important point is that HNSW is *approximate*. The greedy walk can miss the true nearest neighbor if the path to it is not reachable from the region that was explored. This is the source of recall loss in ANN. Raising ef_search recovers more of the true neighbors, at the cost of latency. As ef_search increases toward the size of the index, ANN recall approaches exact recall. So ANN is not a single fixed thing. It is a knob, and where one sets that knob is a recall and latency decision. Many production RAG systems do not search the whole index. They restrict the search with metadata filters. Common ones include a tenant identifier in a multi-tenant system, a recency window, a document category or source, an access control list that limits which documents a user is allowed to see, a location or geographic bound for place-aware corpora, a language or locale, and a status such as published-versus-draft or active-versus-archived. Numeric attribute ranges such as a rating, price, or confidence threshold are filters too. What they share is that they cut the searchable set down to the documents that are valid for this particular query. ANN search can be combined with such filters, but *how* the filter is applied makes a large difference to recall, and this is a common source of confusion. There are two ways to filter an ANN query: The two forms look almost identical in the query body, which is why the difference is easy to miss. The wrapper form places the filter *outside* the knn clause, in a surrounding boolean query: ``` GET /documents/_search{ "query": { "bool": { "must": [ { "knn": { "embedding": { "vector": [...], "k": 128 } } } ], "filter": [ { "term": { "tenant_id": "acme" } }, { "range": { "created_at": { "gte": "now-30d" } } }, { "term": { "language": "en" } } ] } }} ``` The efficient form places the filter *inside* the knn clause, as a parameter of the vector search itself: ``` GET /documents/_search{ "size": 10, "query": { "knn": { "embedding": { "vector": [...], "k": 10, "filter": { "bool": { "filter": [ { "term": { "tenant_id": "acme" } }, { "range": { "created_at": { "gte": "now-30d" } } } ] } } } } }} ``` In the first query the filter is applied to whatever the k=128 vector search returned; in the second, the filter constrains the vector search as it runs. If recall on filtered queries is lower than expected, the wrapper form is the first thing to check. The engine can also be told when to switch from filtered ANN to an exact scan over the filtered set, through an index setting that defines the threshold below which exact search is used: ``` PUT /documents/_settings{ "index.knn.advanced.filtered_exact_search_threshold": 20000 } ``` With a threshold like this, when the filter narrows the candidate set in a segment below the threshold, the engine performs an exact search over those candidates automatically, giving exact recall for selective filters without any change to the query. Exact K-Nearest Neighbor (KNN) search does what ANN avoids. It computes the similarity between the query vector and every candidate document, and returns the true top matches. Because it is a brute-force scan, it does not scale to a full index the way ANN does. But it has a property ANN does not: it never misses a true neighbor, so its recall is exact by definition. Exact KNN is expressed as a script score query. A filter selects the candidate set, and the script computes the vector similarity for every candidate that passes it: ``` GET /documents/_search{ "size": 10, "query": { "script_score": { "query": { "bool": { "filter": [ { "term": { "tenant_id": "acme" } }, { "range": { "created_at": { "gte": "now-30d" } } } ] } }, "script": { "source": "knn_score", "lang": "knn", "params": { "field": "embedding", "query_value": [...], "space_type": "innerproduct" } } } }} ``` Here the bool.filter runs first and produces the candidate set, and the knn_score script computes the exact similarity for each of those candidates. There is no k and no graph. Every document inside the filter is scored, so the result is exact. Even with efficient filtering available, there are cases that call for exact KNN. How many documents survive the filter depends on the filter dimensions. A highly selective combination might leave a few thousand, while a looser one can leave hundreds of thousands. When the filtered set is small, an exact scan over it is cheap and guarantees that no relevant document inside the filter is missed, with no dependence on how k or a filtered-search threshold is tuned. As the filtered set grows into the hundreds of thousands, the exact scan gets more expensive, which is where efficient filtering earns its place by letting the engine switch to filtered ANN above the exact-search threshold. If the recall requirement is strict, for example in a system where missing a relevant document has a real cost, then exact scoring over the filtered candidate set is often the correct choice, not an approximation of it. Two classes of problem show up once a RAG system is under real load. **Recall problems.** The most common cause of silent recall loss is the wrapper form of filtering described earlier, where the ANN query returns its k candidates and the filter is applied afterward, leaving fewer than k matching results when the filter is selective. This is easy to miss because the query still returns something; it is just missing relevant documents. A second and related recall problem is simply using too small a k or too small an ef_search, so the graph is not explored deeply enough and true neighbors are left undiscovered. Both are silent, since the system returns results, they are just not the best ones, which is why filtered recall has to be measured against an exact baseline rather than assumed. **Latency problems.** On the exact path, latency is driven almost entirely by the number of candidates that survive the filter, because the similarity is computed for every one of them. A loose filter that lets a large candidate set through will be slow no matter what else is tuned. On the ANN path, latency is driven by ef_search and m, since exploring more of the graph for higher recall costs more distance computations. In both cases, the cost is dominated by the number and the cost of the per-document similarity computations, which leads directly to the next two optimizations. The similarity between two vectors can be measured in different ways, and the choice affects both correctness and latency. The two most common measures in RAG are cosine similarity and inner product (dot product). They are closely related: ``` cosine(q, d) = (q . d) / (|q| * |d|) ``` The term q . d is the dot product, the sum of element-wise products across all dimensions. Cosine similarity is that same dot product, divided by the two vector magnitudes |q| and |d|. In other words, dot product is the numerator of cosine similarity. Computing cosine does not avoid computing the dot product. It computes the dot product and then does additional work to normalize it. This has a direct latency consequence on the exact path. If the document vectors are not normalized to unit length when they are indexed, then cosine similarity has to recompute the document magnitude |d| for every document it scores. That magnitude is a second full pass over all the dimensions of the vector, plus a square root, done per document. Over a large candidate set, this roughly doubles the vector arithmetic. The query magnitude |q| is a constant for the whole query and should be computed once, not per document, but a naive scoring function may recompute even that on every document. Dot product avoids all of this. It computes only the numerator and returns it. When the document vectors are normalized at index time, dot product and cosine produce identical rankings, so one gets the exact same results at lower cost. Even when the vectors are not normalized, dot product can still match cosine in practice if the vector magnitudes are close to constant across the corpus, because then the normalization term does not reorder the top results. It also has the advantage of matching the space that the ANN graph is built in when that graph uses inner product, so the exact path and the ANN path score consistently. In the exact score script, the measure is selected by the space_type parameter. Cosine similarity is: ``` "script": { "source": "knn_score", "lang": "knn", "params": { "field": "embedding", "query_value": [...], "space_type": "cosinesimil" }} ``` and inner product is the same query with one word changed: ``` "script": { "source": "knn_score", "lang": "knn", "params": { "field": "embedding", "query_value": [...], "space_type": "innerproduct" }} ``` There is a second, separate choice hiding here: not just which metric, but how the score is computed. OpenSearch can score with a painless script (lang: "painless", calling cosineSimilarity(...)) or with the k-NN score script (lang: "knn", source: "knn_score") shown above. The painless path runs through an interpreted, sandboxed scripting engine, and pays per-document interpreter and parameter-marshalling overhead for every candidate it scores. The k-NN score script runs a compiled native similarity function instead. On a brute-force scan over a large filtered candidate set, this native path is meaningfully faster, and the gain is independent of the metric: it applies whether the space is cosinesimil or innerproduct. So there are two stacked wins available. Moving from painless cosineSimilarity to the native knn_score_script gives the engine win on its own; additionally switching the space from cosinesimil to innerproduct (on normalized, or near-constant-magnitude, vectors) adds the smaller win of skipping the per-document norm recomputation. Measure the two separately, since the native-engine effect usually dominates. Painless still has its place: it is the escape hatch for custom scoring that the built-in spaces cannot express, for example blending vector similarity with a recency decay, a popularity boost, or a business rule into a single score. When the scoring is a plain vector distance, prefer the native score script; when it needs arbitrary custom logic, painless is the tool, at the cost of the interpreter overhead. The practical guidance is: This must be measured, not assumed. If the embedding model changes such that vector magnitudes spread out, the ranking can shift, and one should re-measure or normalize. Embedding vectors are usually stored as 32-bit floating point numbers. A 1024-dimensional vector then occupies four kilobytes. Across hundreds of millions of documents, this is a large amount of memory, and, more importantly for the exact path, it is a large number of bytes that must be streamed through the processor for every similarity computation. The exact scan is bound by memory bandwidth, so the size of each vector directly affects its latency. Scalar quantization reduces this. A 16-bit floating point (fp16) encoder stores each dimension in two bytes instead of four, halving the size of every vector. This halves the memory needed for the index and, on the exact path, roughly halves the bytes that must be streamed per document, which reduces latency. It is configured as an encoder on the vector field at index creation time: ``` "embedding": { "type": "knn_vector", "dimension": 1024, "method": { "name": "hnsw", "engine": "faiss", "space_type": "innerproduct", "parameters": { "ef_construction": 512, "m": 16, "encoder": { "name": "sq", "parameters": { "type": "fp16" } } } }} ``` Without the encoder block, the vectors are stored at full 32-bit precision, so adding it is the difference between four bytes and two bytes per dimension. The concern with any quantization is recall regression: does reducing the precision change which documents are retrieved? For fp16 scalar quantization, the impact is typically negligible. The reason is that embedding similarity is a smooth, aggregate quantity summed over hundreds or thousands of dimensions. The small rounding error introduced in each dimension by fp16 averages out across the sum and rarely changes the ordering of the top results. This is very different from the recall loss of ANN, which comes from the graph missing a neighbor entirely. fp16 still compares against the same vectors. It just compares slightly less precise copies of them. The guidance is to enable fp16 scalar quantization and validate recall against the full-precision baseline on a representative query set. In practice the recall difference is within noise, and the memory and bandwidth savings are real. More aggressive quantization, such as product quantization to a single byte per dimension, saves more memory but has a larger and less predictable recall impact, and should only be used if the recall measurement supports it. To reason about latency, it helps to understand how a search is actually executed across a cluster, because that is what sharding controls. An index is split into *shards*, and the shards are distributed across the data nodes. A search runs in two phases. In the **query phase**, the node that received the request, the *coordinator*, forwards the query to one copy of every shard. Each shard executes the query locally against its own portion of the data, finds its own top matching document IDs and their scores, and returns just that small list of IDs and scores to the coordinator. The coordinator then merges the per-shard lists into a single global ranking and keeps the top results. In the **fetch phase**, the coordinator asks the relevant shards for the stored content of only those final documents, assembles them, and returns the response. The important consequence is that the shards work *in parallel*. Each shard searches its own slice of the data at the same time, with roughly one search thread per shard, so the wall-clock time of the query phase is set by how long a single shard takes, not by the total data size. This is why sharding is the main lever for *single-query* latency: the work of one query is divided across the shards it touches. On the exact path, where the cost is scanning the filtered candidates, splitting those candidates across more shards means more cores work on the same query at once, and the query finishes sooner. The way to size shards is in terms of cores, not just data size: There is a point of diminishing returns. Each shard the query touches adds fixed overhead: graph entry, the coordinator merging one more per-shard list, and a wider fetch phase across more shards. Below roughly 150,000 to 200,000 documents per shard, that overhead starts to eat the parallelism gain. So the useful range is bounded: enough shards that a single query spreads across many cores, but not so many that per-shard overhead dominates. The tradeoff to keep in mind is latency versus throughput. More shards lower the latency of a single query by parallelizing it, but they raise the total per-query overhead, which lowers the maximum queries per second the cluster can sustain. If the system is bound by single-query latency, meaning a user waiting on one RAG response, more shards help. If it is bound by throughput, meaning many concurrent queries already keeping the cores busy, the existing sharding is likely fine, and adding shards would hurt. Shard count is set at index creation and changing it requires a reindex, so it is worth deciding deliberately: ``` PUT /documents{ "settings": { "index": { "number_of_shards": 80, "number_of_replicas": 1 } }} ``` Replicas are a separate matter: they do not lower single-query latency, but they add copies of each shard that let the cluster place more queries in parallel, so they raise throughput and provide redundancy. The two-phase execution above also explains a distinct optimization. Recall that the vector similarity is computed in the *query phase*, from the vector values, and that the stored _source document is only loaded in the *fetch phase*, and only for the handful of documents that are actually returned. The scoring never needs _source, so what a query returns is a separate cost from how it is scored. This matters because the embedding vector is stored inside _source by default. A 1024-dimensional vector serialized as JSON text is far larger than its binary form, so every returned document drags a large embedding array through the fetch phase and across the network, even though the caller never uses it. A RAG system only needs the text and the few metadata fields used to build the LLM context. There are two independent controls, and they solve two different costs. The first is to stop *storing* the embedding in _source at index time, by excluding it in the mapping: ``` "_source": { "excludes": ["embedding", "raw_payload"] } ``` This shrinks the stored document on disk and in the page cache. The vector is still indexed for search, since only its copy inside _source is removed, so search is unaffected while the stored documents become much smaller and cache better. The second is to control what each *query* returns in the fetch phase. The most direct way is to disable _source retrieval entirely and ask for only the required fields explicitly, which are returned from doc values or stored fields rather than by deserializing the whole document: ``` GET /documents/_search{ "_source": false, "fields": ["chunk_text", "title", "url", "tenant_id"], "query": { "knn": { "embedding": { "vector": [...], "k": 10 } } }} ``` A lighter-weight alternative, if the response shape needs to stay as _source, is to filter _source to just the needed fields instead of turning it off: ``` { "_source": { "includes": ["chunk_text", "title", "url"] }, "query": { "knn": { "embedding": { "vector": [...], "k": 10 } } }} ``` The difference is that _source: false with fields avoids loading and deserializing the stored document at all, pulling values from column-oriented doc values, which is cheaper when the stored document is large. The _source include filter still loads and deserializes the document, then strips it, so it saves network cost but not the deserialization cost. One caveat with fields: values come back as arrays under a fields key rather than under _source, so the response-parsing code has to change, and any field requested this way must have doc values or be stored (a text-only field with neither will not be returned). Two smaller operational levers help: POST /documents/_forcemerge?max_num_segments=1 2. **Warm the cache.** After a deployment or a merge, the graph and the vectors are cold. Warming them into memory before serving traffic avoids the first queries paying a cold-load latency penalty. GET /_plugins/_knn/warmup/documents The retrieval layer is where a production RAG system succeeds or fails, and it is governed by the tension between recall and latency. ANN gives scale at the cost of some recall; exact KNN gives exact recall at the cost of scale; filtering forces a choice between them. The optimizations described here, namely scoring with dot product instead of cosine, quantizing to fp16 without regressing recall, filtering efficiently so ANN and exact search combine cleanly, and trimming the response payload, reduce latency while holding recall, so one does not have to trade one axis away to improve the other. None of these should be adopted on faith. Each depends on the embedding model, the data distribution, and the filter characteristics of the specific system. The consistent theme is to establish the exact result as ground truth, make one change at a time, and measure recall and latency against that baseline on a representative query set before rolling the change out. The right configuration is the one the measurements support, not the one that is expected to work. [OpenSearch Optimizations for Production RAG](https://pub.towardsai.net/opensearch-optimizations-for-production-rag-c335ac49f3e5) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.