RAG is Only as Good as its Search: Why AI Search is the Real Differentiator

A pattern in RAG implementations shows that retrieval, not the language model, is the root cause of incorrect answers, according to a technical analysis. The article argues that semantic search using embedding models and vector similarity, such as OpenSearch's neural search, fixes keyword search's failures by retrieving by meaning, picking up intent, and providing actionable confidence scores. Teams should prioritize building the search layer over choosing an LLM.

There’s a pattern that plays out in almost every RAG implementation. The team spends weeks evaluating language models, GPT-4 vs. Claude vs. Gemini, fine-tuning vs. prompt engineering, temperature settings and system prompts. They get the generation step looking good in demos. Then they ship, and the answers are still wrong. Not hallucination-wrong. Not obviously broken. Just quietly, confidently wrong in ways that are hard to pin down and even harder to explain to stakeholders. The problem is almost never the language model. It’s retrieval. When engineers sketch out a RAG system, the LLM usually ends up at the centre, with data feeding in from one side and answers coming out the other. But that’s backwards: the search layer is the system. The LLM is really just the output formatter. Retrieval decides what information the model has to work with, and the model then does something useful with that: summarizing it, synthesizing it, pulling out what’s relevant. But it can’t fix bad inputs. Give it the wrong document and it’ll confidently summarize the wrong document. Give it nothing relevant and it’ll improvise. Nearly every quality problem in a RAG system comes down to one of two things. Either retrieval handed the model the wrong context, or it handed over nothing useful at all. The model itself is rarely where things actually go wrong. The most important infrastructure decision in a RAG deployment isn’t which LLM you pick. It’s how you build the search layer underneath it. Most enterprise data already lives behind some form of search: an OpenSearch cluster for logs and documents, a database with full-text search, a SharePoint index somewhere. So when teams start building RAG, they naturally reach for whatever’s already there. Keyword search, BM25-based retrieval, is genuinely good at what it was built for: finding documents that contain specific terms. For decades it was the best tool around, and for certain kinds of queries it still holds up fine. But it breaks RAG in three specific ways. AI search, semantic search built on embedding models and vector similarity, directly fixes the first two problems, and changes how you can even approach the third. It retrieves by meaning, not by words. An embedding model turns text into a high-dimensional vector, essentially a point in a space where meaning, not exact wording, determines how close two things sit to each other. Documents and queries that mean similar things end up near each other, whether or not they share any vocabulary at all. That’s how a search for “how do I reset my credentials” can surface a document titled “Account Recovery Procedures” even though none of those words appear in the query. The meaning lines up, the vectors sit close together, and retrieval succeeds exactly where keyword search would have come up empty. OpenSearch’s neural search is built around exactly this idea. A query comes in, it passes through the same embedding model used when the documents were indexed, gets converted into a vector, and OpenSearch finds the nearest matches using approximate nearest neighbour search, all inside the same platform already running your keyword search. No separate vector database bolted on, no extra infrastructure to babysit. It picks up on intent in a way keyword search never could. Embedding models are trained on enormous amounts of natural language, so they’ve picked up something real about how meaning works, not just which words tend to appear together, but what a question is actually reaching for. Someone asking “am I covered if my laptop gets stolen?” and a document titled “theft protection coverage” end up close together in vector space, because the intent behind the question and the substance of the document line up. The search picks up on that alignment even though the wording doesn’t match at all. It gives you a confidence score you can actually do something with. Vector similarity returns a distance metric. A high-similarity result is a strong match, a low-similarity one is weak. That opens up something keyword search never really allowed: setting a threshold. If nothing clears a minimum similarity score, the system can say “I don’t have reliable information on that” instead of surfacing a mediocre match and letting the model run with it anyway. It doesn’t fully solve the always-returns-something problem, but it gives you a lever to pull. Tuning that threshold against a real evaluation set is one of the more valuable, and more overlooked, calibration steps in a RAG deployment. Vector search isn’t a silver bullet either. It has its own failure modes, and understanding them is really what separates a search layer that works in a demo from one that holds up once it’s live. Precision failures. Vector search is approximate by design, it finds what’s close, not what’s exact. Search for invoice number “INV-2024–0098” and you might get back a handful of similar-looking invoices instead. For exact identifiers, product codes, order numbers, error codes, contract references, that’s actually a step backwards from keyword search. Nuance failures. Embedding models squeeze meaning into fixed-size vectors, and something gets lost in that compression. The gap between “liability is capped” and “liability is not capped” is one word, but it flips the meaning entirely, and that distinction can wash out in vector space. Two documents reaching opposite legal conclusions can end up sitting right next to each other. Domain failures. General-purpose embedding models are trained on general-purpose text. They don’t necessarily capture your domain’s specific terminology or relationships with much precision. A model that’s great with everyday English can fall apart on specialised legal, medical, financial, or technical content. The fix for all three is hybrid search: running semantic vector search and keyword search side by side, then combining what each one finds. They fail in opposite directions, so together they cover each other’s blind spots. Keyword catches exact terms and identifiers where vector search gets fuzzy. Semantic catches meaning and intent where keyword search goes blind. Across real enterprise data, hybrid consistently beats either approach running alone. This is one of the places where OpenSearch’s design quietly pays off. Because it stores your text fields and your vector fields in the same index, a hybrid query is a single request, not two separate systems whose results you’re stitching together in application code. The platform runs both searches in parallel and handles the combining itself, through what it calls a search pipeline. At any real scale, that simplicity adds up. The combining step is something most articles skip past entirely. BM25 scores and cosine similarity scores sit on completely different scales, so you can’t just merge the two lists and hope for the best. OpenSearch’s normalisation step puts both sets of scores onto a shared scale first, then blends them. You get to decide the weighting, how much to lean toward meaning versus exact terms. A corpus full of identifiers and codes leans keyword-heavy. A conversational knowledge base leans semantic. Where exactly that balance sits is something you tune against your own data, not something you set once and forget. Re-ranking sits on top of all that. Once hybrid retrieval returns its top candidates, a cross-encoder model scores each one against the original query with more precision than the retrieval step could manage on its own. In a lot of mature RAG systems, re-ranking ends up being the single biggest quality improvement you can make. OpenSearch runs it as part of the same search pipeline, so retrieve, normalise, and re-rank all happen in one query, with one response, from one platform. Calling AI search a feature, something you bolt on to make search a bit better, undersells what it actually is, and it leads teams to make real architectural mistakes. In a RAG system, AI search is the thing everything else stands on. The LLM’s quality is bounded by retrieval quality. Swap in a more capable model and you’ll gain nothing if retrieval keeps handing it the wrong context. Run something smaller and cheaper and you can get excellent answers, as long as retrieval is reliably surfacing the right one. The model sits downstream of search, directly and consequentially. Data quality only matters because search expresses it. A well-maintained corpus, clean metadata, no duplicates, nothing stale, is valuable precisely because good retrieval can actually surface it. That’s the whole point of the search layer: it turns data quality into answer quality. But no search platform, however good, can rescue a badly maintained corpus. It can only find what’s actually there. OpenSearch gives you the filtering, the metadata indexing, the field-level control to make data quality something you can act on. What it can’t do is invent a needle that was never in the haystack to begin with. Security and access control belong in the search layer, not somewhere downstream. Enterprise RAG has to respect whatever permissions already govern the data. If someone isn’t allowed to see a document, retrieval shouldn’t surface it, no matter how relevant it looks. That’s not something you can patch at the model level. OpenSearch’s document-level security handles it right at retrieval time: roles get defined, documents get tagged, and filtering happens before anything reaches the LLM. The model never even sees what the user isn’t supposed to see, because it was never pulled in the first place. And evaluation, at its core, is a retrieval question. When a RAG answer comes back wrong, the first thing to ask is whether retrieval failed to find the right document, or found it and the model failed to use it properly. In practice, across most production systems, it’s almost always the first one. Which tells you where the bulk of your measurement and improvement effort should actually go: into tracking precision, recall, and ranking quality, not just judging the final answer. So what does all this actually change about how you’d build and evaluate a RAG system? Start by evaluating retrieval on its own. Build a real evaluation set, actual questions with known correct answers, and measure retrieval quality precision at k, recall at k, mean reciprocal rank separately from how the final answer reads. Teams that do this honestly are often surprised by how often retrieval is simply wrong. No amount of prompt tweaking fixes that. Default to hybrid rather than pure vector search. If you’re already on OpenSearch, both search types live in the same index and a hybrid query is one API call, so the extra cost is minimal. The quality gain on real enterprise data is consistently worth it. Don’t skip it just because pure vector worked fine in the demo. Treat your embedding model as a real system dependency, not an implementation detail. It sets your retrieval ceiling, so it deserves the same scrutiny as your LLM choice. OpenSearch’s ML Commons lets you run embedding models on dedicated ML nodes inside the cluster, with ingest pipelines that generate vectors automatically at index time, so switching models later means re-indexing rather than rewriting application logic. That makes iterating easier, but it also means your first choice matters. Test it against your own domain’s vocabulary before building anything on top of it. Design for the “I don’t know” case on purpose. Set a similarity threshold below which the system openly admits it doesn’t have reliable information, rather than quietly surfacing a weak match. It’s an uncomfortable trade-off, some questions will come back unanswered, but it heads off the much worse failure where confident misinformation goes out the door just because something happened to get retrieved. And put real effort into data hygiene, proportional to how much you’re investing in retrieval technique. Hybrid search with re-ranking can find the needle. It cannot find a needle that was never there. Stale documents, contradictory versions, duplicates nobody’s cleaned up, none of that gets fixed by better search technique. It’s unglamorous work, and it’s also some of the highest-leverage work available. Every conversation about which LLM to use for a RAG system is a conversation that isn’t happening about retrieval quality, data freshness, hybrid search tuning, and evaluation. Those are the harder conversations to have. The answers are messier, the improvements take longer to show up, and there’s no leaderboard ranking retrieval quality on your specific domain. But they’re the conversations that actually matter. The LLM will keep getting better. Whatever’s released six months from now will outperform what’s available today, and that improvement will compound on top of good retrieval, and mostly go to waste on top of bad retrieval. OpenSearch has quietly become the practical foundation for a lot of production AI search, not because it’s the newest thing out there, but because it brings keyword search, neural search, hybrid pipelines, re-ranking, access control, and real operational maturity together in one platform that enterprises already know how to run. The teams getting the most out of it aren’t the ones who tacked neural search onto an existing cluster as an afterthought. They’re the ones who rebuilt their retrieval layer from the ground up with AI search as the foundation, and then actually measured whether it worked. The teams building AI systems people can trust are the ones treating retrieval as the system itself. Not the LLM, not the prompt, not how big the model is. The ones who measured it, tuned it, and built their data infrastructure around it from the start. AI search isn’t something you bolt onto RAG. It’s the infrastructure RAG runs on. Treat it that way. NetApp Instaclustr provides fully managed OpenSearch with native support for neural search, hybrid search pipelines, ML Commons, and document-level security, the retrieval layer that enterprise RAG runs on. RAG is Only as Good as its Search: Why AI Search is the Real Differentiator https://pub.towardsai.net/rag-is-only-as-good-as-its-search-why-ai-search-is-the-real-differentiator-531d33fbab71 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.