I watched a customer demo their new RAG application last month. They had spent three months building it. The interface was beautiful. The LLM responses were eloquent and well-formatted. And almost every answer was subtly, confidently wrong.
“The model keeps hallucinating,” they told me, frustrated. They had tried three different LLMs. They had adjusted temperature settings. They had rewritten their prompts a dozen times. But here is what nobody had looked at yet: the search layer feeding context into those prompts.
This is the part of RAG that deserves more attention than it gets: most quality problems are retrieval problems. When your generative AI makes things up, it is usually because the context it received was off. Too much, too little, or just plain irrelevant. The LLM is doing exactly what you asked: generating fluent text based on the information you provided. If that information misses the mark, well, garbage in, eloquent garbage out.
The challenge is that retrieval tends to get treated as a solved problem. Chunk the documents, throw everything into a vector database, retrieve the top 10 results, done. That works for demos. In production, though, you need the system to distinguish between a precise question about part number XJ-447 and an abstract question about how plumbing works. That distinction is where things get genuinely interesting.
Here is something I find fascinating about document chunking: it looks like a technical problem, but it is actually a semantic one.
The conventional wisdom goes something like this: take your documents, split them into 512-token chunks with 50-token overlap, generate embeddings, done. Clean. Systematic. And it often destroys retrieval quality in ways that are genuinely hard to diagnose.
Think about what you are actually doing when you chunk a document. You are deciding what the atomic unit of meaning is for your system. That 512-token window might slice right through the middle of a critical explanation. It might mash together two unrelated concepts that happened to share a page. Your chunks are not just storage units. They are the answers your system will retrieve.
I have seen teams spend weeks optimizing their embedding models while using paragraph-based chunking that splits “The solution to this problem is…” from the actual solution in the next paragraph. Once you see it, the fix is obvious. But it is easy to miss because chunking happens early in the pipeline and its effects show up late, as vague or wrong answers from the LLM.
The approach that works: chunk semantically, not mechanically. If you are working with structured documents, and most enterprise content is structured, respect that structure. A section about pricing belongs together. A troubleshooting procedure should stay intact. A paragraph explaining a concept should not be split because it crossed some arbitrary token threshold.
In my own work, I use a tiered strategy: first split at structural markers (headings, section breaks), then embed sliding sentence windows and split at valleys in cosine similarity between adjacent windows. Those valleys are the points where the topic naturally shifts. This technique (well described visually here) is now built into LangChain and LlamaIndex as their default semantic chunkers. The embedding-based valley detection catches topic boundaries that structural markers miss, without the cost of sending every chunk through an LLM.
Here is the related subtlety: not every chunk needs to be retrieved independently. Sometimes you want to retrieve a full document, like when someone needs that PDF about 1957 nail prices. In those cases, use chunks as subdocuments with a parent-child relationship. The chunks help you find the right document, but you return the whole thing. Other times, especially in RAG scenarios, you want the chunk itself. You are not looking for a document about plumbing. You want the specific paragraph that explains how a P-trap works.
Most teams pick one approach and apply it everywhere, which explains why their system is great at some queries and puzzling at others.
Here is something about vector search that is genuinely counterintuitive once you see it: vectors never return zero results.
That is not a bug, it is math. In vector space, everything has a nearest neighbor. Ask a question about quantum physics to a database of cooking recipes, and it will cheerfully return the five “most relevant” results. They will be completely wrong, but the distance metrics will look reasonable.
This is especially tricky in RAG systems because the LLM will work with whatever you give it. Feed it those irrelevant cooking recipes as context for a quantum physics question, and it will generate a confident-sounding answer that blends both domains into nonsense. The user has no signal that the underlying search missed.
Lexical search does not have this problem. If your keywords do not match, you get nothing back. That absence is actually useful information. When someone asks about part number XJ-447, you want exact matching. You want lexical search. When they ask about “solutions for preventing pipe corrosion in coastal environments,” you want semantic search to correlate terms by the company they keep.
The right approach is not choosing between lexical and semantic search. It is knowing when to use each, and increasingly, using both together. Hybrid search strategies that combine keyword matching with semantic correlation consistently outperform either approach alone.
But you need something to orchestrate that decision. Is this a precise query or an abstract one? Does it contain specific identifiers that demand exact matching? Are there domain-specific terms that semantic search might miss? Most teams try to solve this with increasingly complex query logic. The more interesting move is letting an agent figure it out.
So what does a properly architected RAG search layer look like?
Start with the retrieval target. Before you chunk anything, ask: what am I trying to retrieve? If you are building a document discovery system, your target is documents. Chunk for findability, but return the full document. If you are building a question-answering system, your target is the specific piece of information that answers the question. Chunk for semantic coherence, and return just that chunk.
For RAG specifically, smaller and more semantically coherent beats larger and more comprehensive. You are not trying to give the LLM everything that might be relevant. You are trying to give it exactly what is relevant. A tightly focused paragraph that directly addresses the question will outperform a kitchen-sink approach that includes three pages of tangentially related content. This is where reranking enters the picture, and where it helps to be precise about what reranking actually does. Reranking is not a magic quality boost you toggle on. It is a precision tool. If your initial retrieval is pulling back mostly irrelevant results, reranking just reorders them. The conventional wisdom says to over-sample: retrieve 50 results, rerank them, take the top 10. Think about what that means. You are deliberately retrieving low-relevance results (because that is what results 11 through 50 usually are) and hoping the reranker will find hidden gems.
Sometimes that works. More often, you are adding latency and cost while marginally improving already-poor results. The better approach: write queries that retrieve high-precision results in the first place, then use reranking to fine-tune the order. If you need to retrieve 50 results to get 10 good ones, the query strategy is where the leverage is.
This is also where agentic frameworks start to make sense. Instead of building the perfect query upfront, let an agent run multiple query strategies, evaluate the results, and decide what to pass forward. Amazon OpenSearch Service has internal agentic capabilities that can orchestrate this: running lexical and semantic searches in parallel, applying different ranking strategies, and using an LLM to judge which results actually answer the question.
That last point is becoming the new standard for search quality evaluation. Traditionally, measuring precision and recall required humans to manually review results. That does not scale. But an LLM can evaluate whether retrieved chunks contain information relevant to the query, and it can do it in real time. This lets you build systems that are self-critical, that recognize when search results are poor and try a different approach rather than confidently generating wrong answers.
The RAG systems that work in production are not the ones with the fanciest LLMs or the most sophisticated prompts. They are the ones that treat retrieval as a first-class problem worthy of the same engineering attention as the generative layer.
We are moving away from the “embed everything, retrieve top-k, generate answer” pipeline toward systems that reason about queries, orchestrate multiple search strategies, and critically evaluate their own results before generating responses. The search layer is becoming intelligent, not just functional.
If you are building or fixing a RAG system this week, try starting not with the LLM but with a simple question: when I retrieve context for this query, am I getting back the specific information that actually answers it? Pull up your logs, look at what is being retrieved, and check whether a human could answer the question from that context alone. Because if a human cannot, the LLM will not either. It will just be more eloquent about being wrong. And that is a problem worth the s/prompt engineering/actual engineering/.