RAG for Developers: What Actually Happens Between a User Query and an AI Answer

A developer explains the inner workings of Retrieval-Augmented Generation (RAG) pipelines, detailing the steps from document parsing and chunking to embedding, retrieval, reranking, and final generation. The article emphasizes the importance of retrieval quality and suggests hybrid search combining keyword and semantic methods for technical queries.

If you've built an LLM application, you've probably discovered a problem pretty quickly: The model doesn't know your application's data. Your product documentation might change tomorrow. Your company's API may be private. Your database may contain information that never appeared in the model's training data. You could fine-tune a model, but that isn't always the right answer. This is where Retrieval-Augmented Generation RAG comes in. The Short Version A basic RAG pipeline looks like this: User Query ↓ Retriever ↓ Relevant Chunks ↓ Context ↓ LLM ↓ Answer The retriever finds useful information. The LLM turns that information into an answer. Simple, right? The interesting engineering problems start after that. A More Realistic Pipeline A production RAG system might look closer to: Documents ↓ Parsing ↓ Chunking ↓ Embeddings ↓ Index ↓ Query Processing ↓ Retrieval ↓ Reranking ↓ Context Construction ↓ LLM ↓ Answer + Sources There are really two workflows here. The first prepares your knowledge. The second runs every time somebody asks a question. Your data might come from: Markdown PDFs Websites Databases Internal documentation APIs Cloud storage The raw files usually need to be cleaned and structured before retrieval. If your source material is messy, retrieval starts with a disadvantage. Long documents are usually divided into smaller passages. Consider API documentation containing sections for authentication, pagination, rate limits, and webhooks. If somebody asks: "What's the API rate limit?" Returning the entire documentation page is unnecessary. A focused chunk is more useful. But chunking has a trade-off. Too large → unnecessary context. Too small → missing context. There isn't one perfect chunk size for every application. Test it against your actual documents and queries. An embedding model converts chunks into numerical representations. The point isn't the numbers themselves. The useful property is that semantically related text can be represented similarly. For example: Query: "What's the maximum number of API requests?" Document: "Requests are limited to 100 calls per minute." The wording isn't identical, but the meaning is closely related. At query time, the user's question is used to find relevant chunks. There are several ways to do this. Keyword retrieval Good when exact words matter. Vector retrieval Good when semantic meaning matters. Hybrid retrieval Combines both. For developer-facing systems, hybrid retrieval can be particularly useful because technical searches often contain exact identifiers alongside natural-language questions. The first retrieval step may return more results than the LLM actually needs. A reranker can score those candidates again. The result might go from: 20 retrieved chunks ↓ 5 strongest chunks ↓ LLM This helps keep irrelevant material out of the final context. At this point, the application combines: System instructions + Retrieved context + User question The model can now generate an answer using the retrieved information. This is the "generation" part of RAG. The Important Part: Retrieval Quality Here's the part that's easy to underestimate. If your retriever returns the wrong documents, the LLM doesn't magically know that. It may generate a confident answer from bad context. That's why a RAG evaluation should ask two separate questions: Did we retrieve the right information? and Did the model use that information correctly? Treating those as one problem makes debugging much harder. Why Hybrid Search Often Makes Sense Imagine a developer searches: "ERR AUTH 401 API v3" Semantic search may understand the general concept, but exact identifiers can be important. Keyword search handles exact terms well. Semantic search handles meaning well. Combining them can provide a better retrieval strategy for mixed queries. RAG vs Fine-Tuning A useful mental model: RAG = give the model knowledge at runtime. Fine-tuning = change how the model behaves. If your documentation changes every week, RAG is often much more practical than repeatedly retraining a model to memorize the latest version. Agentic RAG Traditional RAG normally performs retrieval according to a predefined pipeline. Agentic RAG gives an AI agent more control. For example, a user could ask: "Compare our 2025 and 2026 security policies and explain the important changes." An agent might: Search the 2025 policy. Search the 2026 policy. Identify relevant sections. Compare the retrieved content. Generate the final response. This is more flexible, but also more complex to build, monitor, and evaluate. Common Failure Modes Some RAG problems show up repeatedly: Wrong chunks retrieved Poor chunk boundaries Outdated documents Missing metadata Weak exact-term matching Too much context Too little context Incorrect permissions Unsupported model-generated claims So when a RAG application fails, don't immediately replace the LLM. First inspect the retrieval pipeline. Production Considerations A production RAG system should think about: Access control Document freshness Metadata filtering Latency Token costs Retrieval evaluation Answer evaluation Monitoring Prompt injection risks The vector database is only one part of the architecture. Final Takeaway RAG is often described as: Retrieve → Augment → Generate. That's accurate, but it hides most of the engineering work. Reliable RAG depends on the quality of the entire retrieval pipeline. I put together a longer practical guide covering the architecture, embeddings, chunking, retrieval, RAG vs fine-tuning, traditional vs agentic RAG, challenges, and optimization: Full RAG guide — AI Tools Vault