RAG Is Not an Architecture: Choosing the Right Retrieval Strategy for GenAI

A developer argues that Retrieval-Augmented Generation (RAG) is a retrieval strategy rather than a complete architecture, and that production GenAI systems should choose retrieval approaches based on the problem at hand. The writeup contrasts semantic vector search, hybrid retrieval, graph-based retrieval (GraphRAG), agentic retrieval, and long-context approaches, noting trade-offs in accuracy, latency, cost, and complexity. It recommends that architecture follow the problem rather than defaulting to a single vector-search pipeline.

Retrieval-Augmented Generation RAG has become one of the default patterns for building GenAI applications. But there is a problem. Many systems treat RAG as an architecture rather than a retrieval strategy. The typical design looks like this: User Query ↓ Vector Search ↓ Top-K Chunks ↓ LLM ↓ Answer It works. It can even look impressive in a demo. But production systems are rarely that simple. The real question isn't: «Should I use RAG?» It is: «What kind of retrieval does this problem actually require?» RAG Is a Pattern, Not a Complete Architecture RAG fundamentally means retrieving external information and providing it to a generative model as context. That's useful—but it doesn't tell you: Those are architecture decisions. And choosing the wrong retrieval strategy can create problems with accuracy, latency, cost, and maintainability. The simplest implementation is semantic vector search. Query ↓ Embedding ↓ Vector Database ↓ Top-K Chunks ↓ LLM ↓ Answer This works well when the user's question can be answered from relatively independent pieces of text. Typical examples: But semantic similarity has limitations. Suppose a user searches for: INC-847291 A semantically similar result isn't necessarily the correct result. Sometimes the exact token matters more than semantic meaning. That's where hybrid retrieval becomes useful. Hybrid retrieval combines multiple retrieval mechanisms, commonly: Conceptually: ┌── Vector Search ──┐ Query ───────────┤ ├──→ Candidate Results └── Keyword Search ─┘ ↓ Reranking ↓ LLM This is particularly useful when your data contains: For many enterprise applications, hybrid retrieval is a more practical starting point than pure vector search. Some questions aren't really about finding similar text. They're about understanding relationships. Imagine a knowledge base containing: Customer ↓ Purchased ↓ Product ↓ Affected by ↓ Incident ↓ Caused by ↓ Service Now consider a question such as: «Which customers were affected by incidents caused by a particular service?» This isn't simply a semantic similarity problem. The answer requires following relationships across multiple entities. That's where graph-based retrieval can become valuable. GraphRAG can help when the knowledge domain contains: But GraphRAG also introduces additional complexity. A graph isn't automatically better just because it is more sophisticated. Now consider a question where one retrieval operation isn't enough. An agentic system can decide: User Query ↓ Reason about task ↓ Retrieve information ↓ Evaluate results ↓ Retrieve again if necessary ↓ Use tools ↓ Synthesize ↓ Verify ↓ Answer The retrieval process becomes dynamic rather than fixed. This can be useful for tasks requiring: But there is a trade-off. More autonomy means more system complexity. It can also increase: Agentic RAG should therefore solve a real problem—not simply make the architecture sound more advanced. There is another option that is frequently overlooked: Don't retrieve aggressively. Modern LLMs can process substantially larger contexts than earlier models. For some workloads, it may be better to provide a large, carefully selected context rather than splitting everything into small chunks and hoping retrieval finds the right pieces. This can be particularly useful when: This doesn't mean "long context is better than RAG." It means retrieval and context management should be evaluated together. The Architecture Should Follow the Problem A production GenAI system might look more like this: User Query ↓ Intent Detection ↓ Retrieval Strategy Selection ↓ ┌────────────────┼────────────────┐ ↓ ↓ ↓ Hybrid Search Graph Search Long Context └────────────────┼────────────────┘ ↓ Reranking ↓ Context Assembly ↓ LLM Reasoning ↓ Verification ↓ Answer And even this isn't universal. Different applications may require completely different architectures. For example: Document Q&A Query → Hybrid Retrieval → Reranking → LLM Relationship-heavy enterprise knowledge Query → Entity Extraction → Graph Traversal → LLM Complex research workflow Query → Planning → Retrieval → Tool Use → Retrieval → Synthesis Small, highly connected document collection Query → Relevant Documents → Long Context → LLM Don't Choose Architecture by Trend One of the easiest mistakes in GenAI engineering is selecting technology before defining the problem. "Let's use GraphRAG." "Let's build an agent." "Let's add a vector database." "Let's use a larger context window." These aren't architecture decisions until you understand the workload. The better approach is to evaluate: Accuracy Can the system consistently retrieve and use the information required to answer correctly? Latency How quickly does the system need to respond? Cost How much retrieval, inference, storage, and token usage can the application afford? Complexity How difficult will the system be to build, debug, and operate? Maintainability Can the architecture evolve as the data, models, and requirements change? The best architecture is usually the one that provides the right balance across all five. RAG Should Be a Design Decision A vector database doesn't automatically make an application well-designed. GraphRAG isn't automatically better than traditional RAG. Agentic RAG isn't automatically more intelligent. And long context isn't automatically cheaper or more accurate. These are tools and strategies. The architecture comes from the problem. Before choosing a retrieval strategy, ask: That's a much better starting point than simply asking: «"Should we use RAG?"» RAG is not the architecture. It is one of the building blocks. The engineering challenge is choosing the right combination of retrieval, reasoning, context, tools, and verification for the problem you're actually solving.