# Production-Ready RAG Architecture: Core Patterns Explained > Source: > Published: 2026-08-31 20:01:01+00:00 Most RAG tutorials are tested against one clean PDF or a Wikipedia dump. That’s the wrong test. The moment you point the same pipeline at real company documents — inconsistent formatting, support tickets, sprawling internal wikis — answers start coming back incomplete, hallucinated, or confidently wrong. Two mental models explain why, and they hold for almost every production RAG failure: Every RAG system splits into an offline **indexing phase** and an online **query phase**: ``` Indexing (offline): documents → chunks → embeddings → vector storeQuery (online): question → embedding → retrieval → context assembly → LLM ``` Production systems add caching, reranking, guardrails, monitoring, and fallback logic on top — but this is the foundation everything else sits on. Get the foundation wrong and no amount of tooling fixes it downstream. Chunking is where most systems fail, and it’s usually invisible until you’re debugging a wrong answer three stages downstream. It matters because embedding models work best on coherent spans of text, LLM context windows are finite, oversized chunks introduce noise, and undersized chunks lose the surrounding information a correct answer depends on. **Fixed-size chunking (with overlap).** Split every *N* tokens, overlap 10–20%. Simple and predictable, but indifferent to sentence and paragraph boundaries — it will cut a definition in half without noticing. Common starting point, rarely optimal. **Recursive / hierarchical chunking.** Try natural boundaries first — double newlines, then single newlines, then sentences, then words — falling back only when a section is still too large. This is the default in most frameworks now (see LangChain’s RecursiveCharacterTextSplitter), and it's a meaningful step up from pure fixed-size. **Structure-aware chunking.** Respect the document’s actual structure — headers, sections, code blocks, tables. For technical docs and markdown this usually wins outright. Store parent-child relationships so you can retrieve a small, precise chunk but expand to the full parent section when the answer needs surrounding context. **Semantic chunking.** Use an embedding model or an LLM to detect where the topic actually shifts, and chunk there. Produces the most coherent chunks, at real extra cost (an embedding or LLM call per boundary decision) — usually reserved for corpora where retrieval quality is worth the spend. **Working defaults:** For the actual implementations behind each strategy, [LangChain’s text splitter docs](https://docs.langchain.com/oss/python/integrations/splitters) and [LlamaIndex’s node parser docs](https://developers.llamaindex.ai/python/framework/module_guides/loading/node_parsers/) are the two references worth having open while you build this. Embedding models map text into a high-dimensional space where semantic similarity becomes geometric closeness, typically scored with cosine similarity or dot product. The model you pick matters, but in practice data quality and chunking usually matter more than which embedding model is in the pipeline. Broad categories: general-purpose open models, domain-specific embeddings (legal, medical, code), and proprietary APIs. Pick based on your domain’s vocabulary, not benchmark leaderboards alone. A vector store’s actual job is fast **Approximate Nearest Neighbor (ANN)** search — exact (brute-force) search doesn’t scale past a small collection. Under the hood, most stores use graph-based indexes like HNSW (Hierarchical Navigable Small World) or cluster-based indexes like IVF to avoid comparing your query against every vector in the collection; that’s the tradeoff you’re implicitly accepting when you pick “approximate” over “exact” — a small, tunable chance of missing the true nearest neighbor in exchange for sub-linear query time. You rarely need to tune this yourself early on, but it’s worth knowing it’s there before you’re debugging a “why didn’t it retrieve the obviously-correct chunk” issue. When choosing a vector store, the questions that matter are: does it support metadata filtering, does it support hybrid search (vector + keyword/BM25) natively, how easy is it to run locally versus managed, and what do persistence, replication, and scaling actually look like at your data volume. Common options: Qdrant, Weaviate, Pinecone, Chroma, pgvector, OpenSearch’s k-NN. Early on, the specific vector store matters less than: clean data, good chunking, proper metadata, and the ability to combine vector and keyword search. **Hybrid search** shows up in nearly every serious production system because pure vector search reliably misses exact-match tokens — error codes, product SKUs, IDs — that a keyword/BM25 pass catches trivially. Weaviate’s [hybrid search documentation](https://docs.weaviate.io/weaviate/search/hybrid) is a clean reference for how the fusion between dense and sparse scores actually works if you want to see it under the hood. ``` # Conceptual shape of a hybrid query — exact API varies by vector storeresults = vector_store.hybrid_search( query=user_question, alpha=0.5, # weight between dense (1.0) and keyword (0.0) search filters={"doc_type": "runbook", "product": "checkout-service"}, top_k=25,) ``` Retrieving the top-k most similar chunks is the starting point, not the finish line. In production you’ll consistently hit: the most similar chunk isn’t the most useful one, an answer’s information is split across multiple chunks, the [“lost in the middle” effect](https://arxiv.org/abs/2307.03172) where models under-attend to information buried in the middle of a long context even when it’s technically present, and embedding models that simply don’t understand your domain’s vocabulary. Techniques that move the needle, roughly in order of ROI: You need an evaluation set, not a vibe. Build 20–50 realistic questions, note which chunk(s) should be retrieved for each, and measure directly: does the correct chunk appear in the top 5 or top 10 (Recall@k)? Layer an LLM-as-judge for relevance scoring once you trust manual inspection, not before. If you never measure retrieval quality independently of final answer quality, you will spend real time optimizing the wrong stage of the pipeline. You don’t have to build this evaluation harness entirely by hand — [Ragas](https://www.ragas.io/) is the most widely used open framework for RAG-specific metrics, and [Qdrant’s guide to RAG evaluation](https://qdrant.tech/blog/rag-evaluation-guide/) is a practical walkthrough of building the eval set itself, not just the metrics math. **LlamaIndex** is data-centric — strong, opinionated abstractions around indexing, retrieval, and query engines. It tends to feel more natural when the core problem is genuinely “I have a lot of documents and need good retrieval.” **LangChain** is general-purpose — it started with chains and agents, and has a much larger integration ecosystem. More flexible, and correspondingly heavier when all you need is retrieval. LangChain’s own [framework comparison](https://www.langchain.com/resources/langchain-vs-llamaindex) is a reasonable starting point if you want the maintainers’ framing directly. Neither is magic — both are orchestration layers over the same fundamental patterns above. Pick one, learn its concepts deeply, and don’t get religious about it: the architecture knowledge transfers regardless of which framework’s syntax you’re writing. If you’ve seen the “RAG is dead, just use a bigger context window” takes going around — that’s the next post. Short version: bigger context windows fix retrieval recall’s easier failure mode and do nothing for LLM recall’s harder one, and there’s a real difference between the two. Full breakdown next. After that: building safe LLM interfaces — guardrails, policy enforcement, and preventing unsafe or non-compliant outputs before they ship. *Originally published at **https://blog.kabirrajsingh.com** on August 30, 2026.* [Production-Ready RAG Architecture: Core Patterns Explained](https://pub.towardsai.net/production-ready-rag-architecture-core-patterns-explained-cda6f361b391) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.