{"slug": "building-a-robust-rag-pipeline-architecture-for-production", "title": "Building a Robust RAG Pipeline Architecture for Production", "summary": "A developer detailed a production-ready RAG pipeline architecture using LangChain, FastAPI, and Docker containers. The system ingests documents, generates embeddings with OpenAI's text-embedding-ada-002, stores them in Pinecone, and retrieves relevant chunks for LLM generation. Key optimizations include 500-character chunks with 20% overlap and scoping SQLAlchemy sessions to the request lifecycle to prevent connection leaks.", "body_md": "**Answer up front:** A RAG pipeline architecture is a set of connected services that ingest raw documents, turn them into embeddings, store them in a vector database, retrieve the most relevant chunks, and finally feed those chunks to a language model for generation. In practice you need a modular design, solid chunking, a fast vector store with hybrid search, and observability that lets you spot bottlenecks before they break your service.\n\nBelow I walk through each piece of that puzzle, share the code I run in production, and point out the trade-offs that kept me up at night.\n\nA RAG pipeline architecture typically consists of:\n\n`text-embedding-ada-002`\n\nor a local BERT) and produces dense vectors.\nIn my last project I ran each component as a small Docker container behind a Cloud Run service. The biggest pain point was the **session leak** in the SQLAlchemy layer that silently ate DB connections after a few thousand requests. I fixed it by scoping sessions to the request lifecycle – see my write-up on [FastAPI SQLAlchemy Session Leak Detection](https://www.logiclooptech.dev/fastapi-session-leak-detection-sqlalchemy-long-running/) for the exact steps.\n\nLangChain already gives you the building blocks – `DocumentLoaders`\n\n, `TextSplitters`\n\n, `Embedders`\n\n, and `Retrievers`\n\n. The trick is to keep the configuration external so you can swap out a component without rebuilding the whole service.\n\n```\n# config.yaml\npipeline:\n  loader: \"pdf\"\n  splitter:\n    type: \"RecursiveCharacter\"\n    chunk_size: 1000\n    chunk_overlap: 200\n  embedder:\n    provider: \"openai\"\n    model: \"text-embedding-ada-002\"\n  vector_store: \"pinecone\"\n  retriever:\n    top_k: 5\n    use_hybrid: true\npython\n# app/pipeline.py\nimport yaml\nfrom fastapi import Depends\nfrom langchain.document_loaders import PyPDFLoader\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain.vectorstores import Pinecone\nfrom langchain.chains import RetrievalQA\n\ndef load_config():\n    with open(\"config.yaml\") as f:\n        return yaml.safe_load(f)\n\ndef build_retriever(cfg=Depends(load_config)):\n    # 1. loader\n    loader = PyPDFLoader(cfg[\"pipeline\"][\"loader_path\"])\n    docs = loader.load()\n\n    # 2. splitter\n    splitter = RecursiveCharacterTextSplitter(\n        chunk_size=cfg[\"pipeline\"][\"splitter\"][\"chunk_size\"],\n        chunk_overlap=cfg[\"pipeline\"][\"splitter\"][\"chunk_overlap\"],\n    )\n    chunks = splitter.split_documents(docs)\n\n    # 3. embedder\n    embedder = OpenAIEmbeddings(model=cfg[\"pipeline\"][\"embedder\"][\"model\"])\n\n    # 4. vector store\n    vect = Pinecone.from_documents(\n        chunks,\n        embedder,\n        index_name=\"rag-index\",\n        namespace=\"prod\",\n    )\n\n    # 5. retriever\n    retriever = vect.as_retriever(\n        search_type=\"mmr\" if cfg[\"pipeline\"][\"retriever\"][\"use_hybrid\"] else \"similarity\",\n        search_kwargs={\"k\": cfg[\"pipeline\"][\"retriever\"][\"top_k\"]},\n    )\n    return retriever\n\ndef get_qa_chain(retriever=Depends(build_retriever)):\n    return RetrievalQA.from_chain_type(\n        llm=OpenAI(temperature=0),\n        chain_type=\"stuff\",\n        retriever=retriever,\n    )\n```\n\nThe FastAPI endpoint then just calls `qa_chain.run(query)`\n\n. Because every step is instantiated on demand, you can replace the `PyPDFLoader`\n\nwith a `SQLLoader`\n\nor swap Pinecone for a self-hosted Qdrant without touching the rest of the code.\n\nChunk size is a classic source of bugs. Too small and you lose context; too large and the embedding vector becomes a blurry average of unrelated sentences. In my experiments:\n\n| Chunk Size | Overlap | Retrieval Latency (ms) | Answer Quality |\n|---|---|---|---|\n| 200 | 50 | 120 | Misses multi-sentence facts |\n| 500 | 100 | 85 | Good balance |\n| 1000 | 200 | 70 | Best for long technical docs |\n\nI settled on 500-character chunks with 20 % overlap. The overlap ensures that a phrase split across a boundary still appears in at least one chunk.\n\nEmbedding choice matters too. OpenAI's `text-embedding-ada-002`\n\nis cheap and works well for English, but for multilingual corpora I switched to a `sentence-transformers`\n\nmodel hosted on a GPU-enabled inference server. The cost difference was stark: $0.0004 per 1k tokens vs $0.0012 for the transformer, but the recall improvement for non-Latin scripts was worth it.\n\n**When not to use a dense embedder:** If your corpus is under 10 k short snippets, a classic TF-IDF vectorizer can be faster and cheaper. You lose semantic matching, but for exact keyword queries the trade-off may be acceptable.\n\nVector stores differ on latency, scalability, and hybrid capabilities.\n\n| Store | Cloud-native | Hybrid (BM25 + ANN) | Cost (per GB/mo) | Known Pitfalls |\n|---|---|---|---|---|\n| Pinecone | Yes | ✅ | $0.30 | Cold start latency |\n| Qdrant | Self-hosted | ✅ (via `qdrant-client` ) |\n$0 (self) | Need to manage replicas |\n| Weaviate | Yes | ✅ (built-in) | $0.25 | Schema migrations can be tricky |\n| Milvus | Self-hosted | ❌ (requires external BM25) | $0 (self) | Complex config for large clusters |\n\nI prefer Pinecone for quick SaaS spin-up, but Qdrant gave me tighter control over latency when I moved the vector store into the same VPC as the FastAPI service. The hybrid query looks like this:\n\n``` python\n# hybrid_retriever.py\ndef hybrid_search(vector_store, query, top_k=5):\n    # 1. embed the query\n    query_vec = OpenAIEmbeddings().embed_query(query)\n\n    # 2. perform ANN search\n    ann_results = vector_store.query(\n        vector=query_vec,\n        top_k=top_k,\n        include_metadata=True,\n    )\n\n    # 3. BM25 fallback (if vector store supports it)\n    bm25_results = vector_store.hybrid_search(\n        query=query,\n        top_k=top_k,\n    )\n\n    # 4. merge and re‑rank\n    combined = ann_results + bm25_results\n    combined.sort(key=lambda r: r[\"score\"], reverse=True)\n    return combined[:top_k]\n```\n\nA common failure mode is **score mismatch**: the ANN scores are in the range `[0, 2]`\n\nwhile BM25 scores can be a hundred-fold larger. Normalizing scores before merging prevents the BM25 side from drowning out the semantic matches.\n\nTwo families of metrics matter:\n\n**Retrieval effectiveness** – measured with `Recall@k`\n\n, `Mean Reciprocal Rank (MRR)`\n\n, and `Precision@k`\n\n. I generate a test set of 200 queries with known ground-truth passages and run the pipeline nightly.\n\n**Generation quality** – `BLEU`\n\n, `ROUGE-L`\n\n, and more importantly **human-rated factuality**. I run a lightweight A/B test where half the traffic hits the new pipeline and the other half hits the previous version. A simple Google Form collects annotator scores.\n\nAutomated latency monitoring is a must. In production I instrument each FastAPI route with OpenTelemetry and push metrics to Cloud Monitoring. A typical latency breakdown looks like:\n\nIf any component spikes beyond its 95th percentile, an alert fires. I once saw a sudden 2× increase in embedding latency caused by a throttling rule on the OpenAI API key. The fix was to add exponential back-off and a fallback local embedder.\n\nEach stage lives in its own container:\n\n`ingestor`\n\n– FastAPI + Celery worker, pulls new docs every hour.`embedder`\n\n– tiny FastAPI service that only does `POST /embed`\n\n.`vector-store`\n\n– managed Pinecone or a self-hosted Qdrant cluster.`api-gateway`\n\n– the public FastAPI endpoint that runs retrieval and generation.I use **Google Cloud Run** for the stateless services because it auto-scales to zero and handles TLS out of the box. The vector store runs on a managed GKE node pool with **node-autoscaling** enabled.\n\nEvery push to `main`\n\ntriggers a GitHub Actions workflow that builds Docker images, runs unit tests, and pushes to Artifact Registry. Then a Cloud Run deployment rolls out the new revision. My CI/CD pipeline is described in detail in [Automating Production: A CI/CD Pipeline for Google Cloud Run with GitHub Actions](https://www.logiclooptech.dev/automating-production-a-cicd-pipeline-for-google-cloud-run-with-github-actions/).\n\n`temperature=0`\n\nand `max_tokens`\n\nlimits to keep per-request spend predictable.`/healthz`\n\nthat runs a quick vector store ping and a tiny embed test. Cloud Run will automatically replace unhealthy instances.**How many chunks should I store per document?**\n\nIt depends on document length and the chosen chunk size. For a 10 k word technical manual, 500-character chunks yield roughly 40–50 chunks, which balances retrieval relevance and storage cost.\n\n**Can I use a RAG pipeline without a vector store?**\n\nYes, if your use-case is tiny (under 1 k documents) and you only need keyword search, a relational DB with full-text indexes can suffice. You lose semantic matching, though.\n\n**What’s the simplest way to add hybrid search to an existing vector store?**\n\nIf your store lacks native hybrid capability, run a parallel SQLite FTS5 index on the same metadata and merge the results as shown in the `hybrid_search`\n\nfunction above.\n\n**Do I need a separate service for embeddings?**\n\nNot strictly. For low traffic you can embed inline in the ingestion step. For high-throughput pipelines, a dedicated embedder service isolates latency spikes and lets you cache results.\n\nThat’s the blueprint I follow for every RAG pipeline I ship to production. It’s not magical, but it’s reliable enough to keep my services up 99.9 % while staying within a predictable budget. Happy building!", "url": "https://wpnews.pro/news/building-a-robust-rag-pipeline-architecture-for-production", "canonical_source": "https://dev.to/ayush_kumar_085a0f2c54e3f/building-a-robust-rag-pipeline-architecture-for-production-3j25", "published_at": "2026-07-14 13:47:18+00:00", "updated_at": "2026-07-14 13:58:40.117100+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["LangChain", "FastAPI", "OpenAI", "Pinecone", "SQLAlchemy", "Docker", "Cloud Run", "Qdrant"], "alternates": {"html": "https://wpnews.pro/news/building-a-robust-rag-pipeline-architecture-for-production", "markdown": "https://wpnews.pro/news/building-a-robust-rag-pipeline-architecture-for-production.md", "text": "https://wpnews.pro/news/building-a-robust-rag-pipeline-architecture-for-production.txt", "jsonld": "https://wpnews.pro/news/building-a-robust-rag-pipeline-architecture-for-production.jsonld"}}