{"slug": "25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready", "title": "25 Million Records Later: Why Your RAG Pipeline Still Isn't Production-Ready", "summary": "A developer analyzed AWS's March 2024 RAG ingestion pipeline experiment, which processed a 25-million-record OSCAR subset using Ray on five g4dn.12xlarge instances with four NVIDIA T4 GPUs each, generating 768-dimensional embeddings with all-mpnet-base-v2 and loading them into OpenSearch (HNSW) and PostgreSQL with pgvector (IVF). The analysis concludes the experiment validates distributed embedding generation but does not address document versioning, idempotent retries, tenant isolation, or knowledge updates, and proposes an ingestion contract with tenant, version, and content-hash metadata plus deterministic chunk IDs and atomic cutover.", "body_md": "**25 million records. Distributed GPUs. Two vector databases. And still, the most difficult part of production RAG is not embedding generation.**\n\nA RAG prototype is easy to demo:\n\n```\nPDF → chunks → embeddings → vector DB → LLM → answer\n```\n\nBut production is where the questions change.\n\nWhat if a policy PDF is revised after it has been indexed? What if an ingestion worker retries the same batch? What if a user from Tenant A retrieves a chunk owned by Tenant B? What if the vector index returns a confident but obsolete paragraph?\n\nThose failures do not disappear by adding GPUs.\n\nI recently revisited AWS's article, [Build a RAG data ingestion pipeline for large-scale ML workloads](https://aws.amazon.com/blogs/big-data/build-a-rag-data-ingestion-pipeline-for-large-scale-ml-workloads/), published March 13, 2024. It is an interesting, substantial experiment. This article examines what it gets right, what it does *not* establish, and how I would extend the design for production.\n\nAWS demonstrates a batch ETL pipeline:\n\n```\nOSCAR JSONL corpus\n        ↓\nAWS Glue → Parquet on S3\n        ↓\nRay cluster on GPU-backed EC2\n        ↓\nChunking + all-mpnet-base-v2 embeddings\n        ↓\n       ┌────────────┴─────────────┐\n       ↓                          ↓\nOpenSearch (HNSW)        RDS PostgreSQL + pgvector (IVF)\n       └────────────┬─────────────┘\n                    ↓\n           Simulated retrieval queries\n```\n\nThe original OSCAR corpus is described as approximately 609 million records; AWS uses a **25-million-record subset** for the experiment. It reports a Ray setup of **five g4dn.12xlarge instances**, each with four NVIDIA T4 GPUs, and the **all-mpnet-base-v2** model producing **768-dimensional embeddings**. These are details from the AWS experiment, **not results from my own benchmark**.\n\nThe key idea is sound: if embedding generation is your bottleneck, parallelizing that workload is more useful than simply tuning the vector database.\n\nConverting large JSONL datasets to Parquet before distributed processing is a sensible design choice. Columnar formats can reduce I/O overhead and make batch reads more efficient.\n\nEmbedding generation can be GPU-bound. Ray makes it possible to distribute work rather than feeding a large corpus through one long-running Python process. It also exposes execution telemetry.\n\nOpenSearch, with HNSW indexing, and PostgreSQL with pgvector, using an IVF index in this experiment, represent different operational trade-offs.\n\nNeither is automatically 'better.'\n\nChoose based on data size, filtering requirements, query patterns, operational expertise, recall targets, costs, and workload isolation—not benchmark headlines alone.\n\nThis is **an ingestion and vector-store experiment**, not a complete architecture for document governance, secure multitenancy, and reliable knowledge updates. That distinction matters.\n\nLarge batch ingestion is one problem. Keeping the knowledge base correct on Tuesday after someone changes a document on Monday is another.\n\nA practical ingestion contract should include:\n\n```\n{\n  \"tenant_id\": \"tenant-123\",\n  \"document_id\": \"policy-456\",\n  \"source_version\": \"v7\",\n  \"content_hash\": \"sha256:...\",\n  \"embedding_model_version\": \"model-v2\",\n  \"ingestion_job_id\": \"job-789\"\n}\n```\n\nThen support **create, update, delete, reprocess, and rollback**.\n\nI would build immutable document versions and publish new chunk sets atomically, or with a controlled cutover. Otherwise, retrieval can mix old and new policy text.\n\nRetries happen. Batch workers crash. Network calls time out *after* the downstream index accepted the write.\n\nA job needs a stable idempotency key, deterministic chunk identifiers, persisted checkpoints, retry limits, and a dead-letter or quarantine path for poison documents.\n\nFor example:\n\n```\nchunk_id = hash(tenant_id, document_id, source_version, chunk_index, pipeline_version)\n```\n\nUse upserts where supported, and track when a version is fully indexed before making it queryable.\n\n**At-least-once delivery plus idempotent writes** is generally a much safer engineering target than pretending a distributed pipeline will execute each step exactly once.\n\nA vector's nearest neighbor is not necessarily an authorized neighbor.\n\nIn a multi-tenant system, filters must be enforced in trusted server-side retrieval logic and backed by access-control checks. Never trust a tenant ID supplied directly by the browser.\n\nAuthorization is more nuanced than a single tenant field: some documents can be department-restricted, user-specific, or revoked. Search-result filtering, index separation, and post-retrieval verification may all be required depending on the threat model.\n\nA fast result can be the wrong result.\n\nMeasure at least:\n\nAn ANN benchmark tells you something about vector search. It does not, by itself, tell you whether your RAG assistant gives trustworthy answers.\n\nThe 2024 AWS post discusses credential-expiry troubleshooting and a workaround involving long-lived IAM user keys. I would not carry that workaround into production.\n\nPrefer short-lived, role-based workload credentials, least-privilege policies, isolated network access, and audited operations. Legacy benchmark instructions should be reviewed against current security practices before deployment.\n\n```\nApproved sources (PDF, web, CMS, CRM, object storage)\n                        ↓\n              Source connector / CDC\n                        ↓\n        Raw object + immutable document version\n                        ↓\n       Durable job queue + idempotency ledger\n                        ↓\n      Parse / OCR / validate / malware scanning\n                        ↓\n        PII classification + ACL extraction\n                        ↓\n           Structure-aware chunking\n                        ↓\n          Versioned embedding workers\n                        ↓\n      Staging index + retrieval evaluation\n                        ↓\n          Atomic publish / index cutover\n                        ↓\n      Tenant- and role-scoped query service\n                        ↓\n       Retrieval + reranking + citations\n                        ↓\n           Guarded LLM response\n```\n\n**Keep three distinct concerns separate:**\n\nA vector database is a **derived index**, not your only copy of the business's knowledge.\n\nRay can be very appropriate when millions of documents or heavy batch embedding jobs justify distributed GPU scheduling.\n\nBut for a knowledge base with thousands of documents, starting with a five-node GPU cluster may be unnecessary complexity. A queue, a small worker pool, object storage, and pgvector may be a much better operational choice.\n\n**Scale from observed bottlenecks.** First measure parse throughput, embedding time, write throughput, index-build time, and p95 retrieval latency. Then optimize the expensive stage.\n\nMy decision framework:\n\n| Requirement | Often start with | \n|---|---|\n| Existing relational app, moderate corpus, transactional metadata | PostgreSQL + pgvector | \n| Strong text-search needs, specialized search operations, large search workloads | OpenSearch | \n| Strict tenant permissions | Either, with enforced authorization architecture | \n| Frequent updates and backfills | Benchmark both against your actual workload | \n| Lowest operational complexity | Whichever your team can operate reliably | \n\nThis is guidance, not a performance ranking. Index algorithm, filtering strategy, corpus distribution, and infrastructure can change the result substantially.\n\nBefore declaring the system production-ready, I would ask:\n\nIf the answer to these is 'not yet,' more embedding throughput will not solve the real reliability problem.\n\nAWS's Ray-based ingestion experiment is a valuable reference for the **scale mechanics** of RAG: bulk data preparation, parallel GPU embeddings, vector-store configuration, and ingestion observability.\n\nMy extension is simple:\n\n**A scalable RAG pipeline gets knowledge into an index. A production RAG platform ensures the right knowledge gets there, stays current, remains authorized, and can be recovered when things fail.**\n\nThat is the architectural shift worth making.\n\n**Reference:** [AWS Big Data Blog — Build a RAG data ingestion pipeline for large-scale ML workloads (March 13, 2024)](https://aws.amazon.com/blogs/big-data/build-a-rag-data-ingestion-pipeline-for-large-scale-ml-workloads/). AWS configurations and dataset counts cited above come from that article; the reliability and security recommendations are my analysis.\n\nI write about software architecture, AI, automation, and real-world engineering challenges.\n\nExplore more of my work and technical insights at [sanukhan.dev](https://sanukhan.dev/).", "url": "https://wpnews.pro/news/25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready", "canonical_source": "https://dev.to/sanukhandev/25-million-records-later-why-your-rag-pipeline-still-isnt-production-ready-32al", "published_at": "2026-10-28 05:39:33+00:00", "updated_at": "2026-10-28 05:41:17.272758+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["AWS", "Ray", "OpenSearch", "pgvector", "PostgreSQL", "OSCAR", "all-mpnet-base-v2", "NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready", "markdown": "https://wpnews.pro/news/25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready.md", "text": "https://wpnews.pro/news/25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready.txt", "jsonld": "https://wpnews.pro/news/25-million-records-later-why-your-rag-pipeline-still-isn-t-production-ready.jsonld"}}