# 25 Million Records Later: Why Your RAG Pipeline Still Isn't Production-Ready > Source: > Published: 2026-10-28 05:39:33+00:00 **25 million records. Distributed GPUs. Two vector databases. And still, the most difficult part of production RAG is not embedding generation.** A RAG prototype is easy to demo: ``` PDF → chunks → embeddings → vector DB → LLM → answer ``` But production is where the questions change. What if a policy PDF is revised after it has been indexed? What if an ingestion worker retries the same batch? What if a user from Tenant A retrieves a chunk owned by Tenant B? What if the vector index returns a confident but obsolete paragraph? Those failures do not disappear by adding GPUs. I recently revisited AWS's article, [Build a RAG data ingestion pipeline for large-scale ML workloads](https://aws.amazon.com/blogs/big-data/build-a-rag-data-ingestion-pipeline-for-large-scale-ml-workloads/), published March 13, 2024. It is an interesting, substantial experiment. This article examines what it gets right, what it does *not* establish, and how I would extend the design for production. AWS demonstrates a batch ETL pipeline: ``` OSCAR JSONL corpus ↓ AWS Glue → Parquet on S3 ↓ Ray cluster on GPU-backed EC2 ↓ Chunking + all-mpnet-base-v2 embeddings ↓ ┌────────────┴─────────────┐ ↓ ↓ OpenSearch (HNSW) RDS PostgreSQL + pgvector (IVF) └────────────┬─────────────┘ ↓ Simulated retrieval queries ``` The original OSCAR corpus is described as approximately 609 million records; AWS uses a **25-million-record subset** for the experiment. It reports a Ray setup of **five g4dn.12xlarge instances**, each with four NVIDIA T4 GPUs, and the **all-mpnet-base-v2** model producing **768-dimensional embeddings**. These are details from the AWS experiment, **not results from my own benchmark**. The key idea is sound: if embedding generation is your bottleneck, parallelizing that workload is more useful than simply tuning the vector database. Converting large JSONL datasets to Parquet before distributed processing is a sensible design choice. Columnar formats can reduce I/O overhead and make batch reads more efficient. Embedding generation can be GPU-bound. Ray makes it possible to distribute work rather than feeding a large corpus through one long-running Python process. It also exposes execution telemetry. OpenSearch, with HNSW indexing, and PostgreSQL with pgvector, using an IVF index in this experiment, represent different operational trade-offs. Neither is automatically 'better.' Choose based on data size, filtering requirements, query patterns, operational expertise, recall targets, costs, and workload isolation—not benchmark headlines alone. This is **an ingestion and vector-store experiment**, not a complete architecture for document governance, secure multitenancy, and reliable knowledge updates. That distinction matters. Large batch ingestion is one problem. Keeping the knowledge base correct on Tuesday after someone changes a document on Monday is another. A practical ingestion contract should include: ``` { "tenant_id": "tenant-123", "document_id": "policy-456", "source_version": "v7", "content_hash": "sha256:...", "embedding_model_version": "model-v2", "ingestion_job_id": "job-789" } ``` Then support **create, update, delete, reprocess, and rollback**. I would build immutable document versions and publish new chunk sets atomically, or with a controlled cutover. Otherwise, retrieval can mix old and new policy text. Retries happen. Batch workers crash. Network calls time out *after* the downstream index accepted the write. A job needs a stable idempotency key, deterministic chunk identifiers, persisted checkpoints, retry limits, and a dead-letter or quarantine path for poison documents. For example: ``` chunk_id = hash(tenant_id, document_id, source_version, chunk_index, pipeline_version) ``` Use upserts where supported, and track when a version is fully indexed before making it queryable. **At-least-once delivery plus idempotent writes** is generally a much safer engineering target than pretending a distributed pipeline will execute each step exactly once. A vector's nearest neighbor is not necessarily an authorized neighbor. In a multi-tenant system, filters must be enforced in trusted server-side retrieval logic and backed by access-control checks. Never trust a tenant ID supplied directly by the browser. Authorization is more nuanced than a single tenant field: some documents can be department-restricted, user-specific, or revoked. Search-result filtering, index separation, and post-retrieval verification may all be required depending on the threat model. A fast result can be the wrong result. Measure at least: An ANN benchmark tells you something about vector search. It does not, by itself, tell you whether your RAG assistant gives trustworthy answers. The 2024 AWS post discusses credential-expiry troubleshooting and a workaround involving long-lived IAM user keys. I would not carry that workaround into production. Prefer short-lived, role-based workload credentials, least-privilege policies, isolated network access, and audited operations. Legacy benchmark instructions should be reviewed against current security practices before deployment. ``` Approved sources (PDF, web, CMS, CRM, object storage) ↓ Source connector / CDC ↓ Raw object + immutable document version ↓ Durable job queue + idempotency ledger ↓ Parse / OCR / validate / malware scanning ↓ PII classification + ACL extraction ↓ Structure-aware chunking ↓ Versioned embedding workers ↓ Staging index + retrieval evaluation ↓ Atomic publish / index cutover ↓ Tenant- and role-scoped query service ↓ Retrieval + reranking + citations ↓ Guarded LLM response ``` **Keep three distinct concerns separate:** A vector database is a **derived index**, not your only copy of the business's knowledge. Ray can be very appropriate when millions of documents or heavy batch embedding jobs justify distributed GPU scheduling. But for a knowledge base with thousands of documents, starting with a five-node GPU cluster may be unnecessary complexity. A queue, a small worker pool, object storage, and pgvector may be a much better operational choice. **Scale from observed bottlenecks.** First measure parse throughput, embedding time, write throughput, index-build time, and p95 retrieval latency. Then optimize the expensive stage. My decision framework: | Requirement | Often start with | |---|---| | Existing relational app, moderate corpus, transactional metadata | PostgreSQL + pgvector | | Strong text-search needs, specialized search operations, large search workloads | OpenSearch | | Strict tenant permissions | Either, with enforced authorization architecture | | Frequent updates and backfills | Benchmark both against your actual workload | | Lowest operational complexity | Whichever your team can operate reliably | This is guidance, not a performance ranking. Index algorithm, filtering strategy, corpus distribution, and infrastructure can change the result substantially. Before declaring the system production-ready, I would ask: If the answer to these is 'not yet,' more embedding throughput will not solve the real reliability problem. AWS's Ray-based ingestion experiment is a valuable reference for the **scale mechanics** of RAG: bulk data preparation, parallel GPU embeddings, vector-store configuration, and ingestion observability. My extension is simple: **A scalable RAG pipeline gets knowledge into an index. A production RAG platform ensures the right knowledge gets there, stays current, remains authorized, and can be recovered when things fail.** That is the architectural shift worth making. **Reference:** [AWS Big Data Blog — Build a RAG data ingestion pipeline for large-scale ML workloads (March 13, 2024)](https://aws.amazon.com/blogs/big-data/build-a-rag-data-ingestion-pipeline-for-large-scale-ml-workloads/). AWS configurations and dataset counts cited above come from that article; the reliability and security recommendations are my analysis. I write about software architecture, AI, automation, and real-world engineering challenges. Explore more of my work and technical insights at [sanukhan.dev](https://sanukhan.dev/).