{"slug": "building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody", "title": "Building an Enterprise GenAI Platform on OCI — Part 2: The Data Pipeline Nobody Talks About", "summary": "A developer building an enterprise generative AI platform on Oracle Cloud Infrastructure detailed the data pipeline behind a RAG system, describing how they moved 70,000+ documents through a staged ingestion process on a resource-constrained OCI Compute instance. The engineer decoupled scraping from preprocessing by using OCI Object Storage as a durable persistence layer, streaming one document at a time to keep memory usage independent of dataset size. The writeup argues that RAG quality is determined by data lifecycle engineering well before embeddings or the LLM are involved.", "body_md": "The quality of a RAG system is decided long before the query reaches the LLM.\n\nI learned that the hard way.\n\nAfter designing the architecture in Part 1, my first instinct was to move straight to embeddings.\n\nAfter all, embeddings are where RAG starts getting interesting, right?\n\n**Not quite.**\n\nBefore generating a single vector, I had a much more basic problem to solve.\n\nBefore generating a single vector, I had a much more basic problem:\n\n**How do you reliably move 70,000+ documents through a pipeline running on constrained infrastructure? **\n\nMy OCI Compute instance wasn't exactly a powerhouse either.\n\nWhat initially looked like a scraping problem quickly became a **data-engineering problem.**\n\nAnd that's where Part 2 begins.\n\nThere is a tendency when building GenAI applications to start here:\n\n```\nDocument → Embedding → Vector DB → LLM\n```\n\nBut... **where did that document come from?**\n\nWhat happens when there are 70,000 of them?\n\nWhat if the process crashes halfway?\n\nWhat if you need to reprocess the data without scraping everything again?\n\nWhat if your embedding strategy changes tomorrow?\n\nSuddenly, this isn't an LLM problem.\n\nIt's a **data lifecycle problem.**\n\nSo I deliberately separated the pipeline into stages:\n\nEach stage produces an artifact that the next stage can consume.\n\nThat decision became much more important later.\n\nThe first component was straightforward.\n\nI needed technical knowledge for the DevOps assistant.\n\nSo I used an OCI Compute instance for scraping and data collection.\n\nBut I made an important decision early:\n\n**The compute instance should process data. It shouldn't become the permanent home of the data.**\n\nWhy?\n\nBecause:\n\nCompute is ephemeral.\n\nInstances can be stopped.\n\nDisks have limits.\n\nApplications change.\n\nPipelines fail.\n\nThe knowledge base needed to survive independently of the machine that created it.\n\nThat made **OCI Object Storage** the natural persistence layer.\n\nFor a prototype, I could have done this:\n\n`scraper/`\n\n├── data/\n\n│   ├── file1.json\n\n│   ├── file2.json\n\n│   ├── file3.json\n\n│   └── ...\n\nAnd initially, that feels simpler.\n\nBut now the data is tied to the Compute instance.\n\nWhat happens when preprocessing moves to OCI Data Science?\n\nI would need to:\n\nOr...\n\nI could make both services communicate through durable object storage.\n\nThat is exactly what I did.\n\nNow Compute and Data Science don't need to know anything about each other.\n\nThey only need to understand the **storage contract**.\n\nThat's **decoupling**.\n\nAnd this was one of the first architectural decisions that made the platform considerably easier to evolve.\n\nThere was another constraint.\n\nThe **Compute instance**, I was using had limited resources.\n\nLoading thousands of documents into memory before uploading them would have been unnecessary and risky.\n\nSo instead of thinking:\n\nI moved toward:\n\nOne object at a time.\n\nThis sounds like a small implementation detail.\n\nIt isn't.\n\nIt changes the memory profile of the ingestion pipeline.\n\nInstead of memory consumption increasing with dataset size, the worker only needs enough memory for the data currently being processed.\n\nConceptually:\n\n```\nfor document in documents:\n\n    content = scrape(document)\n\n    cleaned = basic_clean(content)\n\n    upload_to_object_storage(cleaned)\n\n    del content\n```\n\nThe actual implementation evolved, but the principle stayed the same:\n\nMove data through the pipeline instead of accumulating it inside the worker.\n\nInitially, I thought of Object Storage as:\n\n\"The place where I'll put my files.\"\n\nThat definition quickly became too simplistic.\n\nIt became the data backbone connecting the different stages of the platform.\n\nI organised the bucket roughly like this:\n\n`mlops-llm-data/`\n\n├── datasets/\n\n├── processed/\n\n├── features/\n\n├── models/\n\n└── logs/\n\nEach represented a different stage of the data lifecycle.\n\n| Prefix | Responsibility | \n|---|---|\n| `datasets/` | Raw ingested data | \n| `processed/` | Cleaned and chunked documents | \n| `features/` | Generated embedding artifacts | \n| `models/` | Model and retrieval artifacts | \n| `logs/` | Pipeline and operational logs | \n| `docker-build/` | Docker build-related artifacts | \n| `conda/` | Conda/environment-related artifacts | \n\nThese aren't traditional filesystem directories. But architecturally, they create clear boundaries between stages.\n\nThis became one of the most important rules in the pipeline:\n\nNever destroy the original dataset just because your processing logic changed.\n\nSuppose I scraped 70,000 documents.\n\nThen I cleaned them.\n\nA week later, I change my cleaning logic.\n\nIf I overwrote the originals, I'd have to scrape everything again.\n\nInstead: `datasets/` -> remained the source of truth.\n\nAnd: `processed/` -> contained derived data.\n\nThat meant I could rebuild the downstream pipeline without repeating ingestion.\n\nThis is essentially an **immutable raw-data pattern.**\n\nAnd it gave me something extremely valuable: **reproducibility.**\n\nNow the data existed. But an LLM retrieval pipeline doesn't necessarily want entire documents.\n\nImagine retrieving a 5,000-word article because the answer exists in three sentences somewhere in the middle.\n\nThat creates several problems:\n\nSo documents needed to be divided into smaller semantic units.\n\n**Chunks.**\n\nConceptually:\n\n`Document`\n\n│\n\n├── Chunk 1\n\n├── Chunk 2\n\n├── Chunk 3\n\n├── Chunk 4\n\n└── Chunk 5\n\nThose chunks would later become the units used for embedding and retrieval.\n\nBut chunking introduces its own engineering question.\n\n**How big should a chunk be?**\n\nMake chunks too large and retrieval becomes noisy.\n\nMake them too small and you destroy context.\n\nConsider:\n\nDocker containers package applications together with their dependencies, allowing them to run consistently across environments.\n\nA sensible chunk preserves that idea.\n\nBut an aggressive split could produce:\n\n`Chunk 1:\n\nDocker containers package applications together\n\nChunk 2:\n\nwith their dependencies, allowing them\n\nChunk 3:\n\nto run consistently across environments.`\n\nEach chunk now carries less meaning on its own. That's where **chunk overlap** helps.\n\nInstead of:\n\n`AAAA | BBBB | CCCC`\n\nwe can create:\n\nAAAA\n\n   AABBBB\n\n        BBBCCCC\n\nSome information is intentionally repeated across neighbouring chunks.\n\nThat gives the retriever a better chance of preserving concepts that happen to cross chunk boundaries.\n\nBut overlap isn't free.\n\nMore overlap means:\n\n`more chunks -> more embeddings -> larger index -> more storage -> more processing`\n\nThere is no universally perfect chunk size.\n\nIt depends on the documents, embedding model, retrieval strategy, and downstream context window.\n\nThis is a recurring theme in RAG:\n\nEvery retrieval optimisation has a cost somewhere else.\n\nEventually, preprocessing was producing roughly 70,000 chunks.\n\nAnd that's when another lesson became obvious:\n\nCode that works beautifully for: `100 documents` doesn't necessarily behave beautifully for: `70,000 documents`\n\nThe naive approach would be:\n\n```\nchunks = []\n\nfor file in all_files:\n    chunks.append(load(file))\n\nprocess(chunks)\n```\n\nbasically says:\n\n\"Load everything first. Worry about memory later.\"\n\nNot ideal.\n\nEspecially under constrained infrastructure. So the next architectural decision was obvious. **Batch processing.**\n\nRather than loading the entire dataset at once, I processed smaller groups:\n\n70,000 objects\n\n      ↓\n\n┌───────────────┐\n\n│ Batch 1       │\n\n│ 200 objects   │\n\n└───────────────┘\n\n      ↓\n\n   Process\n\n      ↓\n\n   Release\n\n      ↓\n\n┌───────────────┐\n\n│ Batch 2       │\n\n│ 200 objects   │\n\n└───────────────┘\n\n      ↓\n\n   Process\n\n      ↓\n\n   Release\n\n      ↓\n\n     ...\n\nThe exact batch size is tunable.\n\nThe principle is what matters:\n\nBound the amount of data being processed at any given moment.\n\nThis gives you predictable memory usage and makes larger datasets manageable on relatively modest infrastructure.\n\nThis was one of my favourite lessons from the entire pipeline.\n\nThe processed dataset contained roughly:\n\n**70,000 chunks**\n\nBut the next stage reported:\n\nTotal feature files: 1000\n\nTotal vectors in index: 999\n\nTraining vectors shape: (999, 384)\n\nWait.\n\n**\n\n70,000 chunks in.\n\n1,000 feature files out?**\n\nSomething was wrong.\n\nAnd here's the interesting part:\n\nIt wasn't FAISS.\n\nIt wasn't the embedding model.\n\nIt wasn't the AI.\n\n**The data pipeline was incomplete.**\n\nThe pipeline was unintentionally limiting the number of objects being listed from OCI Object Storage.\n\nA listing operation wasn't traversing the complete collection.\n\nThe result?\n\nNo crash.\n\nNo exception.\n\nJust incomplete data.\n\n**A pipeline can be technically successful and logically wrong.**\n\nThat's a much scarier failure mode than a simple application crash.\n\nCloud APIs commonly paginate large responses.\n\n`Request`\n\n   ↓\n\nObjects 1–1000\n\n   ↓\n\nNext Page Token\n\n   ↓\n\nObjects 1001–2000\n\n   ↓\n\nNext Page Token\n\n   ↓\n\n...\n\n   ↓\n\nAll Objects\n\nIf you forget pagination, your code can still run perfectly.\n\nNo obvious error.\n\nIt simply processes an incomplete dataset.\n\nAfter fixing pagination, the downstream stages could finally see the complete collection.\n\nThat changed how I thought about validation.\n\n**Counts Became a Data Quality Check**\n\nFrom that point onward, counts became a basic sanity check.\n\nAt every stage:\n\nHow many records entered?\n\nHow many succeeded?\n\nHow many failed?\n\nHow many were skipped?\n\nHow many artifacts were produced?\n\nIf ingestion produces:\n\n**70,000 documents ** but embedding generation sees: 1,000 chunks, something is clearly **wrong**.\n\nThis is a simple form of **data observability.**\n\nYou don't need a huge monitoring platform to start.\n\nSometimes a few carefully placed counters can save hours of debugging.\n\nAs processing became heavier, running everything on Compute became increasingly uncomfortable.\n\nI could have simply increased the Compute shape.\n\nInstead, I asked:\n\n**Does this workload actually belong on the same machine?**\n\nThe answer was no.\n\nScraping and preprocessing have different resource characteristics.\n\nSo heavier processing moved toward OCI Data Science notebook sessions.\n\nThe responsibilities became:\n\nCompute collected the data.\n\nObject Storage persisted it.\n\nData Science transformed it.\n\nAgain:\n\n**separation of concerns.**\n\nOnce multiple OCI services started communicating, authentication became part of the design.\n\nInstead of putting credentials inside configuration files, the notebook used **OCI Resource Principals.**\n\nOCI resource has an identity\n\n            +\n\nIAM defines permissions\n\nThis changes the model from:\n\nApplication possesses credentials\n\nto:\n\nCloud resource has an identity\n\n            +\n\nIAM controls what it can access\n\nThat's a much better foundation for cloud-native workloads.\n\nAnd it reinforces another principle: **Least Privilege Access.**\n\nRAG may sound like an AI problem.\n\nBut once the application touches Object Storage, Data Science, Model Deployment, or Container Registry, **identity becomes part of the AI architecture too.**\n\nAt this point, the data flow looked like this:\n\nNotice what's missing.\n\nThe LLM.\n\nAnd that's intentional.\n\nBefore generating a single response, we've already had to solve:\n\nThat's the point.\n\nBuilding RAG made me appreciate something that's easy to forget in the GenAI hype cycle:\n\nRAG is as much a data-engineering problem as it is an AI problem.\n\nThe LLM only sees what the retrieval pipeline gives it.\n\nThe retrieval pipeline only searches what was indexed.\n\nThe index only contains what was embedded.\n\nEmbeddings only represent what was processed.\n\nAnd processing can only operate on what ingestion successfully collected.\n\nSo the dependency chain is:\n\n`Data Quality -> Chunk Quality -> Embedding Quality -> Retrieval Quality -> Context Quality -> LLM Response Quality`\n\nA failure near the beginning propagates through everything downstream.\n\nA bigger model won't fix missing data.\n\nPrompt engineering won't fix a dataset accidentally truncated at 1,000 objects.\n\nAnd a re-ranker can't rank documents that never entered the index.\n\nThis stage changed how I approached the rest of the project.\n\n**1. Treat Object Storage as an architectural boundary**\n\n*It keeps ingestion, processing, and downstream workloads decoupled.*\n\n**2. Keep raw data immutable**\n\n*Processing strategies will change. Your original data shouldn't disappear with them.*\n\n**3. Design for bounded memory**\n\n*Batching becomes increasingly important as datasets grow.*\n\n**4. Never assume an API returned everything**\n\n*Pagination bugs can silently create incomplete ML datasets.*\n\n**5. Validate every stage**\n\n*Counts, failures, skips, and artifact counts are simple but powerful observability signals.*\n\n**6. Separate workloads by responsibility**\n\n*Scraping, preprocessing, embedding generation, and inference don't necessarily belong on the same infrastructure.*\n\n**7. Treat cloud identity as part of application architecture**\n\n*Resource Principals and IAM matter just as much as your Python code once services start communicating.*\n\nWe started with raw technical documents.\n\nWe now have cleaned, chunked, reproducible data sitting in Object Storage.\n\nRoughly **70,000 chunks** are waiting.\n\nBut there's one problem.\n\n**FAISS can't search text.**\n\nIt searches **vectors**.\n\nSo somehow this:\n\n\"How does Kubernetes service discovery work?\"\n\nneeds to become something like:\n\n[0.018, -0.042, 0.091, ..., 0.027]\n\nAnd documents discussing Kubernetes networking need to end up close to that query in vector space.\n\nThat's where things get considerably more interesting.\n\nBecause in ***Part 3***, we're going from:\n\nWords → Numbers → Meaning → Search\n\nWe'll look at:\n\n**Embeddings. Sentence Transformers. 384-dimensional vectors. Similarity search. FAISS. IVF indexes. Centroids. nlist. nprobe. **\n\nThat warning ended up teaching me more about vector search than simply getting the code to run ever could.\n\nIf you'd like to follow my work or connect, you can find me here:", "url": "https://wpnews.pro/news/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody", "canonical_source": "https://dev.to/yugandharsurya/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody-talks-about-a22", "published_at": "2026-09-10 19:54:00+00:00", "updated_at": "2026-09-10 20:11:54.476226+00:00", "lang": "en", "topics": ["generative-ai", "ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["Oracle Cloud Infrastructure", "OCI Compute", "OCI Object Storage", "OCI Data Science"], "alternates": {"html": "https://wpnews.pro/news/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody", "markdown": "https://wpnews.pro/news/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody.md", "text": "https://wpnews.pro/news/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody.txt", "jsonld": "https://wpnews.pro/news/building-an-enterprise-genai-platform-on-oci-part-2-the-data-pipeline-nobody.jsonld"}}