# Building an Enterprise GenAI Platform on OCI — Part 2: The Data Pipeline Nobody Talks About > Source: > Published: 2026-09-10 19:54:00+00:00 The quality of a RAG system is decided long before the query reaches the LLM. I learned that the hard way. After designing the architecture in Part 1, my first instinct was to move straight to embeddings. After all, embeddings are where RAG starts getting interesting, right? **Not quite.** Before generating a single vector, I had a much more basic problem to solve. Before generating a single vector, I had a much more basic problem: **How do you reliably move 70,000+ documents through a pipeline running on constrained infrastructure? ** My OCI Compute instance wasn't exactly a powerhouse either. What initially looked like a scraping problem quickly became a **data-engineering problem.** And that's where Part 2 begins. There is a tendency when building GenAI applications to start here: ``` Document → Embedding → Vector DB → LLM ``` But... **where did that document come from?** What happens when there are 70,000 of them? What if the process crashes halfway? What if you need to reprocess the data without scraping everything again? What if your embedding strategy changes tomorrow? Suddenly, this isn't an LLM problem. It's a **data lifecycle problem.** So I deliberately separated the pipeline into stages: Each stage produces an artifact that the next stage can consume. That decision became much more important later. The first component was straightforward. I needed technical knowledge for the DevOps assistant. So I used an OCI Compute instance for scraping and data collection. But I made an important decision early: **The compute instance should process data. It shouldn't become the permanent home of the data.** Why? Because: Compute is ephemeral. Instances can be stopped. Disks have limits. Applications change. Pipelines fail. The knowledge base needed to survive independently of the machine that created it. That made **OCI Object Storage** the natural persistence layer. For a prototype, I could have done this: `scraper/` ├── data/ │ ├── file1.json │ ├── file2.json │ ├── file3.json │ └── ... And initially, that feels simpler. But now the data is tied to the Compute instance. What happens when preprocessing moves to OCI Data Science? I would need to: Or... I could make both services communicate through durable object storage. That is exactly what I did. Now Compute and Data Science don't need to know anything about each other. They only need to understand the **storage contract**. That's **decoupling**. And this was one of the first architectural decisions that made the platform considerably easier to evolve. There was another constraint. The **Compute instance**, I was using had limited resources. Loading thousands of documents into memory before uploading them would have been unnecessary and risky. So instead of thinking: I moved toward: One object at a time. This sounds like a small implementation detail. It isn't. It changes the memory profile of the ingestion pipeline. Instead of memory consumption increasing with dataset size, the worker only needs enough memory for the data currently being processed. Conceptually: ``` for document in documents: content = scrape(document) cleaned = basic_clean(content) upload_to_object_storage(cleaned) del content ``` The actual implementation evolved, but the principle stayed the same: Move data through the pipeline instead of accumulating it inside the worker. Initially, I thought of Object Storage as: "The place where I'll put my files." That definition quickly became too simplistic. It became the data backbone connecting the different stages of the platform. I organised the bucket roughly like this: `mlops-llm-data/` ├── datasets/ ├── processed/ ├── features/ ├── models/ └── logs/ Each represented a different stage of the data lifecycle. | Prefix | Responsibility | |---|---| | `datasets/` | Raw ingested data | | `processed/` | Cleaned and chunked documents | | `features/` | Generated embedding artifacts | | `models/` | Model and retrieval artifacts | | `logs/` | Pipeline and operational logs | | `docker-build/` | Docker build-related artifacts | | `conda/` | Conda/environment-related artifacts | These aren't traditional filesystem directories. But architecturally, they create clear boundaries between stages. This became one of the most important rules in the pipeline: Never destroy the original dataset just because your processing logic changed. Suppose I scraped 70,000 documents. Then I cleaned them. A week later, I change my cleaning logic. If I overwrote the originals, I'd have to scrape everything again. Instead: `datasets/` -> remained the source of truth. And: `processed/` -> contained derived data. That meant I could rebuild the downstream pipeline without repeating ingestion. This is essentially an **immutable raw-data pattern.** And it gave me something extremely valuable: **reproducibility.** Now the data existed. But an LLM retrieval pipeline doesn't necessarily want entire documents. Imagine retrieving a 5,000-word article because the answer exists in three sentences somewhere in the middle. That creates several problems: So documents needed to be divided into smaller semantic units. **Chunks.** Conceptually: `Document` │ ├── Chunk 1 ├── Chunk 2 ├── Chunk 3 ├── Chunk 4 └── Chunk 5 Those chunks would later become the units used for embedding and retrieval. But chunking introduces its own engineering question. **How big should a chunk be?** Make chunks too large and retrieval becomes noisy. Make them too small and you destroy context. Consider: Docker containers package applications together with their dependencies, allowing them to run consistently across environments. A sensible chunk preserves that idea. But an aggressive split could produce: `Chunk 1: Docker containers package applications together Chunk 2: with their dependencies, allowing them Chunk 3: to run consistently across environments.` Each chunk now carries less meaning on its own. That's where **chunk overlap** helps. Instead of: `AAAA | BBBB | CCCC` we can create: AAAA AABBBB BBBCCCC Some information is intentionally repeated across neighbouring chunks. That gives the retriever a better chance of preserving concepts that happen to cross chunk boundaries. But overlap isn't free. More overlap means: `more chunks -> more embeddings -> larger index -> more storage -> more processing` There is no universally perfect chunk size. It depends on the documents, embedding model, retrieval strategy, and downstream context window. This is a recurring theme in RAG: Every retrieval optimisation has a cost somewhere else. Eventually, preprocessing was producing roughly 70,000 chunks. And that's when another lesson became obvious: Code that works beautifully for: `100 documents` doesn't necessarily behave beautifully for: `70,000 documents` The naive approach would be: ``` chunks = [] for file in all_files: chunks.append(load(file)) process(chunks) ``` basically says: "Load everything first. Worry about memory later." Not ideal. Especially under constrained infrastructure. So the next architectural decision was obvious. **Batch processing.** Rather than loading the entire dataset at once, I processed smaller groups: 70,000 objects ↓ ┌───────────────┐ │ Batch 1 │ │ 200 objects │ └───────────────┘ ↓ Process ↓ Release ↓ ┌───────────────┐ │ Batch 2 │ │ 200 objects │ └───────────────┘ ↓ Process ↓ Release ↓ ... The exact batch size is tunable. The principle is what matters: Bound the amount of data being processed at any given moment. This gives you predictable memory usage and makes larger datasets manageable on relatively modest infrastructure. This was one of my favourite lessons from the entire pipeline. The processed dataset contained roughly: **70,000 chunks** But the next stage reported: Total feature files: 1000 Total vectors in index: 999 Training vectors shape: (999, 384) Wait. ** 70,000 chunks in. 1,000 feature files out?** Something was wrong. And here's the interesting part: It wasn't FAISS. It wasn't the embedding model. It wasn't the AI. **The data pipeline was incomplete.** The pipeline was unintentionally limiting the number of objects being listed from OCI Object Storage. A listing operation wasn't traversing the complete collection. The result? No crash. No exception. Just incomplete data. **A pipeline can be technically successful and logically wrong.** That's a much scarier failure mode than a simple application crash. Cloud APIs commonly paginate large responses. `Request` ↓ Objects 1–1000 ↓ Next Page Token ↓ Objects 1001–2000 ↓ Next Page Token ↓ ... ↓ All Objects If you forget pagination, your code can still run perfectly. No obvious error. It simply processes an incomplete dataset. After fixing pagination, the downstream stages could finally see the complete collection. That changed how I thought about validation. **Counts Became a Data Quality Check** From that point onward, counts became a basic sanity check. At every stage: How many records entered? How many succeeded? How many failed? How many were skipped? How many artifacts were produced? If ingestion produces: **70,000 documents ** but embedding generation sees: 1,000 chunks, something is clearly **wrong**. This is a simple form of **data observability.** You don't need a huge monitoring platform to start. Sometimes a few carefully placed counters can save hours of debugging. As processing became heavier, running everything on Compute became increasingly uncomfortable. I could have simply increased the Compute shape. Instead, I asked: **Does this workload actually belong on the same machine?** The answer was no. Scraping and preprocessing have different resource characteristics. So heavier processing moved toward OCI Data Science notebook sessions. The responsibilities became: Compute collected the data. Object Storage persisted it. Data Science transformed it. Again: **separation of concerns.** Once multiple OCI services started communicating, authentication became part of the design. Instead of putting credentials inside configuration files, the notebook used **OCI Resource Principals.** OCI resource has an identity + IAM defines permissions This changes the model from: Application possesses credentials to: Cloud resource has an identity + IAM controls what it can access That's a much better foundation for cloud-native workloads. And it reinforces another principle: **Least Privilege Access.** RAG may sound like an AI problem. But once the application touches Object Storage, Data Science, Model Deployment, or Container Registry, **identity becomes part of the AI architecture too.** At this point, the data flow looked like this: Notice what's missing. The LLM. And that's intentional. Before generating a single response, we've already had to solve: That's the point. Building RAG made me appreciate something that's easy to forget in the GenAI hype cycle: RAG is as much a data-engineering problem as it is an AI problem. The LLM only sees what the retrieval pipeline gives it. The retrieval pipeline only searches what was indexed. The index only contains what was embedded. Embeddings only represent what was processed. And processing can only operate on what ingestion successfully collected. So the dependency chain is: `Data Quality -> Chunk Quality -> Embedding Quality -> Retrieval Quality -> Context Quality -> LLM Response Quality` A failure near the beginning propagates through everything downstream. A bigger model won't fix missing data. Prompt engineering won't fix a dataset accidentally truncated at 1,000 objects. And a re-ranker can't rank documents that never entered the index. This stage changed how I approached the rest of the project. **1. Treat Object Storage as an architectural boundary** *It keeps ingestion, processing, and downstream workloads decoupled.* **2. Keep raw data immutable** *Processing strategies will change. Your original data shouldn't disappear with them.* **3. Design for bounded memory** *Batching becomes increasingly important as datasets grow.* **4. Never assume an API returned everything** *Pagination bugs can silently create incomplete ML datasets.* **5. Validate every stage** *Counts, failures, skips, and artifact counts are simple but powerful observability signals.* **6. Separate workloads by responsibility** *Scraping, preprocessing, embedding generation, and inference don't necessarily belong on the same infrastructure.* **7. Treat cloud identity as part of application architecture** *Resource Principals and IAM matter just as much as your Python code once services start communicating.* We started with raw technical documents. We now have cleaned, chunked, reproducible data sitting in Object Storage. Roughly **70,000 chunks** are waiting. But there's one problem. **FAISS can't search text.** It searches **vectors**. So somehow this: "How does Kubernetes service discovery work?" needs to become something like: [0.018, -0.042, 0.091, ..., 0.027] And documents discussing Kubernetes networking need to end up close to that query in vector space. That's where things get considerably more interesting. Because in ***Part 3***, we're going from: Words → Numbers → Meaning → Search We'll look at: **Embeddings. Sentence Transformers. 384-dimensional vectors. Similarity search. FAISS. IVF indexes. Centroids. nlist. nprobe. ** That warning ended up teaching me more about vector search than simply getting the code to run ever could. If you'd like to follow my work or connect, you can find me here: