{"slug": "part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline", "title": "Part 1-3: From Raw Health Text to a Working Retrieval Pipeline", "summary": "A first-year undergraduate student in Artificial Intelligence Engineering is building a Retrieval-Augmented Generation (RAG) project from scratch, documenting the process in a series of blog posts. In the first three parts, the student implemented a pipeline that converts raw health text into structured data, chunks it, generates embeddings, and stores them in a Chroma vector store for retrieval. The project, called health-rag-assistant, is available on GitHub.", "body_md": "I'm a first-year undergraduate student in Artificial Intelligence Engineering, and this summer I'm building a small RAG (Retrieval-Augmented Generation) project from scratch to learn how these systems actually work under the hood — not just by calling an API, but by building the pipeline piece by piece. This is a log of the first three parts of that journey.\n\n##\nPart 1: Understanding the Building Blocks\n\nBefore writing any real project code, I spent time understanding the core ideas behind RAG:\n\n-\n**Calling an LLM API**: sending a prompt programmatically and getting a response back. I used the Gemini API for this. `llm_test.py`\n\n-\n**Embeddings**: the idea that text can be converted into numerical vectors, where semantically similar sentences end up close to each other in vector space. I tested this with a handful of sentences using `sentence-transformers`\n\nand computed cosine similarity between them — it was satisfying to see related sentences actually cluster together numerically. `embedding_test.py`\n\n-\n**Why not just dump everything into the LLM?**: context windows are limited, it's expensive, and irrelevant information can actually hurt answer quality rather than help it.\n-\n**Vector databases**: conceptually, tools like Chroma or FAISS solve the problem of searching through large numbers of embeddings quickly to find the most relevant ones.\nNo heavy coding in Part 1 — mostly small test scripts to confirm I understood each piece before combining them.\n\n##\nPart 2: Collecting and Preparing Real Data\n\nWith the concepts in place, Part 2 was about getting real data ready for retrieval:\n\n-\n**Data collection**: I gathered short health topic descriptions (e.g. diabetes, epilepsy) and saved them as structured JSON records, each with a `source`\n\nand `text`\n\nfield. Keeping the source attached to each record matters — it means the system can eventually point back to where an answer came from. `data/health_data.json`\n\n-\n**Chunking**: I wrote a script to split each record's text into smaller, paragraph-level pieces while keeping the original source attached to every chunk. Since I intentionally kept the raw text short for this first test run, most entries ended up as a single chunk each — which is fine for now, since the goal was to validate the pipeline, not build a large dataset yet.\n-\n**Why chunk at all?**: embedding models represent short, focused pieces of text more accurately than long documents. Splitting text into meaningful chunks is what makes retrieval actually useful later.\n\n##\nPart 3: Setting Up the Vector Store and First Retrieval\n\nWith chunked, source-tagged data ready, Part 3 turned that data into something actually searchable:\n\n-\n**Vector store setup**: installed and configured Chroma, then embedded every chunk from Part 2 and stored it alongside its text and source metadata. `embed_and_store.py`\n\n-\n**First retrieval test**: wrote a query script, asked a sample question, and retrieved the most similar chunks from the vector store. The results were reasonable given how small and short the dataset still is — a good early sign that the pipeline itself works end to end. `retrieval_test.py`\n\n-\n**A note on limitations**: since the dataset is intentionally tiny at this stage, retrieval quality is limited. That's expected, and it'll improve as the dataset grows in later parts.\n\n##\nWhat's Next\n\nAt this point I have a full (if small-scale) pipeline: raw text → structured data → chunks → embeddings → retrieval. The next step is connecting retrieval to actual answer generation — feeding retrieved chunks into the LLM as context so it can answer health questions grounded in real source material.\n\n👉 Code for this project: [health-rag-assistant](https://github.com/chewiessean/health-rag-assistant)\n\n*Follow along as I build this project part by part — code on GitHub, progress here on Dev.to.*", "url": "https://wpnews.pro/news/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline", "canonical_source": "https://dev.to/bedirhankizilcik/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline-n6b", "published_at": "2026-07-23 17:29:36+00:00", "updated_at": "2026-07-23 18:04:39.125782+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["Gemini", "Chroma", "sentence-transformers", "FAISS", "GitHub", "health-rag-assistant"], "alternates": {"html": "https://wpnews.pro/news/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline", "markdown": "https://wpnews.pro/news/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline.md", "text": "https://wpnews.pro/news/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline.txt", "jsonld": "https://wpnews.pro/news/part-1-3-from-raw-health-text-to-a-working-retrieval-pipeline.jsonld"}}