Talk to Your Medical History: Building a Personal EHR RAG with Milvus and Unstructured.io 🩺

We’ve all been there: digging through a mountain of crumpled hospital printouts, blurry scans, and nested PDFs just to find out what that specific blood test result was three years ago. Medical data is messy, unstructured, and—let's be honest—doctor's handwriting is the final boss of OCR.

In this tutorial, we are building a Personal Electronic Health Record (EHR) RAG system. We will transform those chaotic PDFs and scanned images into a searchable, intelligent knowledge base. By using a Vector Database like Milvus and powerful document partitioning, we'll achieve a seamless Personal Electronic Health Record experience where you can literally "talk" to your medical history. 🚀

Standard RAG (Retrieval-Augmented Generation) often fails on medical documents because:

Our solution combines Unstructured.io for "intelligent" PDF shredding, BGE Embeddings for high-precision medical semantics, and Milvus for industrial-grade vector storage.

Before we dive into the code, let's look at the data flow. We are moving from raw pixels to structured semantic insights.

graph TD
    A[Raw Medical PDFs/Scans] --> B{Unstructured.io}
    B -->|OCR & Partitioning| C[Clean Text Chunks]
    B -->|Table Extraction| D[Structured Data]
    C & D --> E[BGE Embeddings Model]
    E --> F[(Milvus Vector DB)]
    G[User Query: 'What was my glucose trend?'] --> H[Query Embedding]
    H --> I[Milvus Similarity Search]
    I --> J[LlamaIndex Context Synthesis]
    J --> K[LLM Response]

To follow along, you'll need:

pymilvus

, llama-index

, unstructured

, sentence-transformers

.Standard PDF s often break tables or ignore images. Unstructured.io treats a document like a collection of elements (Title, NarrativeText, Table).

from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
    filename="medical_report_2023.pdf",
    strategy="hi_res",           # Best for scanned documents
    extract_images_in_pdf=False,
    infer_table_structure=True,  # Keeps those lab results organized
    chunking_strategy="by_title",# Maintains semantic grouping
    max_characters=1000,
    combine_text_under_n_chars=200
)

from llama_index.core.schema import TextNode

nodes = []
for el in elements:
    nodes.append(TextNode(text=el.to_dict().get("text"), metadata=el.to_dict().get("metadata")))

For medical data, we need high-dimensional accuracy. BGE-M3 is currently a top-tier choice for retrieval. We'll store these in Milvus, which allows us to scale as our medical history grows over decades. 🥑

from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

vector_store = MilvusVectorStore(
    uri="http://localhost:19530", 
    collection_name="personal_ehr", 
    dim=1024  # BGE-Large dimension
)

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-large-en-v1.5")

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex(nodes, storage_context=storage_context, embed_model=embed_model)

Now we can ask complex questions across multiple documents.

query_engine = index.as_query_engine(similarity_top_k=5)

response = query_engine.query(
    "Compare my cholesterol levels between the 2021 checkup and the 2023 report. Is there an improving trend?"
)

print(f"Medical Assistant: {response}")

While this local setup is great for a weekend project, building a HIPAA-compliant or production-grade medical AI requires much deeper architectural considerations—specifically regarding data privacy and advanced reranking.

For more production-ready examples and advanced patterns on handling sensitive healthcare data within RAG architectures, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover the nuances of scaling vector search and ensuring data integrity that go beyond the basics of this tutorial.

You might ask: "Why not just use a simple local vector store?"

year > 2020

before doing the vector search, making queries lightning-fast.By combining Unstructured.io's ability to "see" documents with Milvus's ability to "remember" them, we've turned a pile of useless paper into a life-saving personal assistant. No more digging through drawers; just query and find.

Next Steps:

Happy coding, and stay healthy! 🩺💻