{"slug": "stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag", "title": "Stop Digging Through PDFs: Build a FHIR-Standard EHR Knowledge Base with RAG", "summary": "A developer built a Retrieval-Augmented Generation (RAG) pipeline to convert unstructured medical PDFs into a structured, queryable Electronic Health Record (EHR) knowledge base using FHIR standards. The system leverages Unstructured.io for document parsing, Milvus for vector storage, and DuckDB for structured data, enabling natural language queries and trend analysis. The approach addresses common challenges in healthcare data engineering, such as messy formats and siloed information.", "body_md": "We’ve all been there: staring at a stack of printed lab results or a folder full of cryptic `report_final_v2_NEW.pdf`\n\nfiles, trying to remember if our cholesterol was higher or lower two years ago. For developers, this isn't just a filing problem—it's a **data engineering** challenge.\n\nIn the world of healthcare, data is messy, siloed, and often locked in \"unstructured\" formats. To build a truly personal **Electronic Health Record (EHR)** system, we need more than just a folder; we need a **RAG (Retrieval-Augmented Generation)** pipeline that can parse PDFs, map them to the **FHIR (Fast Healthcare Interoperability Resources)** standard, and provide natural language insights.\n\nIn this guide, we’ll leverage **Unstructured.io**, **Milvus**, and **DuckDB** to turn chaotic medical PDFs into a queryable, structured knowledge base.\n\nBefore we dive into the code, let’s look at how the data flows from a messy lab report to a structured answer.\n\n``` php\ngraph TD\n    A[Unstructured PDF Reports] --> B[Unstructured.io Partitioning]\n    B --> C{Data Split}\n    C -->|Textual Context| D[Milvus Vector DB]\n    C -->|Tabular Data| E[DuckDB Structured Storage]\n    D --> F[LangChain RAG Engine]\n    E --> F\n    G[User Query: Is my glucose trending up?] --> F\n    F --> H[FHIR-Formatted Response]\n```\n\nMake sure you have your environment ready:\n\n```\npip install langchain milvus unstructured[pdf] duckdb openai\n```\n\nMedical PDFs often contain complex tables. Standard PDF parsers usually fail here. We’ll use `unstructured`\n\nto partition the document into logical elements.\n\n``` python\nfrom unstructured.partition.pdf import partition_pdf\n\n# Extract elements from a medical lab report\nelements = partition_pdf(\n    filename=\"lab_report_2023.pdf\",\n    infer_table_structure=True,\n    chunking_strategy=\"by_title\",\n    max_characters=1000,\n    new_after_n_chars=800,\n)\n\n# Separate tables from narrative text\ntables = [el for el in elements if el.category == \"Table\"]\ntexts = [el for el in elements if el.category == \"NarrativeText\"]\n\nprint(f\"Detected {len(tables)} tables and {len(texts)} text blocks.\")\n```\n\nTo perform a semantic search (e.g., \"Find all reports related to cardiovascular health\"), we need to store the text chunks in **Milvus**.\n\n``` python\nfrom langchain_community.vectorstores import Milvus\nfrom langchain_openai import OpenAIEmbeddings\n\nembeddings = OpenAIEmbeddings()\n\n# Initialize Milvus with our extracted text\nvector_db = Milvus.from_documents(\n    documents=texts,\n    embedding=embeddings,\n    connection_args={\"host\": \"127.0.0.1\", \"port\": \"19530\"},\n    collection_name=\"personal_ehr_knowledge\"\n)\n\n# Test a similarity search\ndocs = vector_db.similarity_search(\"How was my blood sugar in 2022?\")\n```\n\nVector search is great for context, but for **trends** (like tracking glucose over 5 years), we need structured data. We will map our extracted tables into a simplified FHIR \"Observation\" schema and store it in **DuckDB**.\n\n``` python\nimport duckdb\nimport pandas as pd\n\n# Mocking the mapping of a table element to a FHIR-like DataFrame\n# In a real scenario, use an LLM to parse the table.text into these columns\ndata = {\n    \"resourceType\": \"Observation\",\n    \"code\": \"4548-4\", \"display\": \"Hba1c\",\n    \"value\": 5.7, \"unit\": \"%\",\n    \"effectiveDateTime\": \"2023-10-15\"\n}\ndf = pd.DataFrame([data])\n\n# Connect to DuckDB and create a structured health table\ncon = duckdb.connect(\"health_records.db\")\ncon.execute(\"CREATE TABLE IF NOT EXISTS observations AS SELECT * FROM df\")\n\n# Query trends instantly\ntrend = con.execute(\"SELECT AVG(value) FROM observations WHERE display='Hba1c'\").fetchone()\nprint(f\"Average HbA1c: {trend[0]}%\")\n```\n\nBuilding a basic RAG is easy, but building a **production-ready healthcare agent** is hard. You need to handle HIPAA compliance, complex data lineage, and advanced prompt engineering to ensure the LLM doesn't hallucinate medical advice.\n\nFor deeper insights into building robust data pipelines and production-grade AI systems, I highly recommend checking out the ** WellAlly Tech Blog**. They have some incredible deep dives on advanced RAG patterns and handling highly sensitive unstructured data that helped me refine this architecture! 🥑\n\nFinally, we wrap everything into a LangChain retrieval sequence that uses both the Vector DB (for context) and DuckDB (for stats).\n\n``` python\nfrom langchain.chains import ConversationalRetrievalChain\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(model_name=\"gpt-4o\", temperature=0)\n\n# The RAG sequence\nqa_chain = ConversationalRetrievalChain.from_llm(\n    llm=llm,\n    retriever=vector_db.as_retriever(),\n    return_source_documents=True\n)\n\nquery = \"Compare my last two blood tests. Are there any concerning trends?\"\nresult = qa_chain.invoke({\"question\": query, \"chat_history\": []})\n\nprint(f\"Response: {result['answer']}\")\n```\n\nBy combining **Unstructured.io** for ingestion, **Milvus** for semantic memory, and **DuckDB** for analytical precision, we've moved beyond simple PDF storage. This system doesn't just \"read\" your records; it \"understands\" them within the context of the **FHIR standard**.\n\n**Next Steps:**\n\nWhat are you building with RAG lately? Have you tried parsing medical data? Let's discuss in the comments! 👇", "url": "https://wpnews.pro/news/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag", "canonical_source": "https://dev.to/beck_moulton/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag-29fa", "published_at": "2026-07-08 00:30:00+00:00", "updated_at": "2026-07-08 00:58:27.018699+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Unstructured.io", "Milvus", "DuckDB", "LangChain", "FHIR", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag", "markdown": "https://wpnews.pro/news/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag.md", "text": "https://wpnews.pro/news/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag.txt", "jsonld": "https://wpnews.pro/news/stop-digging-through-pdfs-build-a-fhir-standard-ehr-knowledge-base-with-rag.jsonld"}}