Let’s be honest: doctors are famous for two things—saving lives and having handwriting that looks like a caffeinated spider crawled across a page. For Data Engineers in the healthcare space, this "unstructured data" is a nightmare.
Traditional OCR and Regex-based parsers often fail because medical notes aren't just messy; they are context-heavy. To achieve true healthcare interoperability, we need to transform these scribbles into the HL7 FHIR (Fast Healthcare Interoperability Resources) standard.
In this tutorial, we are diving into the "Dark Arts" of medical data cleaning. We'll leverage GPT-4o, Few-shot prompting, and JSON Schema to build a robust pipeline that turns raw clinical text into valid FHIR resources.
Before we write any code, we need a solid strategy. We aren't just "asking" the AI to format text; we are building a validation engine.
graph TD
A[Raw Unstructured Medical Note] --> B{GPT-4o Engine}
B -->|Few-Shot Context| C[Extraction Logic]
C --> D[JSON Schema Validation]
D -->|Valid| E[FHIR Patient/Observation Resource]
D -->|Invalid| F[Retry/Human-in-the-loop]
E --> G[(Clinical Data Repository)]
style B fill:#f9f,stroke:#333,stroke-width:2px
style E fill:#00ff00,stroke:#333
To follow along, you'll need:
openai, pydantic (for schema validation), and a basic understanding of FHIR is strict. If your JSON doesn't match the resource definition exactly, the receiving server will reject it. We’ll use Pydantic to define a simplified FHIR Observation resource to ensure our LLM output is production-ready.
from pydantic import BaseModel, Field
from typing import List, Optional
class FHIRCoding(BaseModel):
system: str
code: str
display: str
class FHIRQuantity(BaseModel):
value: float
unit: str
system: str = "http://unitsofmeasure.org"
code: str
class FHIRObservation(BaseModel):
resourceType: str = "Observation"
status: str
category: List[dict]
code: dict # Should contain a LOINC code
subject: dict # Reference to Patient
valueQuantity: Optional[FHIRQuantity] = None
note: Optional[List[dict]] = None
The secret to converting "Dark Data" is Few-shot prompting. By providing the LLM with 2-3 examples of "Messy Input -> Valid FHIR," we drastically reduce hallucinations.
SYSTEM_PROMPT = """
You are a specialized Medical Informatics Agent.
Your task is to parse unstructured clinical notes into valid HL7 FHIR R4 JSON.
Focus on extracting Vital Signs into 'Observation' resources.
Use LOINC codes for observations (e.g., 85354-9 for Blood Pressure).
"""
FEW_SHOT_EXAMPLES = """
Input: "Pt seen today. BP 120/80. HR 72. Complaining of mild headache."
Output: {
"resourceType": "Observation",
"status": "final",
"code": { "coding": [{"system": "http://loinc.org", "code": "85354-9", "display": "Blood Pressure"}] },
"component": [
{"code": {"text": "Systolic"}, "valueQuantity": {"value": 120, "unit": "mmHg"}},
{"code": {"text": "Diastolic"}, "valueQuantity": {"value": 80, "unit": "mmHg"}}
]
}
"""
OpenAI recently released Structured Outputs, which forces the model to adhere strictly to a provided JSON Schema. This is a game-changer for Data Engineering pipelines.
import openai
import json
client = openai.OpenAI()
def extract_medical_data(raw_text: str):
response = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-06",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Extract from this note: {raw_text}"}
],
response_format=FHIRObservation, # Enforcing the Pydantic schema!
)
return response.choices[0].message.parsed
raw_note = "Subject: John Doe. The patient presents with a fever of 102F and a rapid heart rate of 110bpm."
fhir_data = extract_medical_data(raw_note)
print(fhir_data.model_dump_json(indent=2))
In the world of Medical Data Engineering, accuracy isn't just a metric; it's a safety requirement. Using LLMs as a "reasoning bridge" between raw text and structured standards like FHIR allows us to unlock insights from millions of legacy records that were previously "dark" (unsearchable and unusable).
💡 Pro-Tip: For production-ready examples and advanced patterns on handling HIPAA-compliant data pipelines, check out the specialized guides over at the WellAlly Blog. They have fantastic resources on scaling medical AI without compromising on data integrity.
We’ve moved past the era of manual data entry. By combining the reasoning power of GPT-4o with the structural rigor of FHIR and Pydantic, we can automate the most tedious parts of healthcare tech.
What's next?
Are you working with medical data? What's the craziest "doctor handwriting" you've ever had to parse? Let me know in the comments! 👇