I Built a Privacy-First Health Record MCP Server That Runs Entirely on Your Machine

A developer built MedMemory, a privacy-first MCP server that stores personal health data in an AES-256 encrypted local SQLite database and uses Gemini Vision to extract structured information from medical documents. The server exposes eight tools to Claude Desktop, allowing users to query their health history in natural language without sending data to the cloud.

How I used the Model Context Protocol, Gemini Vision, and AES-256 encryption to let Claude answer questions about your personal health data — without sending that data anywhere. Every major health AI product has the same architecture: you upload your documents to their cloud, they process and store them, and you hope their privacy policy holds up. For general wellness tips, that's probably fine. For your actual medical records — prescriptions, lab results, discharge summaries — it's a meaningful trust ask. I wanted something different. I wanted Claude to be able to answer questions like "what medications am I on?" or "is it safe to add Ibuprofen to my current medications?" using my actual health records, without those records ever leaving my machine. The result is MedMemory — an MCP server that stores your health data in an AES-256 encrypted SQLite database locally, extracts structured information from medical documents using Gemini Vision, and exposes 8 tools to Claude Desktop so you can query your health history in natural language. The Model Context Protocol MCP is Anthropic's open standard for giving LLMs access to external tools and data sources. Instead of fine-tuning a model or building a RAG pipeline, you write a server that exposes typed tools — functions with names, descriptions, and input schemas — and the LLM decides when to call them based on the conversation. The key insight is that the tool docstrings are the intelligence layer . When you ask Claude "am I up to date on my vaccines?" , Claude doesn't know it has a vaccination tool until it reads the docstring: php @mcp.tool async def get vaccination status ctx: Context - dict: """Returns all recorded vaccinations and flags overdue or missing ones. Use this when the user asks about: - their vaccination history or immunisation records - whether they are up to date on vaccines - what vaccines they are missing or overdue for - travel vaccine recommendations """ That docstring is what makes Claude call the right tool at the right time. Writing good docstrings is the most important skill in MCP development. MedMemory has three layers: 1. Storage — SQLCipher encrypted SQLite All health data lives in a local SQLite database encrypted with SQLCipher AES-256 . The encryption key is set by the user on first run and never leaves the machine. If you open the database file in any text editor, you see random bytes — not readable SQL. bash $ xxd medmemory.db | head -3 00000000: e7ea 869f 47b1 bec2 8eb8 78f3 8fe9 57e7 ....G.....x...W. 00000010: 44c2 3e72 3056 bb14 ae19 1b53 5530 54e9 D. r0V.....SU0T. The migration from plain SQLite to SQLCipher was exactly 3 lines of code — import sqlcipher3 as sqlite3 , plus two PRAGMA lines in get connection . Everything else stayed identical. SQLCipher is a true drop-in replacement. 2. Ingestion — Gemini Vision for real-world documents Medical documents in the real world are messy — handwritten prescriptions, scanned PDFs, photos taken at angles. I started with Tesseract OCR but it failed completely on handwritten Indian prescriptions. The fix was bypassing OCR entirely for image-based documents and sending the image directly to Gemini Vision, which reads handwriting natively. The extraction pipeline now works like this: The extraction prompt includes explicit brand-to-generic drug name mappings and lab marker normalisation rules, plus a few-shot example showing exactly what output format is expected. This dramatically improved accuracy on real documents: "Glycomet" → "Metformin" "Storvas" → "Atorvastatin" "Gabantin-GRS" → "Gabapentin" "Zerodol-P" → split into "Aceclofenac" and "Paracetamol" "LDL Cholesterol" → "LDL" "S. Creatinine" → "creatinine" 3. MCP Tools — 8 tools covering the full health record | Tool | What it does | |---|---| ingest health document | Parse any medical PDF or image into structured DB records | get current medications | Return active medication list with doses and frequencies | get lab trend | Show historical readings for any lab marker | get visit history | Doctor visits with optional specialty filter | get vaccination status | Cross-reference DB against WHO adult schedule, flag gaps | check drug interaction | Query OpenFDA API against your current medication list | get allergies | Return recorded allergies | generate health summary | Synthesise all tables into a printable one-pager | check drug interaction is the tool I'm most proud of. It takes a newly prescribed drug name, pulls your current medications from the encrypted DB, queries the OpenFDA drug label API for the new drug's interaction data, and returns a structured result combining both. Claude Desktop conversation User: My doctor just prescribed Ibuprofen for back pain. Is it safe with my current medications? Claude: calls get current medications → Metformin, Atorvastatin, Aspirin calls check drug interaction "Ibuprofen" → FDA label data ⚠️ Important concern — Ibuprofen + Aspirin: Both are NSAIDs. Taking them together significantly increases the risk of stomach bleeding. The FDA specifically warns that taking multiple NSAIDs at the same time raises this risk. Metformin + Ibuprofen: Both can affect kidney function. Monitor kidney health if you proceed. Consider Acetaminophen as a safer alternative for back pain. A generic AI without your records would give the same NSAID warning to everyone. This one knows you're already on Aspirin and gives you a personalised, clinically relevant answer. That's the difference between a generic chatbot and a tool that actually knows your situation. Here's exactly what goes where: Stays on your machine: .env , never transmitted Goes to Gemini API during ingestion only : Goes to OpenFDA API: Goes to Anthropic same as any Claude conversation : You can verify the encryption yourself: xxd ~/medmemory.db | head -3 Should show random bytes, not "SQLite format 3" 1. Docstrings are the product. The quality of your tool docstrings determines whether Claude calls the right tool at the right time. I spent as much time on docstrings as on the actual tool logic. 2. Test on real documents early. I built the ingestion pipeline on clean synthetic PDFs and it looked great. When I tested on a real handwritten prescription from a neurologist in Ghaziabad, it extracted nothing. Real-world documents are fundamentally different from test data. 3. SQLCipher is genuinely a 3-line migration. I expected encrypting a SQLite database to be a multi-day project. It was import sqlcipher3 as sqlite3 plus two PRAGMA lines. The API is identical. The hardest part was getting the Python package to compile on Apple Silicon. 4. MCP is architecture, not a feature. The same DB helpers that power the MCP tools also power a FastAPI bridge that serves a Next.js companion UI. The DB layer is the business logic. MCP and FastAPI are just two different ways to expose it. 5. OpenFDA is free and powerful. The drug interaction data comes from the same FDA drug label database that pharmacists use. No API key required for low volume, and the data is authoritative. pip install medmemory-mcp medmemory-setup Then add to your Claude Desktop config: { "mcpServers": { "medmemory": { "command": "medmemory-server" } } } Or try the hosted demo synthetic patient data, no real health records : { "mcpServers": { "medmemory-demo": { "command": "npx", "args": "-y", "mcp-remote", "https://web-production-ba446.up.railway.app/sse" } } } Links: Built with Python, FastMCP, SQLCipher, Gemini Vision, OpenFDA, FastAPI, and Next.js. Questions or feedback welcome in the comments.