How I Built a Self-Hosted Family AI Health Steward (Your Health Data, on Your Shelf)

A developer has open-sourced AI Health Steward, a self-hosted AI health manager that reads photos of lab reports using multimodal LLMs, builds structured per-person health profiles, and answers health questions grounded in the user's own data. The system runs entirely on the user's server, with privacy as the core design principle, and supports pluggable model providers including local Ollama for offline use.

TL;DR— I built and open-sourcedAI Health Steward, a self-hosted, private AI health manager for families. It reads photos of lab reports with multimodal LLMs, builds a structured per-person health profile, shows trends on a dashboard, and answers health questions grounded inyour actual data— all running on your own server. Privacy isn't a feature; it's the whole point. Star it on GitHub . Every family has a shoebox — or a folder — of medical reports: blood tests, blood-pressure logs, prescriptions, scan findings. And every "convenient" health app wants to hold those records for you. But hold them where ? On someone else's cloud, to be monetized, analyzed, or lost when the startup pivots. Health records are the most sensitive data you own. They shouldn't be a product. They should live on your shelf. So I built the opposite: a self-hosted AI health steward where the data never leaves your server. ┌──────────────┐ ┌──────────────────────────────────────┐ │ WebUI │────▶│ FastAPI backend │ │ React + Vite │ │ API / services / providers / prompts │ └──────────────┘ └───────────────┬──────────────────────┘ ┌──────────────┐ │ │ Feishu bot │────▶ WebSocket ◀───┤ └──────────────┘ ▼ ┌──────────────────────────────────────┐ │ PostgreSQL 16 + pgvector RAG │ └──────────────────────────────────────┘ ┌──────────────────────────────────────┐ │ Model providers pluggable │ │ OpenAI-compatible API / Ollama │ └──────────────────────────────────────┘ docker compose up -d FastAPI gives clean async handlers and Pydantic-validated schemas — perfect for the AI-agent-style tool-calling layer. React + Vite keeps the dashboard snappy. PostgreSQL + pgvector avoids a second vector database — one storage engine for structured data and embeddings keeps the deploy story simple for a home server. 1. The "person-level profile" as a single source of truth. Every extracted value carries provenance — which report, which date, confirmed or not. The AI consultation layer reads from this profile rather than re-interpreting raw uploads every time. This is what makes answers grounded instead of hallucinated. 2. Cost is a real concern for a home app. I aggressively cut LLM calls: no duplicate metric-extraction calls per conversation, and periodic summaries skip the LLM entirely when a period had no new data. A self-hosted app that costs pennies to run actually gets used. 3. Structured output + human confirmation. AI extraction is powerful but not infallible. Every report goes through extract → confirm → archive , so garbage never silently enters the health record. 4. Privacy as the default posture. Data stays local. Model calls go only to the provider you configure, and you can go fully offline with Ollama. Optional Bearer-token auth + per-member rate limiting protects the instance. A full dashboard is powerful but heavy for "hey, is this blood pressure okay?" So there are two entrances : Data flows between both automatically. Same profile, same single source of truth. git clone https://github.com/wangzhengpengjay/AI-Health-Steward.git cd AI-Health-Steward cp .env.example .env set MULTIMODAL API KEY and TEXT API KEY cp .env backend/.env docker compose up -d docker exec health-steward-backend alembic upgrade head WebUI: http://localhost:5173 | API docs: http://localhost:8000/docs A one-command demo-data seed makes it easy to explore before wiring up real accounts. If this resonates, give it a ⭐ — it helps other people who want their health data on their own shelf find it. And PRs and feature suggestions are genuinely welcome. AI Health Steward is an open-source project MIT . It is not a medical device, does not provide diagnoses, and is not a substitute for professional medical care.