TL;DR— I built and open-sourcedAI Health Steward, a self-hosted, private AI health manager for families. It reads photos of lab reports with multimodal LLMs, builds a structured per-person health profile, shows trends on a dashboard, and answers health questions grounded inyour actual data— all running on your own server. Privacy isn't a feature; it's the whole point.[Star it on GitHub].
Every family has a shoebox — or a folder — of medical reports: blood tests, blood-pressure logs, prescriptions, scan findings. And every "convenient" health app wants to hold those records for you. But hold them where? On someone else's cloud, to be monetized, analyzed, or lost when the startup pivots.
Health records are the most sensitive data you own. They shouldn't be a product. They should live on your shelf.
So I built the opposite: a self-hosted AI health steward where the data never leaves your server.
┌──────────────┐ ┌──────────────────────────────────────┐
│ WebUI │────▶│ FastAPI backend │
│ React + Vite │ │ API / services / providers / prompts │
└──────────────┘ └───────────────┬──────────────────────┘
┌──────────────┐ │
│ Feishu bot │────▶ WebSocket ◀───┤
└──────────────┘ ▼
┌──────────────────────────────────────┐
│ PostgreSQL 16 + pgvector (RAG) │
└──────────────────────────────────────┘
┌──────────────────────────────────────┐
│ Model providers (pluggable) │
│ OpenAI-compatible API / Ollama │
└──────────────────────────────────────┘
docker compose up -d
FastAPI gives clean async handlers and Pydantic-validated schemas — perfect for the AI-agent-style tool-calling layer. React + Vite keeps the dashboard snappy. PostgreSQL + pgvector avoids a second vector database — one storage engine for structured data and embeddings keeps the deploy story simple for a home server.
1. The "person-level profile" as a single source of truth.
Every extracted value carries provenance — which report, which date, confirmed or not. The AI consultation layer reads from this profile rather than re-interpreting raw uploads every time. This is what makes answers grounded instead of hallucinated.
2. Cost is a real concern for a home app.
I aggressively cut LLM calls: no duplicate metric-extraction calls per conversation, and periodic summaries skip the LLM entirely when a period had no new data. A self-hosted app that costs pennies to run actually gets used.
3. Structured output + human confirmation.
AI extraction is powerful but not infallible. Every report goes through extract → confirm → archive, so garbage never silently enters the health record.
4. Privacy as the default posture.
Data stays local. Model calls go only to the provider you configure, and you can go fully offline with Ollama. Optional Bearer-token auth + per-member rate limiting protects the instance.
A full dashboard is powerful but heavy for "hey, is this blood pressure okay?" So there are two entrances:
Data flows between both automatically. Same profile, same single source of truth.
git clone https://github.com/wangzhengpengjay/AI-Health-Steward.git
cd AI-Health-Steward
cp .env.example .env # set MULTIMODAL_API_KEY and TEXT_API_KEY
cp .env backend/.env
docker compose up -d
docker exec health-steward-backend alembic upgrade head
A one-command demo-data seed makes it easy to explore before wiring up real accounts.
If this resonates, give it a ⭐ — it helps other people who want their health data on their own shelf find it. And PRs and feature suggestions are genuinely welcome.
AI Health Steward is an open-source project ( MIT). It is not a medical device, does not provide diagnoses, and is not a substitute for professional medical care.