# How I Built a Self-Hosted Family AI Health Steward (Your Health Data, on Your Shelf)

> Source: <https://dev.to/wang_zhengpeng_jay/how-i-built-a-self-hosted-family-ai-health-steward-your-health-data-on-your-shelf-59f8>
> Published: 2026-08-11 03:14:41+00:00

TL;DR— I built and open-sourcedAI Health Steward, a self-hosted, private AI health manager for families. It reads photos of lab reports with multimodal LLMs, builds a structured per-person health profile, shows trends on a dashboard, and answers health questions grounded inyour actual data— all running on your own server. Privacy isn't a feature; it's the whole point.[Star it on GitHub].

Every family has a shoebox — or a folder — of medical reports: blood tests, blood-pressure logs, prescriptions, scan findings. And every "convenient" health app wants to hold those records for you. But hold them *where*? On someone else's cloud, to be monetized, analyzed, or lost when the startup pivots.

Health records are the most sensitive data you own. They shouldn't be a product. They should live on **your** shelf.

So I built the opposite: a self-hosted AI health steward where the data never leaves your server.

```
┌──────────────┐     ┌──────────────────────────────────────┐
│   WebUI      │────▶│            FastAPI backend           │
│ React + Vite │     │  API / services / providers / prompts │
└──────────────┘     └───────────────┬──────────────────────┘
┌──────────────┐                     │
│ Feishu bot   │────▶  WebSocket ◀───┤
└──────────────┘                     ▼
                     ┌──────────────────────────────────────┐
                     │  PostgreSQL 16 + pgvector (RAG)      │
                     └──────────────────────────────────────┘
                     ┌──────────────────────────────────────┐
                     │  Model providers (pluggable)         │
                     │  OpenAI-compatible API / Ollama      │
                     └──────────────────────────────────────┘
```

`docker compose up -d`

FastAPI gives clean async handlers and Pydantic-validated schemas — perfect for the AI-agent-style tool-calling layer. React + Vite keeps the dashboard snappy. PostgreSQL + pgvector avoids a second vector database — one storage engine for structured data *and* embeddings keeps the deploy story simple for a home server.

**1. The "person-level profile" as a single source of truth.**

Every extracted value carries provenance — which report, which date, confirmed or not. The AI consultation layer reads from this profile rather than re-interpreting raw uploads every time. This is what makes answers *grounded* instead of hallucinated.

**2. Cost is a real concern for a home app.**

I aggressively cut LLM calls: no duplicate metric-extraction calls per conversation, and periodic summaries skip the LLM entirely when a period had no new data. A self-hosted app that costs pennies to run actually gets used.

**3. Structured output + human confirmation.**

AI extraction is powerful but not infallible. Every report goes through *extract → confirm → archive*, so garbage never silently enters the health record.

**4. Privacy as the default posture.**

Data stays local. Model calls go only to the provider *you* configure, and you can go fully offline with Ollama. Optional Bearer-token auth + per-member rate limiting protects the instance.

A full dashboard is powerful but heavy for "hey, is this blood pressure okay?" So there are **two entrances**:

Data flows between both automatically. Same profile, same single source of truth.

```
git clone https://github.com/wangzhengpengjay/AI-Health-Steward.git
cd AI-Health-Steward
cp .env.example .env        # set MULTIMODAL_API_KEY and TEXT_API_KEY
cp .env backend/.env
docker compose up -d
docker exec health-steward-backend alembic upgrade head
# WebUI: http://localhost:5173   |  API docs: http://localhost:8000/docs
```

A one-command demo-data seed makes it easy to explore before wiring up real accounts.

**If this resonates, give it a ⭐** — it helps other people who want their health data on their own shelf find it. And PRs and feature suggestions are genuinely welcome.

*AI Health Steward is an open-source project ( MIT). It is not a medical device, does not provide diagnoses, and is not a substitute for professional medical care.*