I Built My Own J.A.R.V.I.S. — A Local, Privacy-First Multi-Agent AI Assistant

I wanted a personal AI assistant. Not a chatbot. Not a copilot. Something that actually does things — controls my desktop, reads my screen, talks to my smart home, answers questions in Finnish or English, and never, ever sends my data to a server I don't control.

The cloud options are good. They're not mine.

So I started building one.

This post is the architecture document for that project. It's a work in progress — the docker-compose is ready, the Ollama instance is running, and the first agent is next. I'm writing this because the design decisions are interesting, and because someone else might want to build the same thing.

This isn't a Jarvis like Tony Stark's. It's closer to a privacy-respecting home automation brain — or what you'd get if you crossed a really good CLI assistant with a smart home hub and gave it eyes.

The core idea: a lightweight orchestrator (Ollama) decides what to do. Specialized agents handle how to do it. Each agent runs in its own Docker container. Everything stays on my local network.

What it does:

What it doesn't do: send my commands, my screen contents, or my voice to OpenAI, Google, or anyone else.

                      ┌─────────────────────────────────┐
                      │     Input / Output Layer         │
                      │  (Whisper STT · Piper TTS)     │
                      └──────────────┬──────────────────┘
                                     │
                                     v
                      ┌─────────────────────────────────┐
                      │        OLLAMA (The Brain)       │
                      │   qwen2.5:3b · Function Calling  │
                      │   Response time: < 200ms         │
                      └──────────────┬──────────────────┘
                                     │
         ┌────────────────────────────┼────────────────────────────┐
         │ HTTP/REST                  │ HTTP/REST                  │ HTTP/REST
         v                            v                            v
┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────────┐
│   System Agent      │  │   Vision Agent     │  │   Home Assistant Agent  │
│  (Linux / DBus)    │  │  (Screen capture)  │  │  (Smart home control)   │
│ · Volume / Brightness│  │ · maim / grim      │  │ · Lights, sensors       │
│ · Window management │  │ · Moondream2 VLM   │  │ · WebSocket API         │
│ · Process control   │  │ · Screen analysis  │  │ · Automations           │
└─────────────────────┘  └─────────────────────┘  └─────────────────────────┘

Each agent is a FastAPI service inside a Docker container. They don't know about each other. They just expose a clean REST interface and wait for instructions from Ollama.

Ollama runs small, capable LLMs locally. The key feature here is function calling (tool calling) — Ollama doesn't execute commands directly. It returns a structured JSON decision: "call the set_volume function with parameter level: 75".

This is cleaner than giving the LLM raw bash access. The function list is a strict allowlist — defined in JSON Schema. The model can only call what's explicitly permitted. No rm -rf /, no matter how creatively phrased.

The models I'm using are intentionally small:

All fit comfortably on a mid-range GPU or run on CPU. My Intel i5 desktop handles it fine.

Controls the running Linux session via wmctrl, xdotool, playerctl, and DBus. Things it can do:

pamixer) xbacklight or brightnessctl) gtk-launch or xdg-open) playerctl) No raw bash. Every function is explicitly defined.

Takes a screenshot (maim or grim) or reads a camera stream, then runs the image through a lightweight VLM (Moondream2). Can answer: "What window is in the foreground?", "Did that build succeed?", "What's on my second monitor right now?"

Talks to my Home Assistant instance over its WebSocket API. Lights, switches, climate, sensors — all accessible. I can say: "Turn off the office lights, but only if nobody's in there" and it checks the occupancy sensor before acting.

Pulls in external data when needed: weather forecasts, electricity prices (relevant in Finland), InfluxDB logs, or web searches via a headless browser.

Giving an AI system control over your desktop is a significant trust boundary. I'm implementing human-in-the-loop for sensitive operations:

For destructive or significant actions, the system doesn't just execute. It:

This isn't paranoid — it's sensible. The function allowlist prevents most problems at the model level. The confirmation prompt handles the remaining edge cases.

This is where it lives:

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: jarvis-ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  jarvis-core:
    build: ./core
    container_name: jarvis-core
    restart: unless-stopped
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - DEFAULT_MODEL=qwen2.5:3b
    ports:
      - "8000:8000"
    depends_on:
      - ollama

  agent-system:
    build: ./agents/system
    container_name: jarvis-agent-system
    restart: unless-stopped
    network_mode: host   # Needs access to host D-Bus / display

volumes:
  ollama_data:

For voice input, I'm using Faster-Whisper (or whisper.cpp for CPU-only) — push-to-talk on a hotkey. For output, Piper TTS — fast, local, and surprisingly natural Finnish voices available.

The pipeline: hotkey → Whisper → Ollama → function call or text → Piper TTS → speaker. Total latency target: under 1 second end-to-end.

Home Assistant has an Assist feature withwyoming-satellite and OpenAI-compatible endpoints. That's actually the longer-term plan — integrate as an Assist pipeline so it works with HA's voice push button.

But building the agents standalone first means they work independently of HA, and can be composed in other ways later. Modular by design.

qwen2.5:3b, test function calling with a Python script I'll post updates as the build progresses. The code will be on GitHub when it's worth sharing.

Building your own tools is the whole point of this site. If you're interested in self-hosted AI, the Ollama docs are a good starting point: ollama.com. And if you want something more polished out of the box, Home Assistant's Assist feature is worth exploring.