{"slug": "i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant", "title": "I Built My Own J.A.R.V.I.S. — A Local, Privacy-First Multi-Agent AI Assistant", "summary": "A developer has published the architecture for a local, privacy-first multi-agent AI assistant that uses Ollama with small models (qwen2.5:3b, llama3.2:3b, phi3:mini, and Moondream2 for vision) as an orchestrator that dispatches structured function calls over REST to isolated FastAPI agents running in Docker containers for system control, screen vision, and Home Assistant smart-home integration. The design relies on a JSON Schema allowlist of functions rather than raw shell access, keeping all voice, screen, and command data on the local network via Whisper STT and Piper TTS.", "body_md": "I wanted a personal AI assistant. Not a chatbot. Not a copilot. Something that actually *does things* — controls my desktop, reads my screen, talks to my smart home, answers questions in Finnish or English, and never, ever sends my data to a server I don't control.\n\nThe cloud options are good. They're not *mine*.\n\nSo I started building one.\n\nThis post is the architecture document for that project. It's a work in progress — the docker-compose is ready, the Ollama instance is running, and the first agent is next. I'm writing this because the design decisions are interesting, and because someone else might want to build the same thing.\n\nThis isn't a Jarvis like Tony Stark's. It's closer to a privacy-respecting home automation brain — or what you'd get if you crossed a really good CLI assistant with a smart home hub and gave it eyes.\n\nThe core idea: a lightweight orchestrator (Ollama) decides *what to do*. Specialized agents handle *how to do it*. Each agent runs in its own Docker container. Everything stays on my local network.\n\nWhat it does:\n\nWhat it doesn't do: send my commands, my screen contents, or my voice to OpenAI, Google, or anyone else.\n\n```\n                      ┌─────────────────────────────────┐\n                      │     Input / Output Layer         │\n                      │  (Whisper STT · Piper TTS)     │\n                      └──────────────┬──────────────────┘\n                                     │\n                                     v\n                      ┌─────────────────────────────────┐\n                      │        OLLAMA (The Brain)       │\n                      │   qwen2.5:3b · Function Calling  │\n                      │   Response time: < 200ms         │\n                      └──────────────┬──────────────────┘\n                                     │\n         ┌────────────────────────────┼────────────────────────────┐\n         │ HTTP/REST                  │ HTTP/REST                  │ HTTP/REST\n         v                            v                            v\n┌─────────────────────┐  ┌─────────────────────┐  ┌─────────────────────────┐\n│   System Agent      │  │   Vision Agent     │  │   Home Assistant Agent  │\n│  (Linux / DBus)    │  │  (Screen capture)  │  │  (Smart home control)   │\n│ · Volume / Brightness│  │ · maim / grim      │  │ · Lights, sensors       │\n│ · Window management │  │ · Moondream2 VLM   │  │ · WebSocket API         │\n│ · Process control   │  │ · Screen analysis  │  │ · Automations           │\n└─────────────────────┘  └─────────────────────┘  └─────────────────────────┘\n```\n\nEach agent is a **FastAPI service** inside a Docker container. They don't know about each other. They just expose a clean REST interface and wait for instructions from Ollama.\n\nOllama runs small, capable LLMs locally. The key feature here is **function calling** (tool calling) — Ollama doesn't execute commands directly. It returns a structured JSON decision: *\"call the `set_volume` function with parameter `level: 75`\"*.\n\nThis is cleaner than giving the LLM raw bash access. The function list is a strict allowlist — defined in JSON Schema. The model can only call what's explicitly permitted. No `rm -rf /`, no matter how creatively phrased.\n\nThe models I'm using are intentionally small:\n\n| Model | Size | Why | \n|---|---|---|\n| `qwen2.5:3b` | ~2GB | Main orchestrator. Fast, good Finnish support | \n| `llama3.2:3b` | ~2GB | Fallback / general reasoning | \n| `phi3:mini` | ~2GB | Lightweight tasks | \n| *Moondream2* | ~1GB | Vision agent — screen capture analysis | \n\nAll fit comfortably on a mid-range GPU or run on CPU. My Intel i5 desktop handles it fine.\n\nControls the running Linux session via `wmctrl`, `xdotool`, `playerctl`, and DBus. Things it can do:\n\n`pamixer`)` xbacklight` or `brightnessctl`)` gtk-launch` or `xdg-open`)` playerctl`)\nNo raw bash. Every function is explicitly defined.\n\nTakes a screenshot (`maim` or `grim`) or reads a camera stream, then runs the image through a lightweight VLM (Moondream2). Can answer: *\"What window is in the foreground?\"*, *\"Did that build succeed?\"*, *\"What's on my second monitor right now?\"*\n\nTalks to my Home Assistant instance over its WebSocket API. Lights, switches, climate, sensors — all accessible. I can say: *\"Turn off the office lights, but only if nobody's in there\"* and it checks the occupancy sensor before acting.\n\nPulls in external data when needed: weather forecasts, electricity prices (relevant in Finland), InfluxDB logs, or web searches via a headless browser.\n\nGiving an AI system control over your desktop is a significant trust boundary. I'm implementing **human-in-the-loop** for sensitive operations:\n\nFor destructive or significant actions, the system doesn't just execute. It:\n\nThis isn't paranoid — it's sensible. The function allowlist prevents most problems at the model level. The confirmation prompt handles the remaining edge cases.\n\nThis is where it lives:\n\n```\nversion: '3.8'\n\nservices:\n  # The Brain\n  ollama:\n    image: ollama/ollama:latest\n    container_name: jarvis-ollama\n    restart: unless-stopped\n    ports:\n      - \"11434:11434\"\n    volumes:\n      - ollama_data:/root/.ollama\n    # GPU passthrough if you have one:\n    # deploy:\n    #   resources:\n    #     reservations:\n    #       devices:\n    #         - driver: nvidia\n    #           count: all\n    #           capabilities: [gpu]\n\n  # The Orchestrator\n  jarvis-core:\n    build: ./core\n    container_name: jarvis-core\n    restart: unless-stopped\n    environment:\n      - OLLAMA_BASE_URL=http://ollama:11434\n      - DEFAULT_MODEL=qwen2.5:3b\n    ports:\n      - \"8000:8000\"\n    depends_on:\n      - ollama\n\n  # Linux System Agent\n  agent-system:\n    build: ./agents/system\n    container_name: jarvis-agent-system\n    restart: unless-stopped\n    network_mode: host   # Needs access to host D-Bus / display\n\nvolumes:\n  ollama_data:\n```\n\nFor voice input, I'm using **Faster-Whisper** (or `whisper.cpp` for CPU-only) — push-to-talk on a hotkey. For output, **Piper TTS** — fast, local, and surprisingly natural Finnish voices available.\n\nThe pipeline: hotkey → Whisper → Ollama → function call or text → Piper TTS → speaker. Total latency target: under 1 second end-to-end.\n\nHome Assistant has an Assist feature withwyoming-satellite and OpenAI-compatible endpoints. That's actually the longer-term plan — integrate as an Assist pipeline so it works with HA's voice push button.\n\nBut building the agents standalone first means they work independently of HA, and can be composed in other ways later. Modular by design.\n\n`qwen2.5:3b`, test function calling with a Python script\nI'll post updates as the build progresses. The code will be on GitHub when it's worth sharing.\n\n*Building your own tools is the whole point of this site. If you're interested in self-hosted AI, the Ollama docs are a good starting point: [ollama.com](https://ollama.com). And if you want something more polished out of the box, Home Assistant's Assist feature is worth exploring.*", "url": "https://wpnews.pro/news/i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant", "canonical_source": "https://dev.to/jronegit/i-built-my-own-jarvis-a-local-privacy-first-multi-agent-ai-assistant-276e", "published_at": "2026-11-14 15:14:37+00:00", "updated_at": "2026-11-14 15:20:55.485433+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "artificial-intelligence"], "entities": ["Ollama", "Home Assistant", "Qwen", "Llama", "Phi-3", "Moondream2", "FastAPI", "Docker"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant", "markdown": "https://wpnews.pro/news/i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant.md", "text": "https://wpnews.pro/news/i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant.txt", "jsonld": "https://wpnews.pro/news/i-built-my-own-j-a-r-v-i-s-a-local-privacy-first-multi-agent-ai-assistant.jsonld"}}