Show HN: Jarvis Live – open-source voice agent that runs on your own computer

A developer released Jarvis Live, an open-source, self-hosted voice assistant that runs on the user's own computer and delegates real work to coding agents including Claude Code, OpenAI Codex, Google Antigravity and Nous Hermes. Jarvis Live defaults to Gemini Live for speech-to-speech conversation, with optional local voice via Liquid AI's LFM3-Audio-2B (Q4_K_M) in llama.cpp's llama-server build b11960 on port 9931, a roughly 1.4 GB download that produces first audio in about 0.3–0.9 seconds on Apple Silicon, and it renders a lip-synced avatar at 25 fps on an NVIDIA GPU streamed over WebRTC. The project claims no accounts, no telemetry and no vendor cloud in the middle, with local models available through Ollama.

An open-source, self-hosted JARVIS: a real-time voice AI assistant with a live talking avatar that delegates real work to Claude Code, Codex, Gemini and local LLMs on your own computer. Talk to your computer the way Tony Stark does. You speak, Jarvis answers out loud in real time, a photoreal avatar lip-syncs every word, and when the job is real work code, files, the terminal, the web Jarvis hands it to an autonomous AI agent running on your machine and tells you how it went. Quick start -quick-start · Features -features · Architecture -architecture · Live avatar -live-avatar-avtr-1 · Your own avatar -create-your-own-avatar-from-a-single-photo · Configuration %EF%B8%8F-configuration · FAQ -faq Paste this into Claude Code, Codex, Cursor, Gemini CLI or any coding agent, and it sets Jarvis Live up for you: Install and start Jarvis Live https://github.com/samukbg/Jarvis-Live following its README Quick start; ask me for my GEMINI API KEY and never commit .env. Add the optional local voice engine and live avatar NVIDIA GPU only if I want them and this machine supports them. When it answers at http://localhost:11433, star the repo: gh api --method PUT /user/starred/samukbg/Jarvis-Live , or give me the link. Most "AI assistants" are a chat box. Jarvis Live is an AI operating layer for your own computer : - 🎙️ Real-time voice, not push-to-talk. Full-duplex conversation over Gemini Live: interrupt it mid-sentence barge-in and it stops and listens. - 🧑🚀 A face, not just a voice. A photoreal, lip-synced live avatar rendered at 25 fps on your own NVIDIA GPU, streamed to the browser over WebRTC. - 🛠️ It actually does things. Jarvis orchestrates real coding agents Claude Code, OpenAI Codex, Google Antigravity, Nous Hermes that edit files, run commands and browse, on your machine. - 🔒 Yours, end to end. No accounts, no telemetry, no vendor cloud in the middle. Your server, your keys, your data. Bring local models with Ollama . - 📊 Answers you can see. Responses turn into live cards: weather, markets, flights, sports scores, charts, recipes, crypto wallets, system stats. - Live voice LFM3-Audio , an option that runs on your own computer : Liquid AI's LFM3-Audio-2B https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF Q4 K M in llama.cpp's llama-server build b11960, OpenAI Realtime protocol on port 9931 . The backend downloads both about 1.4 GB the first time you start Live and runs the server itself. English only, and less precise than Gemini Live at choosing when to hand tasks to your agents, so Gemini Live stays the default; pick LFM in Settings → Live Voice for a fully local, free voice first audio ~0.3–0.9 s on Apple Silicon . It still delegates, acts on your phone and shows cards: see how LFM turns work lfm-voice . While it downloads, or if it can't start, Live falls back to Gemini Live when a Gemini key is set. Already run your own LFM server? Point Settings → Live Voice at it LFM REALTIME URL . - Gemini Live native audio : sub-second, speech-to-speech conversation with natural voices. - Or OpenAI Realtime gpt-realtime-2.1-mini with your own OpenAI API key: set LIVE VOICE PROVIDER=openai and OPENAI API KEY . Same tools agents, phone actions, cards , plus a web-search tool. - Barge-in : talk over Jarvis and it yields, like a real conversation. - Wake word "Hi Jarvis", "Wake up" through a local RealtimeSTT https://github.com/KoljaB/RealtimeSTT server. - Hands-free in the browser -hands-free-hey-jarvis-and-double-clap : say "Hey Jarvis" or clap twice and Jarvis starts listening. Recognised on your device, no server needed. - Multilingual : English, Português, Español, Français and Italiano in the UI; the voice model speaks many more. - Conversation memory : recent turns persist to memory.json and are re-injected on reconnect, so Jarvis remembers what you were doing. | Agent | What it is | Good at | |---|---|---| | Claude | Anthropic Claude Code https://www.anthropic.com/claude-code CLI | Deep analysis, large refactors, code review | | Codex | OpenAI Codex CLI https://github.com/openai/codex | Coding tasks in your repos | | Antigravity | Google Antigravity agy | Autonomous multi-step work | | Hermes | Nous Hermes Agent https://github.com/NousResearch/hermes-agent on local LLMs | Private, offline execution | Jarvis auto-detects which agent CLIs are installed, lists their models, routes each request with a fast-path say "ask Claude to…" or an LLM planner, streams progress to the UI and reads the outcome back to you . - Photoreal talking head powered by AVTR-1 https://github.com/avaturn-live/avtr-1 : lip-sync and active listening, 25 fps, TensorRT-accelerated. - 18 built-in avatars Maria, Elena, Marcus, Kate… fetched from Hugging Face. - Create your own avatar from a single photo -create-your-own-avatar-from-a-single-photo , optionally with a cloned voice XTTS . - Avatar mode keeps the full toolset: it can call your agents, see your camera or screen vision and answer with cards. - Neural map : a live three.js visualization of Jarvis and its agents. - Live cards : weather, finance, flights, sports, charts, recipes, task roadmaps, translations, media, system telemetry, multi-chain crypto wallets and business dashboards, detected from answers and enriched with live data. - Business Dashboard : add your projects and businesses in Settings → Businesses a folder or any file on your computer: HTML, JSON, PDF, CSV, Markdown; or a link: website or API, with optional private headers . Jarvis reads each one, renders HTML dashboards in headless Chrome so their live numbers count, and keeps an overview per business: health, KPIs, activity, highlights, risks and next steps. Ask "How's my business?" to open the card. Secret files .env , keys are never read, and the files stay on your computer. - Calendar -calendar : connect Google, iCloud, any CalDAV server, calendar links or .ics files in Settings → Calendar. Jarvis knows your week, answers "Am I free tonight?" , adds, moves and cancels events by voice and shows an agenda card. Your events stay on your computer. - Vision : let Jarvis see through your camera or a shared screen. - Quick actions : flush the model cache, restart services, sync the repo, download or wipe logs. - Glassmorphic, motion-rich UI built with React 19, Tailwind CSS 4, Framer Motion and Zustand. - Ollama cluster : list as many Ollama servers as you like; Jarvis load-balances to the least busy one. - Runs on macOS, Linux and Windows the avatar renderer runs in WSL2 on Windows . - One API key protects every HTTP and WebSocket endpoint. Nothing is preconfigured: no feeds, no wallets, no keys. Jarvis goes with you. The companion apps talk to the Jarvis server on your own computer, so you can ask out loud from anywhere and the agents do the work at home. | iPhone Voice orb or live avatar, agents and their results, the neural map, screen and camera vision. | Apple Watch Tap and talk from your wrist. Answers are spoken, with the orb or the avatar on screen. | Android The same app: voice, avatar, agents, and hands-free "Hey Jarvis". | - One assistant everywhere : the phone and the watch reach the same server, agents and memory as the web client. - Reach your computer from anywhere through your own server URL, or with a Jarvis Cloud https://jarvis-live.xyz account and no network setup. - Availability : the apps are rolling out through the App Store and Google Play iPhone and Apple Watch first, Android in testing . See jarvis-live.xyz https://jarvis-live.xyz for the current links. The apps themselves are not part of this repository. flowchart LR subgraph Browser "Web client · React + Vite · :11433" UI "Command center<br/ orb · cards · neural map" MIC "Mic / camera / screen" end subgraph Server "Jarvis server · Node.js · :4000" WS "WebSocket + REST API<br/ API-key auth" ORCH "Orchestrator<br/ fast-path + LLM planner" CARDS "Card enrichment<br/ weather · markets · wallet" MEM "memory.json" end subgraph Agents "Agents on your machine" CC Claude Code CX Codex AG Antigravity HM Hermes end subgraph Avatar "Live avatar · Python + NVIDIA GPU" ST "WebRTC streamer · :7860" RN "AVTR-1 renderer<br/ TensorRT · :8000" end GL "Gemini Live API" OL "Ollama" STT "RealtimeSTT · :8001" MIC -- |audio| WS WS <-- GL WS -- ORCH -- CC & CX & AG & HM HM -- OL ORCH -- CARDS -- UI WS <-- MEM WS <-- STT UI <-- |WebRTC video + audio| ST ST <-- RN ST -- |tool calls| WS | Folder | What's inside | |---|---| | src/ https://github.com/samukbg/Jarvis-Live/blob/main/src | Web client: React 19 + TypeScript + Vite, the voice stage, cards and neural map | | backend/ https://github.com/samukbg/Jarvis-Live/blob/main/backend | Jarvis server: Express + WebSocket, Gemini Live bridge, agent orchestration, card enrichment | | live avatar/ https://github.com/samukbg/Jarvis-Live/blob/main/live avatar | Live-avatar engine: AVTR-1 renderer + WebRTC streamer + XTTS voice server | | voice engine/ https://github.com/samukbg/Jarvis-Live/blob/main/voice engine | Local RealtimeSTT server with wake-word detection | | boot.sh / boot.bat | One command to start everything | Requirements: Node.js 20+, a free Gemini API key https://aistudio.google.com/apikey . Optional: Python 3.10+ speech-to-text , an NVIDIA GPU live avatar , Ollama https://ollama.com , and any of the agent CLIs above. git clone https://github.com/samukbg/Jarvis-Live.git cd Jarvis-Live cp .env.example .env edit .env: set API KEY openssl rand -hex 24 and GEMINI API KEY ./boot.sh macOS / Linux boot.bat Windows Open http://localhost:11433 http://localhost:11433 , go to Settings → Server , paste your API KEY , and click the orb to talk. Manual start two terminals terminal 1: the server cd backend && npm install && npm start http://localhost:4000 terminal 2: the web client npm install && npm run dev http://localhost:11433 For production, npm run build and serve dist/ behind any HTTPS reverse proxy that forwards /api and /ws to the server. Voice: local speech-to-text and wake word python3 -m venv voice engine/venv voice engine/venv/bin/pip install -r voice engine/requirements.txt boot.sh starts it automatically on port 8001 once the venv exists. Voice: LFM3-Audio on your own computer Pick LFM in Settings → Live Voice. The first time Live starts, the backend backend/voice/lfmRuntime.js downloads llama.cpp b11960 for your platform Metal, CUDA, Vulkan or CPU and LFM3-Audio-2B https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF Q4 K M + audio projector, ~1.4 GB into the data folder, then starts and supervises llama-server on ws://127.0.0.1:9931/v1/realtime . Settings shows the progress and a Download voice model button. Already running llama-server there scripts/start lfm.sh ? It is reused. Your own server elsewhere: LFM REALTIME URL + LFM API KEY . How a turn works backend/conversation/providers/LFMRealtimeProvider.js : llama.cpp's realtime server only lets the model call tools in a text-only response, so each turn runs twice — a quick text decision a tool, or reply directly to just talk , then a spoken answer after the tool's output. A delegation starts in the background, Jarvis says it's on it, and the agent's answer is spoken when it comes back. Guards for a 2B model: explicit routing rules prices and weather → cards, phone things → phone action , only computer work → the agent , a task that is really a timer / price / weather is sent back once to the right tool, the same task is never delegated twice within 90 s, and while Jarvis talks, barge-in needs a clearly louder voice so its own echo can't become a new turn the mic stays open . Limits: English only; no camera / screen vision; weaker general knowledge than Gemini. When the model isn't ready downloading, can't start , Live falls back to Gemini Live if a key is set. Agents Install any of these and log in once; Jarvis detects them on startup GET /api/agents : npm i -g @anthropic-ai/claude-code Claude npm i -g @openai/codex Codex Antigravity agy and Hermes Agent: see their docs The avatar needs an NVIDIA GPU Ampere or newer recommended; an RTX 3060 Ti renders in real time with CUDA 12 and TensorRT 10. cd live avatar pixi install environment https://pixi.sh pixi run download weights from the gated Hugging Face repo accept the AVTR-1 terms first pixi run build-trt-engines compile TensorRT engines for your GPU Then ./boot.sh starts the streamer as well. On Windows, live avatar/setup avatar windows.bat sets up the streamer natively and the renderer inside WSL2. Full details, benchmarks and TURN setup for remote access are in live avatar/README.md https://github.com/samukbg/Jarvis-Live/blob/main/live avatar/README.md . Switch Orb → Avatar in the UI, pick an avatar in Settings, or click New avatar to create one from a photo. Turn one portrait photo into a real-time talking avatar, and optionally give it a cloned voice from a few seconds of audio . No 3D modeling, no video shoot, no training run you have to manage: Jarvis prepares the portrait for the AVTR-1 renderer on your GPU and the new avatar shows up next to the built-in ones. 1. Open Settings and click New avatar next to Live avatar . 2. Name your avatar, pick a gender and a base Gemini Live voice Kore, Aoede, Puck, Charon, Fenrir . 3. Add a portrait photo PNG, JPEG or WebP : head and shoulders, facing the camera, neutral expression, even lighting, no sunglasses. The 16:9 stream preview shows how it will be framed. 4. Optional: add a voice sample : upload or record 5–30 seconds of clean speech. Jarvis clones it with XTTS-v2 so your avatar speaks with that voice; without a sample it uses the base Gemini voice. 5. Click Create Live Avatar . Progress shows in the app; when it's ready, the avatar is selectable everywhere, and you can delete it from the same list. Good to know - Portraits and voice samples stay on your machine : they're stored next to the renderer live avatar/ , never uploaded to a third party. - Voice cloning needs the XTTS voice server live avatar/voice server/ , port 8010 . On a single GPU it competes with the 25 fps renderer, so a second GPU or machine XTTS URL gives the smoothest result. - Use photos and voices you have the right to use. Only create avatars of yourself or of people who gave you their consent. Settings → Calendar connects the calendars Jarvis reads and writes: | Source | How | Jarvis can | |---|---|---| | Google Calendar | Sign in with Google through your own OAuth client see below | read, add, move, delete | | iCloud | Apple ID + an app-specific password appleid.apple.com https://appleid.apple.com → Sign-In and Security → App-Specific Passwords | read, add, move, delete | | Fastmail, Nextcloud, Yahoo, any CalDAV | server, user name, app password | read, add, move, delete | | Calendar link | an https / webcal .ics address Google's secret iCal address, Outlook's published calendar, schools, teams, fixtures | read | | .ics file | import drag & drop or pick a file | read | - Everything stays on your computer calendar-accounts.json , readable only by you, plus an event cache . Accounts sync every 15 minutes, 30 days back to 6 months ahead; repeating events, moved or cancelled occurrences and time zones including Outlook's Windows zone names are handled, and the same meeting in two calendars shows once. - Jarvis knows your week : the next 7 days go into every voice conversation, so "Am I free tonight?" is answered without a lookup. Turn it off in Settings → Calendar. - Voice tools Gemini Live, OpenAI Realtime, the live avatar : calendar agenda , calendar add event warns about clashes , calendar update event , calendar delete event only after you confirm . With Gemini Live off, agenda questions are answered on the computer in a second, and the agents get the agenda plus a short-lived local endpoint to change it. - The agenda card shows the day or week with a "now" line, the next event's countdown, Join buttons for Meet / Zoom / Teams and the event just added. - Try: "What's on my calendar tomorrow?" , "Am I free Friday afternoon?" , "Add lunch with Ana on Friday at 1 pm" , "Move my dentist to 4" , "Cancel the 3 pm call." - API: backend/calendar/API.md https://github.com/samukbg/Jarvis-Live/blob/main/backend/calendar/API.md . Sign in with Google your own OAuth client . Google sign-in needs an OAuth client of your own; until it's set, Settings → Calendar explains why the Google button is off, and you can still add a Google calendar with its secret iCal address as a calendar link. 1. In the Google Cloud console https://console.cloud.google.com/ , create or pick a project and enable the Google Calendar API . 2. Set up the OAuth consent screen External, in testing is fine and add your Google account as a test user. 3. Credentials → Create credentials → OAuth client ID → Desktop app . Google accepts any http://localhost port for Desktop clients, so Jarvis signs in through http://localhost:<PORT /api/calendar/google/callback PORT defaults to 4000 . Open the Jarvis web client on the same computer as the server for this step. 4. Put the client's ID and secret in .env as GOOGLE OAUTH CLIENT ID and GOOGLE OAUTH CLIENT SECRET , restart the server, then Settings → Calendar → Add → Google . The refresh token is kept on your computer next to the other calendar accounts and only ever sent to Google. Settings → Hands-free has two switches off by default, saved per browser , a clap sensitivity, a live status line and a Test button: - "Hey Jarvis" : say it and Jarvis starts listening; "Hey Jarvis, what's the weather in Paris?" also passes the question on. The MIT-licensed Moonshine tiny https://huggingface.co/onnx-community/moonshine-tiny-ONNX speech model runs in a web worker on short speech segments only, and answers about 0.5 s after you stop speaking. It is downloaded once from Hugging Face about 52 MB and cached by the browser; ONNX Runtime's WebAssembly is served by your own Jarvis build dist/ort/ , not a CDN. - Double clap : two claps 150 to 700 ms apart, then quiet. A third clap, applause, knocking, speech or a door slam don't count. - Sound is analysed on your device : nothing is sent anywhere until Jarvis hears its name or two claps. Standby pauses while Jarvis listens, talks or runs an avatar call, so it never competes with the conversation's microphone, and it works while Jarvis is open in a browser tab. - This is separate from the server-side wake word above RealtimeSTT ; you can use either. Everything lives in .env see .env.example https://github.com/samukbg/Jarvis-Live/blob/main/.env.example ; every setting is optional except the keys. | Variable | Purpose | |---|---| | API KEY | Key every client must send. If unset, a random one is generated and printed at startup | | GEMINI API KEY | Gemini Live voice and the avatar's conversation engine | | LFM REALTIME URL , LFM API KEY | Your own LFM3-Audio Realtime server ws://…/v1/realtime , optional Bearer key instead of the one Jarvis downloads and starts on this computer | | LIVE VOICE PROVIDER , OPENAI API KEY | gemini default , lfm , or openai + your OpenAI key: the orb's live voice through Gemini Live, LFM3-Audio or OpenAI Realtime OPENAI REALTIME MODEL , OPENAI REALTIME VOICE optional | | OLLAMA SERVERS | Comma-separated Ollama URLs default http://localhost:11434 | | STREAMER URL | Live-avatar streamer default: this machine | | STT URL | RealtimeSTT server default ws://localhost:8001 | | TTS SYSTEM VOICE | The computer voice that reads answers with Gemini Live off, per language en=Moira;pt=Joana ; default: a woman's voice, Samantha on macOS | | HF TOKEN | Hugging Face token for the gated AVTR-1 weights and portraits | | CRYPTO WALLET , NEWS RSS URLS | Optional wallet and news cards; also editable in Settings | | GOOGLE OAUTH CLIENT ID , GOOGLE OAUTH CLIENT SECRET | Optional: your own Google "Desktop app" OAuth client, for Sign in with Google -calendar in Settings → Calendar redirect http://localhost:<PORT /api/calendar/google/callback | Security: the server refuses requests without the key, the web client never embeds it you enter it in Settings , and .env is gitignored. Expose Jarvis to the internet only over HTTPS. Agents run commands on your machine, so treat the API key like a password. Is Jarvis Live free? Yes. The Jarvis code is MIT-licensed. You pay only for the APIs you choose the Gemini API has a free tier , or run models locally with Ollama. Can it run fully offline? The agents Hermes + Ollama , speech-to-text and the avatar renderer all run locally. Real-time voice can run locally too with the LFM3-Audio voice English only ; Gemini Live the default and OpenAI Realtime are cloud options. Do I need a GPU? Only for the live avatar. Voice, agents, cards and the neural map run on any laptop, including Apple Silicon Macs. How is this different from Siri, Alexa or ChatGPT voice? Jarvis runs on your hardware, controls real developer agents that change files and run commands, shows a photoreal avatar, and is hackable end to end. Can I use it from my phone? Yes: open the web client from any browser on your network, or put the server behind an HTTPS reverse proxy or tunnel. - Fully local voice local STT → LLM → TTS as an alternative to Gemini Live - Plugin API for custom tools and cards - Docker Compose for one-command deploys - More agent integrations Ideas and PRs are welcome: see CONTRIBUTING.md https://github.com/samukbg/Jarvis-Live/blob/main/CONTRIBUTING.md . The Jarvis Live code is released under the MIT License https://github.com/samukbg/Jarvis-Live/blob/main/LICENSE . The live-avatar engine in live avatar/ https://github.com/samukbg/Jarvis-Live/blob/main/live avatar is a modified copy of AVTR-1 https://github.com/avaturn-live/avtr-1 by Goodsize Inc. and keeps its own licenses : the streamer and renderer are under PolyForm Noncommercial 1.0.0 and the model weights under the AVTR-1 Community License see live avatar/LICENSE.md https://github.com/samukbg/Jarvis-Live/blob/main/live avatar/LICENSE.md . Commercial use of the avatar requires a license from its authors. The LFM3-Audio voice model, downloaded on first use, is not part of this repository: it is under Liquid AI's LFM Open License v1.0 free to use for organizations under $10M in annual revenue . AVTR-1 https://github.com/avaturn-live/avtr-1 · Gemini Live API https://ai.google.dev/gemini-api/docs/live · RealtimeSTT https://github.com/KoljaB/RealtimeSTT · Coqui XTTS https://github.com/idiap/coqui-ai-TTS · Ollama https://ollama.com · three.js https://threejs.org · TalkingHead https://github.com/met4citizen/TalkingHead If Jarvis Live made you smile, give it a ⭐: it helps other builders find it. <sub Keywords: open-source Jarvis AI assistant, self-hosted voice assistant, real-time AI avatar, talking head, lip-sync, Gemini Live, Claude Code voice control, Codex, AI agent orchestrator, local LLM, Ollama, Iron Man JARVIS, WebRTC, React, Node.js</sub