{"slug": "barnaby-self-hosted-matrix-agent-that-reads-the-room", "title": "Barnaby: Self-hosted Matrix agent that reads the room", "summary": "Developer Paul Kulak released Barnaby, a self-hosted Matrix chat server and AI agent that deploys from a domain, an SSH key, and an OpenRouter API key via one command. Barnaby Home bundles the tuwunel homeserver, Element at chat.<domain>, LiveKit for voice and video calls, and Let's Encrypt certificates, while the Barnaby agent runs in a sandboxed NixOS container, reads the whole room rather than requiring @-mentions, and can write auditable new skills stored in git. The stack requires an x86_64 Linux machine with at least 2 GB of RAM and 20 GB of disk, runs in about 450 MB, and has so far been tested only on an AWS EC2 t3.small.", "body_md": "A private group chat for your family or friends, with an AI agent that fits\ninto it. One script turns a spare computer or a VPS into a complete Matrix\nserver: Element chat, encrypted rooms, voice and video calls. A\n[Barnaby](https://github.com/pkulak/barnaby) agent lives in the main room.\n\n**The agent.** Most chat bots want to be @-mentioned or talked to one-on-one.\nBarnaby reads the whole room, and a tiny decision model picks out the messages\nmeant for it, so people talk normally and it chimes in when someone's talking\nto it. It remembers past conversations, and with just the OpenRouter key it\ncan draw images, read voice messages, follow games, and write itself new\nskills when someone asks.\n\n**The chat server.** Self-hosting Matrix usually means wiring up a homeserver,\na web client, certificates for several subdomains, TURN and LiveKit for calls,\nand some way to handle invites. Barnaby Home does all of it from a domain, an\nSSH key, and an OpenRouter key. You need Docker on your computer, not Nix\nexperience.\n\n- **The stack:**[tuwunel](https://github.com/matrix-construct/tuwunel) as the\nserver,[Element](https://element.io) at`chat.<domain>` ,[LiveKit](https://livekit.io) for calls, and Let's Encrypt certificates.\n- **Invite-only:** sign-up takes a token. Everyone lands in an encrypted\nGeneral room with the agent, which other Matrix servers can't join.\n- **A sandboxed agent:** it runs in its own NixOS container, not on the host.\n- **More skills when you want them:** web search, weather, and calendars, each\nturned on with its own API key.\n- **Self-written skills you can audit:** they're kept in git, so you can see\nand undo them.\n- **Private defaults:** every model it uses has zero data retention (ZDR)\nendpoints.\n- **One command to deploy or update,** with everything built on your computer.\n\n- **An x86_64 machine** with at least 2 GB of RAM (nixos-anywhere needs 1.5 GB\njust to install) and 20 GB of disk. A spare PC or mini PC at home works, and\nso does a VPS. It needs to be running Linux, any kind, and you need to be able\nto SSH into it as root or as a user with sudo. At home, booting it from the\nNixOS installer on a USB stick does that. The install erases the disk. So far\nit's only been tested on an AWS EC2 t3.small. Everything runs in about\n450 MB, and builds happen on your computer, so on a VPS you can probably\nshrink it to 1 GB after the install.\n- **A public IP address.** A VPS has one. At home, your internet connection\nneeds its own IPv4 address (not one shared through CGNAT), and your router\nneeds to forward[these ports](#2-open-the-ports) to the machine.\n- **A domain** , or a subdomain of one you already have.\n- **[Docker](https://docs.docker.com/get-docker/)** on your own computer, and\n10 GB or so of disk for it. On a Mac with Apple Silicon, it runs the x86_64\ncontainer through Rosetta (Docker Desktop's \"Use Rosetta\" setting), which\nis slower.\n- **An SSH key.** If you don't have one,`ssh-keygen -t ed25519` makes one.\n- **An OpenRouter API key.** Turn on \"Zero Data Retention\" in OpenRouter's\nprivacy settings too; it's the only thing that stops a request from going to\na provider that keeps it (see[Privacy](#privacy) ).\n\nAdd two `A` records (and `AAAA`, if the machine has IPv6), both pointing at its\npublic IP:\n\n```\nbarn.example.com      A  203.0.113.10\n*.barn.example.com    A  203.0.113.10\n```\n\nThe wildcard covers `chat.` and `call.`. Let's Encrypt checks these names during\nthe install, so set them up first. The IP has to stay the same, too: on AWS,\nuse an Elastic IP. At home, get a static IP from your ISP, or keep the records\nup to date with your DNS provider's dynamic DNS.\n\nNixOS runs its own firewall, but there's usually another one in front of the machine: a security group on AWS, or your router at home. Open these there, and at home, forward them to the machine:\n\n| Port | Protocol | For | \n|---|---|---|\n| 22 | TCP | SSH | \n| 80, 443 | TCP | The web, Matrix, and certificates | \n| 7881 | TCP | Calls, when UDP is blocked | \n| 50000–51000 | UDP | Calls | \n| 3478 | UDP | Calls through TURN, for people behind strict NAT | \n| 5349 | TCP | Calls through TURN over TLS, for networks that block UDP | \n\nLeave out everything but 22, 80, and 443 if you set `calls.enable = false`.\n\n```\ncurl -fsSLO https://raw.githubusercontent.com/pkulak/barnaby-home/main/template/barnaby-home\nbash barnaby-home setup barn\ncd barn\n```\n\nThat makes a `barn` directory with a git repository, the config, and a copy\nof the script. Everything after this runs from there. The first run takes a few\nminutes, while Docker downloads Nix and Nix downloads everything else.\n\n```\n./barnaby-home configure\n```\n\nIt asks for the domain, the SSH key to log in with (`~/.ssh/id_ed25519`, by\ndefault), how you SSH into the machine now (`ssh nixos@192.168.1.50`, or\n`ssh -i ~/keys/aws.pem ubuntu@203.0.113.10` on AWS, say), your username in the\nchat, the time zone, the agent's name, any extra skills, and the OpenRouter\nkey. It logs in to the machine to find the disk to install onto.\n\nThe answers go into `configuration.nix` and `secrets.env` (which git ignores,\nso the keys never end up in the repository or the Nix store), and you can edit\nboth by hand from here on; see [Configuration](#configuration). You can run\nconfigure again, too, but it starts `configuration.nix` over, so it asks first\nif you've changed anything.\n\nThe agent works fine without knowing anything about the people using it, but\nit's much better when it does. Edit `soul.md` (who's who, where you live,\nwhich teams you follow) and uncomment `soul = ./soul.md;` in\n`configuration.nix`. You can do this later, too; it takes effect on the next\ndeploy.\n\n```\n./barnaby-home install\n```\n\nThis erases the machine's disk, so it has you type the domain first. Then it\nruns nixos-anywhere, which builds the system on your computer, copies it over,\nand reboots the machine into NixOS (take out the USB stick, if you booted from\none). It writes two files you should commit: `facter.json`, which describes the\nmachine's hardware, and `known_hosts`, its new SSH host key.\n\nAt the end, install prints the setup token. Open\n`https://chat.barn.example.com`, choose \"Create account\", and sign up with\nyour username and that token. Within a minute you're a server admin, and an\nadmin room shows up in Element (under \"System Alerts\"). You'll also be in the\nGeneral room with the agent.\n\nIf you lose the token, it's on the server:\n\n```\nssh root@barn.example.com cat /var/lib/barnaby-home/registration-token\n```\n\nMake a one-time invite token by sending this in the admin room:\n\n```\n!admin token issue --once --max-age 7d\n```\n\nSend them the token and the `https://chat.<domain>` link. Once they sign up,\nthey're in the General room too.\n\nThe setup token keeps working, so keep it to yourself.\n\nAfter changing `configuration.nix`, `soul.md`, or `secrets.env`:\n\n```\n./barnaby-home deploy\n```\n\nIt builds on your computer, switches the server over, and if `secrets.env`\nchanged, copies it up and restarts the agent. That replaces the server's copy,\nso make key changes here, not there. If a key that a skill needs is missing,\n`journalctl -u barnaby-home-setup` on the server says so.\n\nTo update Barnaby Home, Barnaby, and NixOS, then deploy:\n\n```\n./barnaby-home update\n```\n\nDocker keeps the Nix store in a volume, so these don't download everything\nagain. It only grows; `docker volume rm barnaby-home-nix` gets the space back,\nand the next run starts over.\n\nEverything is under `barnabyHome` in `configuration.nix`:\n\n| Option | Default | What it does | \n|---|---|---|\n| `domain` |  | The Matrix server name, and the base for `chat.` and`call.` | \n| `disk` |  | The disk to install onto | \n| `admins` | `[ ]` | Usernames that become server admins when they sign up | \n| `location` | `null` | `\"latitude,longitude\"` , the weather skill's default location | \n| `secretsFile` | `/var/lib/barnaby-home/secrets.env` | Where the keys are. Point this at an agenix or sops secret if you use those. | \n| `calls.enable` | `true` | Voice and video calls | \n| `agent.name` | `\"Barnaby\"` | The agent's display name | \n| `agent.username` | `name` , lowercased | Its Matrix username. It's set on first boot, so changing it later does nothing. | \n| `agent.soul` | Barnaby's, with `name` filled in | A file with the agent's personality and instructions, where `@name@` becomes`name` | \n| `agent.model` | `deepseek/deepseek-v4.1-flash` | The OpenRouter model it chats with | \n| `agent.skills` | See below | Skills to turn on or off | \n\nIt's a Nix file, but all you need is what's shown here: `true` or `false`, text\nin quotes, lists in `[ ]`, and a `;` after each setting.\n\nThe agent also uses the system's `time.timeZone`.\n\n| Skill | What it does | Needs | Default | \n|---|---|---|---|\n| `image` | Draws and edits images | `OPENROUTER_API_KEY` | On | \n| `transcribe` | Reads voice messages | `OPENROUTER_API_KEY` | On | \n| `sports-scores` | Scores, schedules, and standings | `OPENROUTER_API_KEY` | On | \n| `sports-monitor` | Watches a game and sends one alert | Nothing more | On | \n| `weather` | Forecasts and conditions | `TOMORROWIO_API_KEY` | Off | \n| `web-search` | Searches the web with Kagi | `KAGI_KEY` | Off | \n| `calendar` | Reads and edits a CalDAV calendar | `CALDAV_URL` ,`CALDAV_USERNAME` ,`CALDAV_PASSWORD` | Off | \n| `skill-writer` | Lets the agent write its own skills | Nothing more | On | \n\nTurn one on with `agent.skills.weather = true;`, or off with `false`. Turning\noff `sports-scores` turns off `sports-monitor` too. A path to a\ndirectory with a `SKILL.md` adds your own; see Barnaby's\n[skills docs](https://github.com/pkulak/barnaby/blob/master/docs/skills.md).\n\nWith `skill-writer` on, anyone can ask the agent to learn something new (\"every\nSunday, check what's due this week\"), and it writes itself a skill. It can't\nchange the bundled skills, its soul, or anything else in your config.\n\n- Its skills live in `/var/lib/barnaby/.agents/skills` , a git repo, and it\ncommits every change. To see what it's done, or undo something, SSH in and\nuse`git log` and`git revert` there.\n- Whenever it adds, changes, or removes a skill, it says so in the General room, even if someone asked in a DM.\n- If a skill needs an API key, it asks for it in a DM, saves it to\n`/var/lib/barnaby/.agents/secrets.env` , and suggests deleting the message.\n- It has the usual command-line tools (git, ripgrep, jq, ffmpeg, ImageMagick,\npandoc, Python, Node, and more), and can `nix shell` anything else. Weekly\ngarbage collection cleans those up again.\n\nThe chat server is yours: messages, accounts, and files stay on it. The agent, of course, has to send what it reads to a model. Here's where it goes:\n\n- **Chat:**`agent.model` , through OpenRouter. The default, DeepSeek V4.1 Flash,\nis an open model with ZDR endpoints at over 20 providers.\n- **Every group message:**[Jev](https://openrouter.ai/typesafe/jev-1.13) , a\ntiny ZDR model, gets each one with the room's last 20 messages and decides\nwhether it's meant for the agent. The agent still gets every message as\ncontext, but only the ones meant for it start a turn (and a call to`agent.model` ). DMs skip Jev and always go to the agent.\n- **Memory:** every night,`agent.model` also reads conversations that have\nbeen quiet for 3 days and writes them up as notes, which the agent can search\nlater. The notes stay on the server, in`/var/lib/barnaby/memory` .\n- **Images and voice messages:** Microsoft's MAI models on Azure, through\nOpenRouter, both ZDR.\n- **Weather, search, and calendar:** Tomorrow.io, Kagi, and your CalDAV server,\nbut only if you turn them on.\n- **The agent's own skills:** whatever services they use.\n\nThe agent has one conversation across every room and DM, so it knows what was said in DMs when it's talking in the General room. It's told not to repeat private things from DMs, but that's an instruction, not a wall. The same goes for API keys: one sent in a DM is still in the agent's context, and in its session files on the server, even after the message is deleted.\n\nNothing in Barnaby Home forces OpenRouter to use ZDR endpoints yet. OpenRouter's account setting does: with it on, a request to a model without a ZDR endpoint fails instead of quietly going somewhere else.\n\n- **No backups yet.** Everything lives in`/var/lib` , including the agent's\nown skills and keys, and losing the machine loses it.\n- **The agent created the General room, so it's the room's admin.** Nothing\nhands that to a person yet.\n- **Phone apps haven't been tested yet.** Element in the browser works,\nincluding calls.\n\nIf you already have Nix, with flakes, you can skip Docker. The script runs these same commands.\n\n```\nmkdir barn && cd barn\nnix flake init -t github:pkulak/barnaby-home\ngit init && git add .\ncp secrets.env.example secrets.env && chmod 600 secrets.env\n```\n\nEdit `configuration.nix` (at least the domain, disk, admin, time zone, and SSH\nkey) and `secrets.env`. Then put `secrets.env` where the server expects it, and\ninstall:\n\n``` js\nmkdir -p extra/var/lib/barnaby-home\nchmod 755 extra extra/var extra/var/lib\nchmod 700 extra/var/lib/barnaby-home\ncp secrets.env extra/var/lib/barnaby-home/\n\nnix run github:nix-community/nixos-anywhere -- \\\n  --flake .#home \\\n  --generate-hardware-config nixos-facter ./facter.json \\\n  --extra-files ./extra \\\n  --target-host root@203.0.113.10\n\nrm -r extra\ngit add facter.json\n```\n\nTo deploy changes (add `--build-host root@barn.example.com` on a Mac):\n\n```\nnix run nixpkgs#nixos-rebuild -- switch --flake .#home --target-host root@barn.example.com\n```\n\nKeys live on the server after that, in `/var/lib/barnaby-home/secrets.env`;\nrestart the agent with `systemctl restart container@barnaby` after changing\nthem.", "url": "https://wpnews.pro/news/barnaby-self-hosted-matrix-agent-that-reads-the-room", "canonical_source": "https://github.com/pkulak/barnaby-home", "published_at": "2026-10-11 02:28:19+00:00", "updated_at": "2026-10-11 02:50:45.836572+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence", "large-language-models", "ai-products"], "entities": ["Barnaby", "Paul Kulak", "Matrix", "tuwunel", "Element", "LiveKit", "OpenRouter", "NixOS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/barnaby-self-hosted-matrix-agent-that-reads-the-room", "markdown": "https://wpnews.pro/news/barnaby-self-hosted-matrix-agent-that-reads-the-room.md", "text": "https://wpnews.pro/news/barnaby-self-hosted-matrix-agent-that-reads-the-room.txt", "jsonld": "https://wpnews.pro/news/barnaby-self-hosted-matrix-agent-that-reads-the-room.jsonld"}}