Barnaby: Self-hosted Matrix agent that reads the room

Developer Paul Kulak released Barnaby, a self-hosted Matrix chat server and AI agent that deploys from a domain, an SSH key, and an OpenRouter API key via one command. Barnaby Home bundles the tuwunel homeserver, Element at chat.<domain>, LiveKit for voice and video calls, and Let's Encrypt certificates, while the Barnaby agent runs in a sandboxed NixOS container, reads the whole room rather than requiring @-mentions, and can write auditable new skills stored in git. The stack requires an x86_64 Linux machine with at least 2 GB of RAM and 20 GB of disk, runs in about 450 MB, and has so far been tested only on an AWS EC2 t3.small.

A private group chat for your family or friends, with an AI agent that fits into it. One script turns a spare computer or a VPS into a complete Matrix server: Element chat, encrypted rooms, voice and video calls. A Barnaby https://github.com/pkulak/barnaby agent lives in the main room. The agent. Most chat bots want to be @-mentioned or talked to one-on-one. Barnaby reads the whole room, and a tiny decision model picks out the messages meant for it, so people talk normally and it chimes in when someone's talking to it. It remembers past conversations, and with just the OpenRouter key it can draw images, read voice messages, follow games, and write itself new skills when someone asks. The chat server. Self-hosting Matrix usually means wiring up a homeserver, a web client, certificates for several subdomains, TURN and LiveKit for calls, and some way to handle invites. Barnaby Home does all of it from a domain, an SSH key, and an OpenRouter key. You need Docker on your computer, not Nix experience. - The stack: tuwunel https://github.com/matrix-construct/tuwunel as the server, Element https://element.io at chat.<domain , LiveKit https://livekit.io for calls, and Let's Encrypt certificates. - Invite-only: sign-up takes a token. Everyone lands in an encrypted General room with the agent, which other Matrix servers can't join. - A sandboxed agent: it runs in its own NixOS container, not on the host. - More skills when you want them: web search, weather, and calendars, each turned on with its own API key. - Self-written skills you can audit: they're kept in git, so you can see and undo them. - Private defaults: every model it uses has zero data retention ZDR endpoints. - One command to deploy or update, with everything built on your computer. - An x86 64 machine with at least 2 GB of RAM nixos-anywhere needs 1.5 GB just to install and 20 GB of disk. A spare PC or mini PC at home works, and so does a VPS. It needs to be running Linux, any kind, and you need to be able to SSH into it as root or as a user with sudo. At home, booting it from the NixOS installer on a USB stick does that. The install erases the disk. So far it's only been tested on an AWS EC2 t3.small. Everything runs in about 450 MB, and builds happen on your computer, so on a VPS you can probably shrink it to 1 GB after the install. - A public IP address. A VPS has one. At home, your internet connection needs its own IPv4 address not one shared through CGNAT , and your router needs to forward these ports 2-open-the-ports to the machine. - A domain , or a subdomain of one you already have. - Docker https://docs.docker.com/get-docker/ on your own computer, and 10 GB or so of disk for it. On a Mac with Apple Silicon, it runs the x86 64 container through Rosetta Docker Desktop's "Use Rosetta" setting , which is slower. - An SSH key. If you don't have one, ssh-keygen -t ed25519 makes one. - An OpenRouter API key. Turn on "Zero Data Retention" in OpenRouter's privacy settings too; it's the only thing that stops a request from going to a provider that keeps it see Privacy privacy . Add two A records and AAAA , if the machine has IPv6 , both pointing at its public IP: barn.example.com A 203.0.113.10 .barn.example.com A 203.0.113.10 The wildcard covers chat. and call. . Let's Encrypt checks these names during the install, so set them up first. The IP has to stay the same, too: on AWS, use an Elastic IP. At home, get a static IP from your ISP, or keep the records up to date with your DNS provider's dynamic DNS. NixOS runs its own firewall, but there's usually another one in front of the machine: a security group on AWS, or your router at home. Open these there, and at home, forward them to the machine: | Port | Protocol | For | |---|---|---| | 22 | TCP | SSH | | 80, 443 | TCP | The web, Matrix, and certificates | | 7881 | TCP | Calls, when UDP is blocked | | 50000–51000 | UDP | Calls | | 3478 | UDP | Calls through TURN, for people behind strict NAT | | 5349 | TCP | Calls through TURN over TLS, for networks that block UDP | Leave out everything but 22, 80, and 443 if you set calls.enable = false . curl -fsSLO https://raw.githubusercontent.com/pkulak/barnaby-home/main/template/barnaby-home bash barnaby-home setup barn cd barn That makes a barn directory with a git repository, the config, and a copy of the script. Everything after this runs from there. The first run takes a few minutes, while Docker downloads Nix and Nix downloads everything else. ./barnaby-home configure It asks for the domain, the SSH key to log in with ~/.ssh/id ed25519 , by default , how you SSH into the machine now ssh nixos@192.168.1.50 , or ssh -i ~/keys/aws.pem ubuntu@203.0.113.10 on AWS, say , your username in the chat, the time zone, the agent's name, any extra skills, and the OpenRouter key. It logs in to the machine to find the disk to install onto. The answers go into configuration.nix and secrets.env which git ignores, so the keys never end up in the repository or the Nix store , and you can edit both by hand from here on; see Configuration configuration . You can run configure again, too, but it starts configuration.nix over, so it asks first if you've changed anything. The agent works fine without knowing anything about the people using it, but it's much better when it does. Edit soul.md who's who, where you live, which teams you follow and uncomment soul = ./soul.md; in configuration.nix . You can do this later, too; it takes effect on the next deploy. ./barnaby-home install This erases the machine's disk, so it has you type the domain first. Then it runs nixos-anywhere, which builds the system on your computer, copies it over, and reboots the machine into NixOS take out the USB stick, if you booted from one . It writes two files you should commit: facter.json , which describes the machine's hardware, and known hosts , its new SSH host key. At the end, install prints the setup token. Open https://chat.barn.example.com , choose "Create account", and sign up with your username and that token. Within a minute you're a server admin, and an admin room shows up in Element under "System Alerts" . You'll also be in the General room with the agent. If you lose the token, it's on the server: ssh root@barn.example.com cat /var/lib/barnaby-home/registration-token Make a one-time invite token by sending this in the admin room: admin token issue --once --max-age 7d Send them the token and the https://chat.<domain link. Once they sign up, they're in the General room too. The setup token keeps working, so keep it to yourself. After changing configuration.nix , soul.md , or secrets.env : ./barnaby-home deploy It builds on your computer, switches the server over, and if secrets.env changed, copies it up and restarts the agent. That replaces the server's copy, so make key changes here, not there. If a key that a skill needs is missing, journalctl -u barnaby-home-setup on the server says so. To update Barnaby Home, Barnaby, and NixOS, then deploy: ./barnaby-home update Docker keeps the Nix store in a volume, so these don't download everything again. It only grows; docker volume rm barnaby-home-nix gets the space back, and the next run starts over. Everything is under barnabyHome in configuration.nix : | Option | Default | What it does | |---|---|---| | domain | | The Matrix server name, and the base for chat. and call. | | disk | | The disk to install onto | | admins | | Usernames that become server admins when they sign up | | location | null | "latitude,longitude" , the weather skill's default location | | secretsFile | /var/lib/barnaby-home/secrets.env | Where the keys are. Point this at an agenix or sops secret if you use those. | | calls.enable | true | Voice and video calls | | agent.name | "Barnaby" | The agent's display name | | agent.username | name , lowercased | Its Matrix username. It's set on first boot, so changing it later does nothing. | | agent.soul | Barnaby's, with name filled in | A file with the agent's personality and instructions, where @name@ becomes name | | agent.model | deepseek/deepseek-v4.1-flash | The OpenRouter model it chats with | | agent.skills | See below | Skills to turn on or off | It's a Nix file, but all you need is what's shown here: true or false , text in quotes, lists in , and a ; after each setting. The agent also uses the system's time.timeZone . | Skill | What it does | Needs | Default | |---|---|---|---| | image | Draws and edits images | OPENROUTER API KEY | On | | transcribe | Reads voice messages | OPENROUTER API KEY | On | | sports-scores | Scores, schedules, and standings | OPENROUTER API KEY | On | | sports-monitor | Watches a game and sends one alert | Nothing more | On | | weather | Forecasts and conditions | TOMORROWIO API KEY | Off | | web-search | Searches the web with Kagi | KAGI KEY | Off | | calendar | Reads and edits a CalDAV calendar | CALDAV URL , CALDAV USERNAME , CALDAV PASSWORD | Off | | skill-writer | Lets the agent write its own skills | Nothing more | On | Turn one on with agent.skills.weather = true; , or off with false . Turning off sports-scores turns off sports-monitor too. A path to a directory with a SKILL.md adds your own; see Barnaby's skills docs https://github.com/pkulak/barnaby/blob/master/docs/skills.md . With skill-writer on, anyone can ask the agent to learn something new "every Sunday, check what's due this week" , and it writes itself a skill. It can't change the bundled skills, its soul, or anything else in your config. - Its skills live in /var/lib/barnaby/.agents/skills , a git repo, and it commits every change. To see what it's done, or undo something, SSH in and use git log and git revert there. - Whenever it adds, changes, or removes a skill, it says so in the General room, even if someone asked in a DM. - If a skill needs an API key, it asks for it in a DM, saves it to /var/lib/barnaby/.agents/secrets.env , and suggests deleting the message. - It has the usual command-line tools git, ripgrep, jq, ffmpeg, ImageMagick, pandoc, Python, Node, and more , and can nix shell anything else. Weekly garbage collection cleans those up again. The chat server is yours: messages, accounts, and files stay on it. The agent, of course, has to send what it reads to a model. Here's where it goes: - Chat: agent.model , through OpenRouter. The default, DeepSeek V4.1 Flash, is an open model with ZDR endpoints at over 20 providers. - Every group message: Jev https://openrouter.ai/typesafe/jev-1.13 , a tiny ZDR model, gets each one with the room's last 20 messages and decides whether it's meant for the agent. The agent still gets every message as context, but only the ones meant for it start a turn and a call to agent.model . DMs skip Jev and always go to the agent. - Memory: every night, agent.model also reads conversations that have been quiet for 3 days and writes them up as notes, which the agent can search later. The notes stay on the server, in /var/lib/barnaby/memory . - Images and voice messages: Microsoft's MAI models on Azure, through OpenRouter, both ZDR. - Weather, search, and calendar: Tomorrow.io, Kagi, and your CalDAV server, but only if you turn them on. - The agent's own skills: whatever services they use. The agent has one conversation across every room and DM, so it knows what was said in DMs when it's talking in the General room. It's told not to repeat private things from DMs, but that's an instruction, not a wall. The same goes for API keys: one sent in a DM is still in the agent's context, and in its session files on the server, even after the message is deleted. Nothing in Barnaby Home forces OpenRouter to use ZDR endpoints yet. OpenRouter's account setting does: with it on, a request to a model without a ZDR endpoint fails instead of quietly going somewhere else. - No backups yet. Everything lives in /var/lib , including the agent's own skills and keys, and losing the machine loses it. - The agent created the General room, so it's the room's admin. Nothing hands that to a person yet. - Phone apps haven't been tested yet. Element in the browser works, including calls. If you already have Nix, with flakes, you can skip Docker. The script runs these same commands. mkdir barn && cd barn nix flake init -t github:pkulak/barnaby-home git init && git add . cp secrets.env.example secrets.env && chmod 600 secrets.env Edit configuration.nix at least the domain, disk, admin, time zone, and SSH key and secrets.env . Then put secrets.env where the server expects it, and install: js mkdir -p extra/var/lib/barnaby-home chmod 755 extra extra/var extra/var/lib chmod 700 extra/var/lib/barnaby-home cp secrets.env extra/var/lib/barnaby-home/ nix run github:nix-community/nixos-anywhere -- \ --flake . home \ --generate-hardware-config nixos-facter ./facter.json \ --extra-files ./extra \ --target-host root@203.0.113.10 rm -r extra git add facter.json To deploy changes add --build-host root@barn.example.com on a Mac : nix run nixpkgs nixos-rebuild -- switch --flake . home --target-host root@barn.example.com Keys live on the server after that, in /var/lib/barnaby-home/secrets.env ; restart the agent with systemctl restart container@barnaby after changing them.