5,009 failed SSH logins in a week: how we locked down our servers

A production server at Hetzner logged 5,009 failed SSH logins in the seven days before the operator locked it down, with fail2ban having banned more than 1,700 IPs over time and none of the attempts succeeding. The audit, prompted by levelsio's post about taking a VPS off the public internet, also found 11 SSH keys able to log in (only 3 used in the last month), coding-agent sessions running on production for months with the oldest at 173 days, and a server that had not been rebooted in about 21 months. The operator used Claude Code to plan and script the phased lockdown — moving SSH behind Tailscale and rebooting onto the new kernel — while running every root command on production personally, after PHP shipped emergency fixes on October 21 for two critical remote code execution bugs that researchers partly found with an AI model.

The past week has been a lot in the security world. On October 21 PHP shipped emergency fixes for two critical remote code execution bugs php2shell https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching , and attackers were already using them that day. Researchers found them partly with an AI model https://fortune.com/2026/10/21/ai-tools-discovered-the-php-vulnerability-that-could-take-down-70-of-the-internet-now-the-race-is-on-to-patch-it-before-ai-powered-hackers-attack/ . Node.js put out its own security release https://nodejs.org/en/blog/vulnerability/october-2026-security-releases the same day, with a remote code execution bug in its Brotli decoder among the 10 fixes. I think this is only the beginning. Security models, open source ones included, are getting much stronger, and people are pointing swarms of agents at real software to find bugs and get into servers. If you run your own servers, the usual best practices probably aren't enough anymore. I expect a lot more of these over the next few months. Locking down our servers had been on my list for a while and I just didn't have the time. Then levelsio's post about getting your VPS off the public internet https://x.com/levelsio/status/2113587042185576853 reminded me, so on Monday and Tuesday I blocked out the time and did all of it properly. When we finally looked, our production server had 5,009 failed SSH logins in the last week. I started with an audit of the production box, run as root, so I could see who can log in, what's listening and what's actually running. Then I sat down with Claude Code and planned it in phases: what to clean up first, how to move SSH behind Tailscale without locking myself out, and when to reboot onto the new kernel with the least downtime for KaribuKit. Claude wrote the scripts and checked each step from my laptop. Every change on production went through me: I made the calls on what stays and what goes, I ran all the root commands myself, and Claude never had root on production. Our apps run on dedicated servers at Hetzner https://www.hetzner.com/dedicated-rootserver/ : KaribuKit https://karibukit.com our hotel PMS , SlopIt https://slopit.io blogs that coding agents can run , a few smaller products, and a lab box for experiments. Quick shout out to Hetzner here. I have three live servers there, I'm very happy with the service, and we're lucky enough to still be on the older prices. When the lab box needed a disk swapped during this work, support did it overnight at no charge. What we found when we actually looked Our setup was the normal one. SSH keys only with fail2ban in front, and ufw letting in 22, 80 and 443. That's what most guides tell you to do, and it works. But when we ran a proper audit on the production box, a few numbers stood out: - 5,009 failed SSH logins in the last 7 days. fail2ban had banned more than 1,700 IPs over time. None got in. But bots were knocking on port 22 all day, every day, and one OpenSSH bug is all it takes regreSSHion https://www.qualys.com/2024/07/01/cve-2024-6387/regresshion.txt in 2024 hit before login, so it didn't matter that we only allowed keys . - 11 SSH keys could log in, and only 3 had been used in the last month. Keys pile up as people join a project, help out and move on, and nobody goes back to clean them up. - Coding-agent sessions had been running on production for months. Claude Code and Codex sessions started over SSH and were never closed, and the oldest was 173 days old. - The server hadn't been rebooted in about 21 months. Security updates were installed but the server was still running the old kernel. - Our own extra backup of the KaribuKit database was failing most days. The database is on Supabase's Pro plan, which keeps daily backups, so this was a backup of our backup and I wasn't too worried. It ran on my laptop at 2pm and showed a random pop-up asking for my admin password, and I always reject those. So between June and October it only worked on the days I ran it myself, 30 out of 146, and nothing told us when it didn't. What we changed SSH only over Tailscale We installed Tailscale https://tailscale.com on the servers and our laptops, tested SSH over the tailnet with a second session still open, and then closed port 22 to the internet. A small access policy in Tailscale means our laptops can reach the servers, but the lab box can't reach production. Two firewalls instead of one ufw on the box, plus Hetzner's own firewall in front of it. One thing to know is that Hetzner's firewall for dedicated servers is stateless unlike the Hetzner Cloud one . So if you only allow 80 and 443, you also block the replies to your server's own outgoing traffic and things like apt stop working. You need a rule for TCP replies ACK flag, ports 32768-65535 and for DNS and NTP replies. Hetzner's docs https://docs.hetzner.com/robot/dedicated-server/firewall/ explain it. While we were in there we deleted two old rules we didn't need anymore. One set of keys on production Only my keys can SSH into production now. Anyone else who needs a shell gets their own user on the lab server, without sudo or Docker. Deploys don't change, because our GitHub Actions runners connect out to GitHub and never needed inbound SSH. Those runners still run CI on the production box though, so anyone who can push to our repos can run code there, and moving CI to the lab box is next on the list. No agents on production We stopped the old sessions and retired one agent setup we weren't using anymore. New agent work goes on the lab box. Docker doesn't respect ufw We tested this on the freshly rebuilt lab box: a container started with a plain -p 18080:80 was reachable from the internet even though ufw blocked that port. Docker writes its own iptables rules and ufw never sees the traffic. We set "ip": "127.0.0.1" in /etc/docker/daemon.json , so published ports default to localhost, and added a DOCKER-USER rule that drops new connections from the public interface Docker's docs https://docs.docker.com/engine/network/packet-filtering-firewalls/ describe the ufw problem . Testing again from outside, both a default -p and an explicit 0.0.0.0 publish were blocked. A reboot, then Livepatch KaribuKit and SlopIt were down for 2 minutes and 20 seconds, and I told the team before we did it. We then turned on Livepatch https://ubuntu.com/security/livepatch , part of Ubuntu Pro, so most kernel fixes now apply without a reboot. Backups you've actually restored KaribuKit's database is on Supabase, so it isn't on these servers at all, and Supabase already backs it up daily. We still want our own copy outside Supabase, so we rebuilt the laptop job as a plain pg dump that logs in as a read-only database user, with its password stored in the macOS Keychain so there's no pop-up. Every dump gets checked before it's kept size and table count . We also restored a dump into a throwaway Postgres and compared row counts with live, and they matched. Every backup now also gets encrypted with age https://github.com/FiloSottile/age and copied to a Hetzner Storage Box https://www.hetzner.com/storage/storage-box/ €3.20 a month plus VAT for 1 TB, with daily snapshots . That covers SlopIt's database too. If a job fails or doesn't run at all, Healthchecks.io https://healthchecks.io emails me, and we tested that too. How not to lock yourself out I didn't want to lock myself out of production, so we did it in this order. - Test SSH over Tailscale in a new terminal while your current session is still open. Only then close port 22. - Compare the server's host key over the tailnet with the one you saw over the public IP, so you know it's the same box. - Know your way back in if Tailscale ever breaks. On Hetzner that's a temporary firewall rule for your own IP plus the rescue system. We rehearsed it once on the lab box before touching production. A quick check you can run today failed SSH logins in the last week sudo journalctl -u ssh --since -7d | grep -cE 'Failed|Invalid user' who can log in as you ssh-keygen -lf ~/.ssh/authorized keys how long since the last reboot, and is one pending? uptime -p; cat /var/run/reboot-required 2 /dev/null anything listening on all interfaces that shouldn't be? sudo ss -tlnp | grep -v 127.0.0.1 — NJ I run SimbaStack https://simbastack.com/ , where we build AI agents and the systems they work in, including KaribuKit https://karibukit.com and SlopIt https://slopit.io from this post. If you need help getting an agent into production, tell me what you're building: nj@simbastack.com mailto:nj@simbastack.com .