Blog ran on Ubuntu 16.04 for 10 years. I migrated it to FreeBSD

After running on an outdated Ubuntu 16.04 LTS Digital Ocean VPS for over ten years, the author migrated their blog to a cheaper, more powerful Hetzner virtual machine running FreeBSD. The old server had been unsupported for at least five years, posing security risks, while the new setup offers double the memory and CPU for less than €6 per month. The author chose FreeBSD to gain hands-on experience with its integrated design, security features, and Jails virtualization system.

This blog has been running on a Digital Ocean VPS for over ten years. A machine hosted in New York City, running Ubuntu 16.04 LTS. An LTS that hasn’t been in support for at least 5 years. It was about time to change it. After some considerations, I migrated to a Hetzner virtual machine that is way better than my old Ubuntu one, less than half the price of what I used to pay, and just across the country from me. Not only that, but I took the challenge to move my stack to FreeBSD. It’s a long text, but stay for a cool introduction of FreeBSD Jails with Bastille and some interesting site load benchmarks. Motivation If you know how releases on Ubuntu work I’m not very familiar myself , once the release is out of support, the apt package repository is out, so you can’t get any updates from it anymore. There are several implications of running such an outdated system, and the most obvious is that your server is just not as secure anymore. There might be several bots out there just trying to find nodes with vulnerabilities to introduce malicious stuff onto them. Luckily I think , nothing ever happened. Not that there was anything important to be stolen in there either way. But I remember a long time ago, one WordPress blog that I had, which was also running on an old VPS, randomly got a lot of very suspicious links to casino and gambling spread across the text in the posts. I was already using a Hetzner VPS as a remote development machine, where I SSH into from anywhere, and it’s been a reliable good VPS for the price. So I decided to start by comparing the specs. This was the droplet running my blog and my other websites: My old Digital Ocean server That has 2GB of RAM, one vCPU, 50GB disk, 2TB of monthly traffic and runs Ubuntu 16.04 x64. It’s located in their datacenter in New York City. That’s probably why it’s so expensive, I was paying $13 monthly for that. Hetzner is an established European IT company and has big data-centers in Germany, where I live. The cheapest Hetzner server I could get, at only 3.56 euros is already way better than my old one: A cup of coffee per month Double the memory and CPU, slightly less storage space, but ten times more traffic. But I decided to go with a beefier setup, for less than €6 a month: A fancier cup of coffee per month It might even be a bit overkill for my sites, but why not? The old setup My old setup was serving a few more sites than just this blog. Nothing too popular; this blog, the most popular of all sites in there, wouldn’t get more than a couple thousand page views a month. Except when a couple of posts went viral on Hacker News, there wasn’t a lot of traffic. In the end, the machine was basically serving static sites, no fancy CGI or custom code running. The stack was simple. Everything was served with nginx/1.10.3, statically. So I’d just basically have several config files in /etc/nginx/sites-available for each one of the sites. Extra necessary programs like static site generators and a LaTeX suite e.g.: this blog is generated by Hugo were installed either via apt or snap . In fact, my process to update my blog was: - write the text locally - commit and push to the repository - ssh into the server - pull the repo updates - run hugo During the first years of this VPS, I also used it to run some tests and do some programming. So it was bloated with a lot of outdated software that I didn’t use anymore. But it worked. And worked quite well. Yep, it was running Linux 4.4 So well, in fact, that its uptime was 1491 days when I shut it off That’s roughly 4 years without interruption Why FreeBSD Not gonna lie, one of my main motivations was to get my hands on something different. I’ve been reading and watching a lot of stuff about BSDs in general, and I had a short previous experience with FreeBSD, so I thought it would be a good way to put it to a real-world test. FreeBSD is usually praised for its stability, due to its integrated design, security, and Jails. We’ll get to that in a bit. I don’t wanna sound like I knew exactly what I was doing, but when I read about Jails, I knew exactly what I wanted to do. Jails is a form of virtualization/containerization that’s been part of FreeBSD for over 25 years, way before Docker was even a thing. At first glance, it does exactly what you’d expect from a Docker container: it sandboxes a “minisystem” within it so you can run stuff that doesn’t have access to your host system. The main difference, I think, is that Docker and other container solutions are more suited for “packaging programs”. It’s ephemeral and immutable. It will operate on data that comes from outside the container, but within it, everything will always be the same. Whereas Jails are really subsystems, almost like mini-VMs, but that are in reality sharing the same kernel. On top of that, its filesystem, ZFS, is actually really good and useful for servers. If you come from a Linux world, you probably have heard about Btrfs, which is the newer filesystem that’s been adopted by more and more distros lately. It has some similarities to ZFS: data integrity and snapshots. Except ZFS is probably way more mature than Btrfs. If I take frequent snapshots of my system, I don’t need to rely on the VPS provider snapshot or backup system, which I have to pay extra. My idea was to have one Jail for each one of my sites with whatever tools they needed to build like Hugo, for the blog and an instance of nginx to serve it. And one Jail for the main web server that connected all of them with the world via reverse proxy. That way, if one of the jails get compromised, I can just destroy it and create a new one. I’ll get more in detail how to set them up. Hetzner VPS Here are more details about it, for all you fetchists: I don’t know why fastfetch always report more memory being used than the actual values. I’ve never seen more than 3GiB used in btop for this server Setup I’ll go through quickly how I set this server up and what the most important points are. I usually keep a dev journal, so I was documenting as I was doing. However, I wouldn’t take this as a guide because I might have had missed steps. This is more of a taste of what it was to set up a web server with FreeBSD on Hetzner. Installing FreeBSD Hetzner offers some images upon the creation of the VM, but they’re fairly limited: No BSD But I saw a guide explaining how to install it on Hetzner from the official FreeBSD YouTube channel. Totally recommend the channel, by the way. The thing is that Hetzner actually provide the FreeBSD image, but it’s just hidden, a couple of steps away. It offers as an ISO image. So, when prompted to select an OS image in the creation process, you just need to pick any, then you’ll eventually wipe it all out. After creating the VM, all you need is to go to the ISO Images tab in the Console and mount the image you want: FreeBSD ISO Images available I picked 14.3. And rebooted. Then I basically followed the installer. Don’t remember if I changed anything from the standard options, but I was following the install video from the official FreeBSD channel. Boom, in no time, I had the system installed and running. Bastille I mentioned Jails a few times already. But I’m not only using Jails, but also Bastille, a system that helps managing Jails. The thing is that creating Jails by hand is a bit more complicated than I would like. There are several different steps that need to be taken care of individually, and for every new Jail created. Bastille simplifies that and offers everything you need one command away from bastille . Such as bastille list to show all the jails in the system, bastille create to create a new one, bastille console to open a shell in a jail, etc. Installing and enabling it is almost as easy as: pkg install bastille sysrc bastille enable="YES" There are also other Jail managers, I just went with the one with the coolest name. The Stack The whole idea is having a Jail running Caddy to serve all the sites and deal with domains and their SSL certificates. Then each site will have its own jail with whatever is necessary for them to build and serve. The sever jail will reverse proxy all the traffic to the respective jails. So the first thing needed is to configure an internal virtual network adapter. We can think of this stack as something similar to a bunch of virtual machines in a network. To set up a virtual network interface: sudo sysrc cloned interfaces+="lo1" sudo sysrc ifconfig lo1 name="bastille0" sudo service netif cloneup sudo sysrc ifconfig bastille0="inet 10.0.0.1 netmask 255.255.255.0" It basically clones a loopback interface, names it “bastille0” and assigns network parameters to it. The jails will work on that network interface, only. The caddy jail will have to access to the outside world, since that’ll be the one listening to the requests. For that, we need to create some internet access rule using PF Packet Filter , FreeBSD’s firewall. We need to edit /etc/pf.conf : the main interface vtnet0 and the virtual one we created bastille0 ext if = "vtnet0" int if = "bastille0" vpn if = "tailscale1" skip internal traffic set skip on $int if set skip on $vpn if allow outbound access to the internet from the jails nat on $ext if from 10.0.0.0/24 to any - $ext if redirect HTTP/HTTPS traffic from the web to the server jail 10.0.0.5 rdr pass on $ext if proto tcp from any to any port {80, 443} - 10.0.0.5 block everything else block all allows outbound traffic from host pass out quick on $ext if keep state As commented in the config, it’s basically setting all internal traffic on our bastille0 and tailscale1 networks; then setting up outbound traffic for our jails and host system; and then redirecting all the 80 or 443 HTTP and HTTPS traffic to the internal ip 10.0.0.5 , which will be our Caddy server. Now we need to enable PF: sysrc pf enable="YES" service pf start sysrc gateway enable="YES" That’s all for the networking stack. Let’s create the Caddy server jail. Creating the first Jail: Caddy Server My old server ran on good old nginx . Except for a bit of Apache , I ran nginx for most of my life. But there’s an annoying thing with nginx that Caddy solves swiftly: SSL certificates. Back in my old server, I’d have to run certbot every now and then to renew the domain SSL certificates, and I missed renewals more often than I’d like. Caddy deals with that automatically. To create a new jail, first we need to bootstrap with the version of FreeBSD we’ll be using, in my case it’s 14.3-RELEASE : bastille bootstrap 14.3-RELEASE Then we create the actual jail: bastille create caddy 14.3-RELEASE 10.0.0.5 bastille0 bastille start caddy We create a jail named caddy , using 14.3-RELEASE , with ip 10.0.0.5 on the bastille0 interface. To check the state of the jail: bastille list JID Name Boot Prio State Type IP Address Published Ports Release Tags 3 caddy on 99 Up thin 10.0.0.5 - 14.3-RELEASE - There, it’s already running Remember, jails are not supposed to be ephemeral like a Docker container, where you set up the whole config of the container and just start it and expect the machine to be exactly the same forever. We’re almost dealing with virtual machines here. We can access a shell at any moment with: bastille console caddy caddy : root@caddy:~ Setting up the Caddy Server Since we’re already within the caddy jail, we just need to install Caddy: pkg install caddy sysrc caddy enable="YES" service caddy start All the configuration for Caddy will be in /usr/local/etc/caddy/Caddyfile , within the jail. If we want to be able to access that config file from the host system, we need to mount a directory from the host system into the jail. We can even do it in read-only mode, so the jail can’t change the config. It’s something like: bastille mount caddy /usr/local/etc/my-caddy-config /usr/local/etc/caddy nullfs ro 0 0 Ok, the server is almost all configured, but we don’t have a site yet. So let’s create a site jail, a