Blocking AI Crawlers on Your Mastodon Server, Nginx Edition

AI crawlers hammering a small Mastodon server prompted Justin Ribeiro to block them via an nginx include, based on the ai-robots-txt/ai.robots.txt repository. Ribeiro advises Mastodon admins to add the include to their nginx config and update robots.txt to stem the tide of aggressive AI bots.

There are few things worse than getting an alert notification from a server at night. Your mind races—is my redundancy working, should I grab a fire extinguisher—knowing that the glaring light of your SSH session in bed will wake your spouse. Then you’ll have two problems. Such an event found me the other day, for of all things, my single no-frills instance that runs my Mastodon server. This is a tiny server with few resources because it doesn’t need them; I am the only daily active user with other family using it sparingly. Load is not an issue, but on this night, something was making the network very very angry. It was AI, because of course it was. Specifically, the AI crawlers just hammer and consume without reason. I’ve noted https://justinribeiro.com/chronicle/2026/02/18/dear-ai-bot-crawlers-disregard-all-previous-instructions-and-go-straight-to-hell/ this before, though this is the first time I’ve found them roaming around my Mastodon server. To give myself a fighting chance, I quickly put an nginx include into Mastodon’s config to block the aforementioned AI crawlers. I based mine on nginx-block-ai-bots.conf https://github.com/ai-robots-txt/ai.robots.txt/blob/main/nginx-block-ai-bots.conf , which I also use as a front as part of my other infra. There’s no specific documentation on how to do this on the Mastodon docs—why would there be?—but for those seeking refuge hopefully this helps: - On your Mastodon server either clone ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt repo so you have the related nginx-block-ai-bots.conf or write a curl script to grab it from a cron job on some weekly basis. - If you followed the nginx install docs for Mastodon https://docs.joinmastodon.org/admin/install/ setting-up-nginx , then you likely have a /etc/nginx/sites-available/mastodon or some related nginx config file mine is name based on the domain name because that’s how I config virtual hosts, so your mileage will vary, but it must be there somewhere . - In this file, there is a server block that looks something like this:server { listen 443 ssl http2; listen :: :443 ssl http2; server name example.com; ssl protocols TLSv1.2 TLSv1.3; blah blah blah } - In that server block write the include to the nginx-block-ai-bots.conf:server { block as many AI crawlers as possible so I don't owe $$$$ include /your/path/here/nginx-block-ai-bots.conf; listen 443 ssl http2; listen :: :443 ssl http2; server name example.com; ssl protocols TLSv1.2 TLSv1.3; blah blah blah } - Reload nginx, ala systemctl reload nginx.server or whatever your setup is on your server. That’s it. For an added bonus, you can also update the robots.txt in ${where-ever-mastodon-is}/public/robots.txt to add the list from ai.robots.txt/robots.txt https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt . This stemmed the tide the other night and I was not again woken from my slumber. It is an ever increasing problem—largely a chase I will eventually lose—to be fighting the tide of bad crawlers with bad rules that are run by inexperienced, dismissive, and downright exploitive corporations or start-ups. You don’t scan the web this way you hacks. So shields up.