I've Been Deindexed by Google

A blogger has confirmed that their website is fully deindexed from Google after a process that took at least 3 months, following an initial blanket robots.txt disallow that had no effect. The blogger discovered that Googlebot must be explicitly allowed in robots.txt so it can read the `noindex, nofollow` meta tag, and applied the same allow-then-disallow pattern to several other common bots. The blogger cited relentless AI scraping as the reason for removing the site from search engines.

I've Been Deindexed by Google I don't remember exactly when but, earlier in the year, I decided I didn't want this website to be indexed by search engines anymore. A lot of it has to do with AI and the relentless scraping that plagues the internet now. I know that as long as my blog is accessible by the public internet, there are no real measures I can put in to stop it but at least some of it will be respected. I started out by setting a blanket disallow in the robots.txt. User-agent: Disallow: / When I realised that didn't do anything after a few weeks of waiting and checking, I set the robots meta tag. <meta name="robots" content="noindex, nofollow" / I thought that would do the trick. What I didn't realise is that you have to actually, ironically, allow the bot that will do the deindexing to visit your website so that it can see the meta tag that tells it to not index your page anymore. User-agent: Googlebot Allow: / User-agent: Disallow: / I also did this with several other common bots and removed them from the robots.txt whenever I found that my website had been deindexed. It's been at least 3 months since I did my last change to have Google deindex me and, after checking today, I'm happy to announce that this website is free from Google 🥳