{"slug": "i-ve-been-deindexed-by-google", "title": "I've Been Deindexed by Google", "summary": "A blogger has confirmed that their website is fully deindexed from Google after a process that took at least 3 months, following an initial blanket robots.txt disallow that had no effect. The blogger discovered that Googlebot must be explicitly allowed in robots.txt so it can read the `noindex, nofollow` meta tag, and applied the same allow-then-disallow pattern to several other common bots. The blogger cited relentless AI scraping as the reason for removing the site from search engines.", "body_md": "# I've Been Deindexed by Google!\n\nI don't remember exactly when but, earlier in the year, I decided I didn't want this website to be indexed by search engines anymore. A lot of it has to do with AI and the relentless scraping that plagues the internet now. I know that as long as my blog is accessible by the public internet, there are no real measures I can put in to stop it but at least some of it will be respected.\n\nI started out by setting a blanket disallow in the robots.txt.\n\n```\nUser-agent: *\nDisallow: /\n```\n\nWhen I realised that didn't do anything after a few weeks of waiting and checking, I set the `robots` meta tag.\n\n```\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n```\n\nI thought that would do the trick. What I didn't realise is that you have to actually, ironically, **allow** the bot that will do the deindexing to visit your website so that it can see the meta tag that tells it to not index your page anymore.\n\n```\nUser-agent: Googlebot\nAllow: /\n\nUser-agent: *\nDisallow: /\n```\n\nI also did this with several other common bots and removed them from the robots.txt whenever I found that my website had been deindexed.\n\nIt's been at least 3 months since I did my last change to have Google deindex me and, after checking today, I'm happy to announce that this website is free from Google! 🥳", "url": "https://wpnews.pro/news/i-ve-been-deindexed-by-google", "canonical_source": "https://kennyqin.com/deindexed-by-google/", "published_at": "2026-10-08 07:59:30+00:00", "updated_at": "2026-10-08 08:19:00.934621+00:00", "lang": "en", "topics": ["ai-crawlers"], "entities": ["Google", "Googlebot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ve-been-deindexed-by-google", "markdown": "https://wpnews.pro/news/i-ve-been-deindexed-by-google.md", "text": "https://wpnews.pro/news/i-ve-been-deindexed-by-google.txt", "jsonld": "https://wpnews.pro/news/i-ve-been-deindexed-by-google.jsonld"}}