{"slug": "crawling-settings", "title": "Crawling settings", "summary": "Buttondown replaced its raw user-agent-string textarea with a Crawling add-on in Settings → Archives that gives users one toggle per crawler category: Search engines (Googlebot, Bingbot, DuckDuckBot, Applebot), AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and a few dozen more), and Everything else (scrapers, SEO tools like AhrefsBot, archivers, and unidentified bots). The company said it made the change after finding users do not know or care about user agent strings, and it based the three categories on usage data and support volume.", "body_md": "Some folks want their archives available so they can link to them, but don't want them to come up in Google results.\n\nSome folks want to prohibit LLMs and other scrapers from crawling their archives.\n\nBoth of these are eminently reasonable! We tried to handle a lot of this with a very engineer-brained approach: give you a big ol' textarea where you can type user agent strings to block.\n\n*People do not know or care about user agent strings*, we realized. So we replaced all of it with a **Crawling** add-on in [Settings → Archives](https://buttondown.com/settings/archives), holding one toggle per kind of crawler.\n\n[on our live demo site](https://demo.buttondown.com/settings/archives), too.\n\nWe looked at usage (and our support volume) and found the two biggest reasons/avenues that y'all care about, plus a catch-all for everyone else:\n\n| Toggle | Examples | \n|---|---|\n| **Search engines** | Googlebot, Bingbot, DuckDuckBot, Applebot | \n| **AI crawlers** | GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and a few dozen more | \n| **Everything else** | Scrapers, SEO tools like AhrefsBot, archivers, and bots that don't say who they are | \n\nThe [docs](https://docs.buttondown.com/email-archives#crawling) have the full rundown of what each toggle does under the hood.", "url": "https://wpnews.pro/news/crawling-settings", "canonical_source": "https://buttondown.com/blog/crawling-settings", "published_at": "2026-09-12 00:00:00+00:00", "updated_at": "2026-10-02 00:14:07.517106+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-policy", "ai-tools"], "entities": ["Buttondown", "Googlebot", "Bingbot", "DuckDuckBot", "Applebot", "GPTBot", "ClaudeBot", "CCBot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crawling-settings", "markdown": "https://wpnews.pro/news/crawling-settings.md", "text": "https://wpnews.pro/news/crawling-settings.txt", "jsonld": "https://wpnews.pro/news/crawling-settings.jsonld"}}