{"slug": "robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control", "title": "Robots-check: see what your robots.txt is serving (Cloudflare AI Crawl Control)", "summary": "A new open-source tool called robots-check reveals that Cloudflare's AI Crawl Control feature silently appends its own rules to websites' robots.txt files, often blocking AI crawlers like GPTBot and Google-Extended without site owners' knowledge. The tool, created by developer Jani Bert and available via GitHub, fetches the live robots.txt over HTTPS and reports whether it is Cloudflare-managed, which AI bots are blocked or allowed, and whether a sitemap is declared. It requires Node 18+ and has no dependencies.", "body_md": "See what your site's `robots.txt` **actually serves** — including whether\nCloudflare's \"AI Crawl Control\" feature is silently appending its own rules\non top of yours.\n\nWhile deploying a few small sites behind Cloudflare, we noticed our own\n`robots.txt` files weren't what was actually being served. Cloudflare\ninjects a managed block (look for `# BEGIN Cloudflare Managed content`) that\nadds its own AI-crawler-blocking rules — real search engines are usually\nleft alone, but plenty of site owners have no idea this is happening at all,\nand it's easy to assume the file in your repo/origin is the one being served\nwhen it isn't.\n\n`robots-check` fetches the live file over HTTPS and tells you, in plain\nterms:\n\n- whether it's Cloudflare-managed (and where to check/adjust it)\n- whether real search engines (Googlebot, Bingbot, etc.) are actually allowed to crawl\n- which AI training/crawling bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, and others) are blocked vs. allowed\n- whether a sitemap is declared at all\n\nNo API keys, no signup, no dependencies — just Node's built-in `https`\nmodule.\n\n```\nnpx github:janibert1/robots-check example.com\n```\n\nOr clone it and run directly:\n\n```\ngit clone https://github.com/janibert1/robots-check\ncd robots-check\nnode robots-check.js example.com\n```\n\nRequires Node 18+.\n\n```\nexample.com — 47 non-blank lines\n\n⚠ This robots.txt is (at least partly) Cloudflare-managed.\n  Cloudflare's \"AI Crawl Control\" feature injects its own block into what's\n  actually served — your origin server's own robots.txt file may say\n  something different from what's shown below. Check your Cloudflare\n  dashboard under Bots → AI Crawl Control if this doesn't match your intent.\n\nReal search engines:\n  (any not listed above default to the same as User-agent: * → allowed)\n\nAI training/crawling bots:\n  gptbot                         blocked\n  google-extended                blocked\n  ccbot                          blocked\n  ...\n\nSitemap: https://example.com/sitemap.xml\n```\n\nIt can't see your origin server's own `robots.txt` file directly — only\nwhat's actually served over HTTPS to a real request, which is what matters\nfor crawling anyway. If Cloudflare (or another CDN/WAF) sits in front of\nyour site, that's the version to trust regardless of what's in your repo.\n\nMIT", "url": "https://wpnews.pro/news/robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control", "canonical_source": "https://github.com/janibert1/robots-check", "published_at": "2026-09-09 03:15:27+00:00", "updated_at": "2026-09-09 03:20:56.310335+00:00", "lang": "en", "topics": ["developer-tools", "ai-policy"], "entities": ["Cloudflare", "robots-check", "Jani Bert", "GitHub", "GPTBot", "Google-Extended", "CCBot"], "alternates": {"html": "https://wpnews.pro/news/robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control", "markdown": "https://wpnews.pro/news/robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control.md", "text": "https://wpnews.pro/news/robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control.txt", "jsonld": "https://wpnews.pro/news/robots-check-see-what-your-robots-txt-is-serving-cloudflare-ai-crawl-control.jsonld"}}