{"slug": "crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp", "title": "Crawlerpeek – check which AI crawlers your site lets in (CLI, MCP)", "summary": "Crawlerpeek, a new Node 20+ open-source package with no runtime dependencies, lets site operators check whether AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot can reach their pages, combining RFC 9309 robots.txt evaluation with live fetches using each bot's user-agent string. The tool ships as a library, a CLI for CI pipelines and an MCP server exposing three read-only tools (check_ai_access, see_as_bot, generate_robots_txt), and can generate a robots.txt that explicitly allows AI crawlers. Its CLI reports a robots.txt that cannot be fetched as \"unknown\" rather than blocked, a deliberate departure from RFC 9309 §2.3.1.4, under which crawlers treat an unreachable robots.txt as disallowing everything.", "body_md": "Check whether AI crawlers can access your site. For each bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more) it tells you:\n\n- whether `robots.txt` allows the URL's path for that bot (RFC 9309: user-agent groups, wildcards,`$` , longest match wins), and\n- what the page actually returns when fetched with that bot's user-agent string: status, final URL after redirects, content type, size, title, and whether the response looks like a block or challenge page.\n\nIt also generates a `robots.txt` that welcomes AI crawlers. A library, a CLI for CI, and an MCP server. Node 20 or newer, no runtime dependencies, ESM and CommonJS, TypeScript types included.\n\nThe same checks as a web tool: [crawlerpeek.com](https://crawlerpeek.com). Documentation with examples for CI and MCP clients: [crawlerpeek.com/cli](https://crawlerpeek.com/cli).\n\n```\nnpm install crawlerpeek       # library\nnpx crawlerpeek --help        # CLI, no install needed\nnpx crawlerpeek check https://example.com\nhttps://example.com\nrobots.txt: ok (HTTP 200) at https://example.com/robots.txt\n\nBot                 robots    HTTP\nGPTBot              blocked   200  FAIL\nClaudeBot           allowed   403 (block page)  FAIL\nGoogle-Extended     allowed   -\nnpx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked\nnpx crawlerpeek check https://example.com --bots all --json\nnpx crawlerpeek check https://example.com --no-http          # robots.txt only\nnpx crawlerpeek see https://example.com --bot ClaudeBot      # one bot, full detail\nnpx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml\n```\n\nExit codes: `0` ok, `1` the `--fail-on` condition was met, `2` usage error (including an unknown option) or request error. `see` exits `0` when the page was fetched, whatever its HTTP status, and `2` when the request failed or was refused.\n\n| `--fail-on` | Fails when | \n|---|---|\n| `blocked` | robots.txt closes the path for a bot, or the page answers that bot with 401/403/406/451 or a challenge page | \n| `any-non-200` | as above, or any other status (404, 5xx, redirects that end elsewhere) or a failed request | \n\nWithout `--fail-on` the exit code is `0` whatever the result. `--bots` takes tokens or `all`; the default is 15 bots: the main AI crawlers plus Googlebot and Bingbot. A robots.txt that cannot be fetched (timeout, 5xx, a redirect that is refused or has no target) is reported as `unknown`, with the reason in `robotsTxt.error`, and never counts as blocked. Note: this deliberately differs from RFC 9309 §2.3.1.4, where crawlers treat an unreachable robots.txt (5xx) as \"disallow everything\" (Google does too), so a site whose robots.txt returns 5xx is effectively closed to many bots even though `--fail-on blocked` passes. Treat `unknown` as a problem to fix.\n\n``` js\nimport { checkAiAccess, seeAsBot, generateRobotsTxt, failingBots } from \"crawlerpeek\";\n\nconst report = await checkAiAccess(\"https://example.com/pricing\", { bots: [\"GPTBot\", \"ClaudeBot\"] });\nreport.bots[0].robots; // { access: \"allowed\", pathAllowed: true, matchedGroup: \"*\" }\nreport.bots[0].http;   // { status: 200, finalUrl, redirects, contentType, bytes, title, looksBlocked, outcome: \"ok\", ... }\nfailingBots(report, \"blocked\"); // [\"ClaudeBot\"] when it is closed or challenged\n\nawait seeAsBot(\"https://example.com\", \"PerplexityBot\");\n\ngenerateRobotsTxt({ disallow: [\"/admin/\"], blockBots: [\"CCBot\"], sitemap: \"https://example.com/sitemap.xml\" });\n```\n\n`generateRobotsTxt` names each AI bot explicitly with `Allow: /`. Disallowed paths are repeated in that group on purpose: a bot with its own group ignores the `*` group.\n\n`crawlerpeek mcp` speaks the Model Context Protocol over stdio and exposes three read-only tools: `check_ai_access`, `see_as_bot`, `generate_robots_txt`.\n\n```\n{ \"mcpServers\": { \"crawlerpeek\": { \"command\": \"npx\", \"args\": [\"-y\", \"crawlerpeek\", \"mcp\"] } } }\n```\n\nResults are cached for 10 minutes per call. No LLM is called; the tools only fetch the URL you give them.\n\nA composite action lives in `action/`; see `action/README.md`.\n\n```\n- uses: crawlerpeek/crawlerpeek/action@v0\n  with: { url: \"https://example.com\", fail-on: blocked }\n```\n\n- **see-as-bot sends the bot's own user-agent string from your machine.** It is not the real bot and not from the operator's IP ranges. Sites that verify bot IPs (Googlebot, GPTBot and others publish theirs) may treat a request that only claims to be the bot differently, often by blocking it. A 403 for`Googlebot` here can therefore be normal; a 403 for an unusual bot with a 200 for a browser usually points at a WAF or CDN rule.`robots.txt` itself is fetched with an honest`crawlerpeek/<version>` user agent.\n- Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers: they get a robots.txt decision but are never fetched.\n- Pages are fetched as raw HTML. No JavaScript runs.\n- robots.txt is a request, not access control. Some user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow it; the operators' documentation says which.\n- The bot list and user-agent strings come from the operators' documentation and can drift; the list was last compared with the operators' documentation on 2026-10-18.\n\nOne request per bot plus one for robots.txt, sequential with a 300 ms pause (`--delay`); 10 s timeout for the whole request chain; at most 5 redirects, each re-checked (a redirect to a refused target is reported for that bot or for robots.txt, it does not stop the run); response bodies capped at 2 MB; no cookies. Private, loopback, link-local and metadata addresses are refused after DNS resolution (and again at connect time), as are ports other than 80 and 443 and URLs with credentials. `--allow-private` (CLI and `crawlerpeek mcp`) or `allowPrivate: true` lifts these limits for local testing.\n\nIssues and pull requests are welcome, see [CONTRIBUTING.md](https://github.com/crawlerpeek/crawlerpeek/blob/main/CONTRIBUTING.md). Report security problems privately, see [SECURITY.md](https://github.com/crawlerpeek/crawlerpeek/blob/main/SECURITY.md).", "url": "https://wpnews.pro/news/crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp", "canonical_source": "https://github.com/crawlerpeek/crawlerpeek", "published_at": "2026-10-25 21:11:05+00:00", "updated_at": "2026-10-25 21:42:35.798423+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "developer-tools", "ai-tools"], "entities": ["Crawlerpeek", "GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended", "CCBot", "Model Context Protocol", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp", "markdown": "https://wpnews.pro/news/crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp.md", "text": "https://wpnews.pro/news/crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp.txt", "jsonld": "https://wpnews.pro/news/crawlerpeek-check-which-ai-crawlers-your-site-lets-in-cli-mcp.jsonld"}}