Crawlerpeek – check which AI crawlers your site lets in (CLI, MCP) Crawlerpeek, a new Node 20+ open-source package with no runtime dependencies, lets site operators check whether AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot can reach their pages, combining RFC 9309 robots.txt evaluation with live fetches using each bot's user-agent string. The tool ships as a library, a CLI for CI pipelines and an MCP server exposing three read-only tools (check_ai_access, see_as_bot, generate_robots_txt), and can generate a robots.txt that explicitly allows AI crawlers. Its CLI reports a robots.txt that cannot be fetched as "unknown" rather than blocked, a deliberate departure from RFC 9309 §2.3.1.4, under which crawlers treat an unreachable robots.txt as disallowing everything. Check whether AI crawlers can access your site. For each bot GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more it tells you: - whether robots.txt allows the URL's path for that bot RFC 9309: user-agent groups, wildcards, $ , longest match wins , and - what the page actually returns when fetched with that bot's user-agent string: status, final URL after redirects, content type, size, title, and whether the response looks like a block or challenge page. It also generates a robots.txt that welcomes AI crawlers. A library, a CLI for CI, and an MCP server. Node 20 or newer, no runtime dependencies, ESM and CommonJS, TypeScript types included. The same checks as a web tool: crawlerpeek.com https://crawlerpeek.com . Documentation with examples for CI and MCP clients: crawlerpeek.com/cli https://crawlerpeek.com/cli . npm install crawlerpeek library npx crawlerpeek --help CLI, no install needed npx crawlerpeek check https://example.com https://example.com robots.txt: ok HTTP 200 at https://example.com/robots.txt Bot robots HTTP GPTBot blocked 200 FAIL ClaudeBot allowed 403 block page FAIL Google-Extended allowed - npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked npx crawlerpeek check https://example.com --bots all --json npx crawlerpeek check https://example.com --no-http robots.txt only npx crawlerpeek see https://example.com --bot ClaudeBot one bot, full detail npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml Exit codes: 0 ok, 1 the --fail-on condition was met, 2 usage error including an unknown option or request error. see exits 0 when the page was fetched, whatever its HTTP status, and 2 when the request failed or was refused. | --fail-on | Fails when | |---|---| | blocked | robots.txt closes the path for a bot, or the page answers that bot with 401/403/406/451 or a challenge page | | any-non-200 | as above, or any other status 404, 5xx, redirects that end elsewhere or a failed request | Without --fail-on the exit code is 0 whatever the result. --bots takes tokens or all ; the default is 15 bots: the main AI crawlers plus Googlebot and Bingbot. A robots.txt that cannot be fetched timeout, 5xx, a redirect that is refused or has no target is reported as unknown , with the reason in robotsTxt.error , and never counts as blocked. Note: this deliberately differs from RFC 9309 §2.3.1.4, where crawlers treat an unreachable robots.txt 5xx as "disallow everything" Google does too , so a site whose robots.txt returns 5xx is effectively closed to many bots even though --fail-on blocked passes. Treat unknown as a problem to fix. js import { checkAiAccess, seeAsBot, generateRobotsTxt, failingBots } from "crawlerpeek"; const report = await checkAiAccess "https://example.com/pricing", { bots: "GPTBot", "ClaudeBot" } ; report.bots 0 .robots; // { access: "allowed", pathAllowed: true, matchedGroup: " " } report.bots 0 .http; // { status: 200, finalUrl, redirects, contentType, bytes, title, looksBlocked, outcome: "ok", ... } failingBots report, "blocked" ; // "ClaudeBot" when it is closed or challenged await seeAsBot "https://example.com", "PerplexityBot" ; generateRobotsTxt { disallow: "/admin/" , blockBots: "CCBot" , sitemap: "https://example.com/sitemap.xml" } ; generateRobotsTxt names each AI bot explicitly with Allow: / . Disallowed paths are repeated in that group on purpose: a bot with its own group ignores the group. crawlerpeek mcp speaks the Model Context Protocol over stdio and exposes three read-only tools: check ai access , see as bot , generate robots txt . { "mcpServers": { "crawlerpeek": { "command": "npx", "args": "-y", "crawlerpeek", "mcp" } } } Results are cached for 10 minutes per call. No LLM is called; the tools only fetch the URL you give them. A composite action lives in action/ ; see action/README.md . - uses: crawlerpeek/crawlerpeek/action@v0 with: { url: "https://example.com", fail-on: blocked } - see-as-bot sends the bot's own user-agent string from your machine. It is not the real bot and not from the operator's IP ranges. Sites that verify bot IPs Googlebot, GPTBot and others publish theirs may treat a request that only claims to be the bot differently, often by blocking it. A 403 for Googlebot here can therefore be normal; a 403 for an unusual bot with a 200 for a browser usually points at a WAF or CDN rule. robots.txt itself is fetched with an honest crawlerpeek/