{"slug": "ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0", "title": "AI Crawler Index (dev.workers.pathwren.www/ai-crawler-index@1.1.0)", "summary": "The AI Crawler Index, version 1.1.0, was published to the official MCP registry on 2026-09-01, offering a hosted remote service to identify AI crawlers by user-agent or IP and verify them against operator ranges. The index is filed under Web Scraping and includes a gate that pins tool contracts and holds calls when drift occurs, but the tool has not yet been evaluated, with a verdict of REVIEW or UNVERIFIED. The index also lists related MCP servers such as Scrape-LE, ReefAPI, and Alterlab for web scraping tasks.", "body_md": "[← Index](/leaderboard)\n\n# AI Crawler Index\n\nIdentify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.\n\nThe current version, v1.1.0, was published to the official MCP registry on 2026-09-01. It is distributed as a hosted remote over streamable-http, filed under Web Scraping by this index. Removals and unreachable sources are measured across every server this index tracks, contract drift across the servers that answer in consecutive snapshots; [the index's current counts](/stats) put this record in context.\n\n## Using AI Crawler Index in Claude, Cursor, Gemini CLI, Cline, or Zed?\n\nMCP tool contracts can change remotely with no version bump. The mcpindex gate pins each contract and **HOLDs the call** when it drifts-before your agent acts. Zero credentials. This is not the package install for this server itself (use **Install this server** for that).\n\nRewrites your MCP host config so each server launches behind the gate. Inspect first: curl -fsSL https://mcpindex.ai/install.sh | less\n\n```\nuv tool install mcpindex-gate && mcpindex-config-wire\n```\n\n[method](/methodology)\n\nVerdict not yet evaluated for this tool. The semantic screen takes adversarial cases first; coverage rolls out as the corpus expands (15/150 labels to graduation). The deterministic conformance probe is built but has not yet run on the public corpus, so a recorded verdict here is REVIEW or UNVERIFIED, never a clearing ALLOW. Until a verdict is recorded, an agent should treat this tool as not-yet-cleared and fall back to its own checks. Method: [the eval, four-state verdict, honest limits](/methodology).\n\nOwn this server? [Screen its description →](/screen)\n\n## That verdict was true at screening time (snapshot 2026-09-01).\n\nContracts can change after screening, with no version bump. The gate pins AI Crawler Index’s tool contracts on first sight and holds any silent change before your agent acts - the check that keeps being true on Tuesday.\n\n[See your first HOLD in 2 minutes →](/guides/install-the-gate-first-hold)\n\nRelated: [how to trust an MCP server](/guides/how-to-trust-an-mcp-server) · [screen before install](/guides/screen-mcp-server-before-install) · [silent contract drift](/guides/mcp-silent-contract-drift)\n\n[methodology](/methodology)\n\n[Scrape-LEio.github.nolindnaidoo/scrape-le](/server/io-github-nolindnaidoo-scrape-le)\n\nAnalyse robots.txt content and report whether a path may be crawled.\n\n[ReefAPIio.github.reefapi/reefapi-mcp](/server/io-github-reefapi-reefapi-mcp)\n\nOne MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.\n\n[io.alterlab/mcp-serverio.alterlab/mcp-server](/server/io-alterlab-mcp-server)\n\nWeb scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.", "url": "https://wpnews.pro/news/ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0", "canonical_source": "https://mcpindex.ai/server/dev-workers-pathwren-www-ai-crawler-index", "published_at": "2026-09-01 00:36:57+00:00", "updated_at": "2026-09-01 01:53:03.491122+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools"], "entities": ["AI Crawler Index", "MCP registry", "Scrape-LE", "ReefAPI", "Alterlab"], "alternates": {"html": "https://wpnews.pro/news/ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0", "markdown": "https://wpnews.pro/news/ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0.md", "text": "https://wpnews.pro/news/ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0.txt", "jsonld": "https://wpnews.pro/news/ai-crawler-index-dev-workers-pathwren-www-ai-crawler-index-1-1-0.jsonld"}}