Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.
The current version, v1.1.0, was published to the official MCP registry on 2026-09-01. It is distributed as a hosted remote over streamable-http, filed under Web Scraping by this index. Removals and unreachable sources are measured across every server this index tracks, contract drift across the servers that answer in consecutive snapshots; the index's current counts put this record in context.
Using AI Crawler Index in Claude, Cursor, Gemini CLI, Cline, or Zed? #
MCP tool contracts can change remotely with no version bump. The mcpindex gate pins each contract and HOLDs the call when it drifts-before your agent acts. Zero credentials. This is not the package install for this server itself (use Install this server for that).
Rewrites your MCP host config so each server launches behind the gate. Inspect first: curl -fsSL https://mcpindex.ai/install.sh | less
uv tool install mcpindex-gate && mcpindex-config-wire
Verdict not yet evaluated for this tool. The semantic screen takes adversarial cases first; coverage rolls out as the corpus expands (15/150 labels to graduation). The deterministic conformance probe is built but has not yet run on the public corpus, so a recorded verdict here is REVIEW or UNVERIFIED, never a clearing ALLOW. Until a verdict is recorded, an agent should treat this tool as not-yet-cleared and fall back to its own checks. Method: the eval, four-state verdict, honest limits.
Own this server? Screen its description →
That verdict was true at screening time (snapshot 2026-09-01). #
Contracts can change after screening, with no version bump. The gate pins AI Crawler Index’s tool contracts on first sight and holds any silent change before your agent acts - the check that keeps being true on Tuesday.
See your first HOLD in 2 minutes →
Related: how to trust an MCP server · screen before install · silent contract drift
Scrape-LEio.github.nolindnaidoo/scrape-le
Analyse robots.txt content and report whether a path may be crawled.
ReefAPIio.github.reefapi/reefapi-mcp
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
io.alterlab/mcp-serverio.alterlab/mcp-server
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.