# AI Crawler Index (dev.workers.pathwren.www/ai-crawler-index@1.1.0)

> Source: <https://mcpindex.ai/server/dev-workers-pathwren-www-ai-crawler-index>
> Published: 2026-09-01 00:36:57+00:00

[← Index](/leaderboard)

# AI Crawler Index

Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.

The current version, v1.1.0, was published to the official MCP registry on 2026-09-01. It is distributed as a hosted remote over streamable-http, filed under Web Scraping by this index. Removals and unreachable sources are measured across every server this index tracks, contract drift across the servers that answer in consecutive snapshots; [the index's current counts](/stats) put this record in context.

## Using AI Crawler Index in Claude, Cursor, Gemini CLI, Cline, or Zed?

MCP tool contracts can change remotely with no version bump. The mcpindex gate pins each contract and **HOLDs the call** when it drifts-before your agent acts. Zero credentials. This is not the package install for this server itself (use **Install this server** for that).

Rewrites your MCP host config so each server launches behind the gate. Inspect first: curl -fsSL https://mcpindex.ai/install.sh | less

```
uv tool install mcpindex-gate && mcpindex-config-wire
```

[method](/methodology)

Verdict not yet evaluated for this tool. The semantic screen takes adversarial cases first; coverage rolls out as the corpus expands (15/150 labels to graduation). The deterministic conformance probe is built but has not yet run on the public corpus, so a recorded verdict here is REVIEW or UNVERIFIED, never a clearing ALLOW. Until a verdict is recorded, an agent should treat this tool as not-yet-cleared and fall back to its own checks. Method: [the eval, four-state verdict, honest limits](/methodology).

Own this server? [Screen its description →](/screen)

## That verdict was true at screening time (snapshot 2026-09-01).

Contracts can change after screening, with no version bump. The gate pins AI Crawler Index’s tool contracts on first sight and holds any silent change before your agent acts - the check that keeps being true on Tuesday.

[See your first HOLD in 2 minutes →](/guides/install-the-gate-first-hold)

Related: [how to trust an MCP server](/guides/how-to-trust-an-mcp-server) · [screen before install](/guides/screen-mcp-server-before-install) · [silent contract drift](/guides/mcp-silent-contract-drift)

[methodology](/methodology)

[Scrape-LEio.github.nolindnaidoo/scrape-le](/server/io-github-nolindnaidoo-scrape-le)

Analyse robots.txt content and report whether a path may be crawled.

[ReefAPIio.github.reefapi/reefapi-mcp](/server/io-github-reefapi-reefapi-mcp)

One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.

[io.alterlab/mcp-serverio.alterlab/mcp-server](/server/io-alterlab-mcp-server)

Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.