cd /news/ai-agents/how-to-scrape-react-next-js-sites-wi… · home › topics › ai-agents › article
[ARTICLE · art-174293] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

How to Scrape React & Next.js Sites with Claude Desktop (Without Puppeteer)

A developer has released x402-cluster, an MCP server that lets Claude Desktop scrape client-rendered React and Next.js sites without running Puppeteer or Docker locally. The server routes requests through a serverless headless browser on Cloudflare's infrastructure, waits for the React DOM to render, and returns JSON matching a supplied schema. It is configured by adding an MCP server entry with a bearer API key to claude_desktop_config.json.

key takeaway

Adding the x402-cluster MCP server to claude_desktop_config.json lets Claude Desktop fetch client-rendered React/Next.js pages as schema-typed JSON via a serverless Cloudflare browser, avoiding a local Puppeteer or Docker setup.

by read1 min views1 publishedNov 22, 2026

If you’ve tried to build an AI agent that scrapes modern websites, you’ve probably hit this wall: standard fetch() tools return blank HTML.

Why? Because sites built with React, Vue, or Next.js render on the client. To actually get the data, you need a headless browser.

Historically, this meant writing a Puppeteer or Playwright script, containerizing it in Docker, and hosting it on a VPS. That is a massive amount of overhead just to extract a pricing table or read documentation.

Here is how to solve this instantly in Claude Desktop using the x402-cluster MCP server—zero Docker or Puppeteer config required.

Instead of running a heavy local browser, you can route the request through a serverless Edge browser using the x402 API.

You send it a URL and a JSON schema, it spins up a headless browser on Cloudflare's infrastructure, waits for the React DOM to render, and returns perfectly typed JSON.

Open your claude_desktop_config.json file and add this block:

json
{
  "mcpServers": {
    "x402-cluster": {
      "url": "[https://x402-cluster.ian-944.workers.dev/v1/mcp.json](https://x402-cluster.ian-944.workers.dev/v1/mcp.json)",
      "headers": {
        "Authorization": "Bearer sk_live_YOUR_API_KEY"
      }
    }
  }
}
── more in #ai-agents 4 stories · sorted by recency
── more on @claude desktop 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-to-scrape-react-…] indexed:0 read:1min 2026-11-22 · —