Giving Your AI Coding Assistant Live Web Access with an MCP Server

A developer walkthrough shows how to give MCP clients like Claude Code, Claude Desktop and Cursor live web access by connecting the XCrawl web data MCP server, either through a hosted endpoint or a local stdio process via the xcrawl-mcp npm package. Setup requires a single config entry or a `claude mcp add` command, with the API key passed in the URL or an Authorization: Bearer header, and new accounts start with 1,000 free credits.

Large language models know a lot, but they don't know what changed on a website this morning. If you use Claude Code, Claude Desktop, Cursor or another MCP client, you can close that gap by connecting a web data server through the Model Context Protocol MCP . This post explains what that gives you, how to set it up in a few minutes, and how to use it without wasting tokens. What MCP adds MCP is an open protocol that lets an AI client call external tools. A web scraping MCP server exposes tools such as "scrape this URL" or "search the web" that the assistant can call on its own when a task needs fresh information: reading the latest docs for a library, checking a changelog, comparing pricing pages, or collecting examples from a site. Which tools you get As an example, the XCrawl https://www.xcrawl.ai/ MCP server npm package xcrawl-mcp, for the XCrawl web data API exposes these tools: Option 1: the hosted endpoint The simplest setup uses the hosted MCP URL, so nothing runs on your machine. In Claude Desktop's claude desktop config.json: { "mcpServers": { "xcrawl": { "url": "https://mcp.xcrawl.com/YOUR API KEY/mcp" } } } In Claude Code you can add the same endpoint from the terminal: claude mcp add --transport http xcrawl https://mcp.xcrawl.com/YOUR API KEY/mcp The server also accepts the key in an Authorization: Bearer header if you'd rather keep it out of the URL. Option 2: run it locally over stdio If you prefer a local process Node.js 18+ : XCRAWL API KEY=YOUR API KEY npx -y xcrawl-mcp or register it with Claude Code: claude mcp add xcrawl -e XCRAWL API KEY=YOUR API KEY -- npx -y xcrawl-mcp Prompts that work well Keeping it efficient Wrapping up Connecting a web data MCP server turns your assistant from "trained up to some date" into something that can check the live web when it matters. Setup is a single config entry, and new XCrawl accounts start with 1,000 free credits, which is plenty to try the workflow on your own projects. Docs are at https://docs.xcrawl.com/ https://docs.xcrawl.com/ .