{"slug": "how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp", "title": "How to give Claude Code web search and scraping capabilities with MCP", "summary": "A tutorial published on Spidra's site shows how to register the hosted Spidra MCP server in Claude Code with the command `claude mcp add --transport http spidra https://mcp.spidra.io/mcp`, authenticate via `claude mcp login spidra`, and verify the connection with `claude mcp list`. The setup gives Claude Code search, JavaScript-rendered page scraping and structured JSON extraction through the Model Context Protocol, which Spidra describes as an open standard for connecting AI applications to external tools and data sources. The walkthrough uses a search for official Python Requests documentation on HTTP timeouts, returning the top 5 results with titles, URLs and descriptions, as its example workflow.", "body_md": "Claude Code has built-in tools for working with code and retrieving information from the web. But when you need to extract structured data from JavaScript-rendered pages or combine search and scraping in one workflow, a dedicated scraping service can help.\n\nYou can connect [Spidra](https://spidra.io/) to Claude Code through the Model Context Protocol (MCP). This lets Claude Code search for webpages, retrieve their content, and extract structured records without requiring you to write a separate scraper.\n\nThis tutorial shows how to set up Spidra MCP, search for official Python documentation, and extract JSON from a JavaScript-rendered website using real examples.\n\n## How MCP connects Claude Code to web data\n\n[Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that allows AI applications to connect to external tools and data sources.\n\nClaude Code supports MCP servers, which expose tools it can call while working on a task.\n\nClaude Code also has built-in web search and fetching tools. Spidra MCP adds a dedicated web data service for workflows that need search, JavaScript-rendered page retrieval, or structured extraction.\n\nThese capabilities serve different purposes:\n\n- **Search:** Find relevant webpages when you don't know their URLs.\n- **Scrape:** Retrieve content from a specific webpage, including pages that require JavaScript rendering.\n- **Structured extraction:** Extract requested information into fields your application can use.\n\nYou can combine them in a single coding session. For example, Claude Code can search for documentation, retrieve a relevant page, and return structured information from its contents.\n\n## Connect Spidra MCP to Claude Code\n\nYou'll need Claude Code installed, a Spidra account, and an internet connection.\n\nSpidra provides a hosted MCP server, so you don't need to install or maintain a separate scraping server on your computer.\n\n### Step 1: Add the Spidra MCP server\n\nOpen your terminal and run:\n\n```\nclaude mcp add --transport http spidra https://mcp.spidra.io/mcp\n```\n\nThis registers Spidra as a remote MCP server in Claude Code.\n\nThe command uses HTTP transport, which Claude Code recommends for remote MCP servers.\n\nBy default, Claude Code saves the configuration with local scope, making the server available in your current project.\n\nTo use Spidra across your projects, add `--scope user` when registering the server.\n\n### Step 2: Authenticate your Spidra account\n\nRun:\n\n```\nclaude mcp login spidra\n```\n\nFollow the authentication flow in your browser and authorize access to your Spidra account.\n\nYou can also complete authentication from inside Claude Code.\n\nStart a session:\n\n```\nclaude\n```\n\nThen enter:\n\n```\n/mcp\n```\n\nSelect Spidra and follow the sign-in instructions if authentication is required.\n\nSpidra's hosted MCP server supports browser-based authentication, so you don't need to manually add an API key for this setup.\n\n### Step 3: Verify the connection\n\nRun:\n\n```\nclaude mcp list\n```\n\nThis displays your configured MCP servers and their connection statuses.\n\nLook for Spidra in the list. A healthy connection should show `Connected`.\n\nYou can also inspect the configuration directly:\n\n```\nclaude mcp get spidra\n```\n\nIf the server is registered but requires authentication, complete the login process before using its tools.\n\nFor additional configuration options, see the [Claude Code MCP documentation](https://code.claude.com/docs/en/mcp).\n\nOnce connected, you can start using Spidra through ordinary prompts.\n\n## Search for Python documentation using Spidra\n\nSuppose you're building a Python application and need to understand how HTTP request timeouts work in the Requests library.\n\nYou could manually search the web and paste documentation into Claude Code. Instead, ask Claude Code to search for the relevant pages.\n\nUse this prompt:\n\nUse Spidra Search to find the official Python Requests documentation about HTTP timeouts. Return the top 5 results with their titles, URLs, and descriptions. Show the Spidra tool used and the actual search results. Do not generate or invent results.\n\nThe captured Claude Code response identified `spidra_search` as the tool and reported the query:\n\n```\nPython Requests documentation timeouts\n```\n\nIt requested five results from the web source without a domain filter.\n\n### What did Spidra Search return?\n\nThe returned results contained five pages:\n\n| # | Result | Source | \n|---|---|---|\n| 1 | Timeout in requests.get | Python Discussions | \n| 2 | Session Objects | Requests documentation | \n| 3 | Timeout for Python requests.get() entire response | Stack Overflow | \n| 4 | Quickstart | Requests documentation | \n| 5 | Guide to Handling Python Requests Timeout | Oxylabs | \n\nTwo results came from the official Requests documentation website.\n\nThe first was the [Advanced Usage documentation](https://requests.readthedocs.io/en/master/user/advanced/), which explains session behavior and request timeout configuration.\n\nThe second was the [Requests Quickstart](https://requests.readthedocs.io/en/latest/user/quickstart/), which covers timeout settings and explains that a timeout doesn't necessarily limit the entire response download.\n\nThe remaining results were relevant to the search query but came from third-party sources.\n\nNo search errors were reported.\n\nThe result illustrates a useful distinction: a webpage can be relevant to your query without being an authoritative source.\n\nFor development tasks that depend on official documentation, you should verify the returned URLs before using their contents.\n\n### Filter search results to official sources\n\nThe original search didn't include a domain filter, which explains why it returned documentation alongside discussions and blog posts.\n\nSpidra Search supports including and excluding domains.\n\nFor a more targeted search, you can ask Claude Code:\n\nUse Spidra Search to find documentation about HTTP request timeouts. Apply the available domain filter to `requests.readthedocs.io` and return only results from that domain. Verify the hostname of each returned URL.\n\nThis is a follow-up prompt to try, not a second completed test.\n\nDomain filtering helps narrow the results, but you should still verify that the returned pages contain the information your application needs.\n\nFor more search options, see our [guide to Spidra Search](https://spidra.io/blog/spidra-search-api).\n\n## Scrape a JavaScript-rendered page with Claude Code\n\nSearch helps you discover URLs. But when you already know the webpage, you can retrieve its contents directly.\n\nLet's use [Quotes to Scrape](https://quotes.toscrape.com/js/), a demonstration website designed for practicing web scraping.\n\nThe JavaScript version loads its quote content through the browser, making it useful for demonstrating rendered-page extraction.\n\nAsk Claude Code:\n\nUse Spidra to scrape [https://quotes.toscrape.com/js/](https://quotes.toscrape.com/js/) and extract the first 10 quotes with their authors as structured JSON. Show which Spidra tool you used, the actual returned records, any errors, and the number of complete records. Don't invent missing information.\n\nClaude Code identified `spidra_scrape` as the tool used.\n\nIts response reported browser rendering with `scrapeMode: \"default\"` and JSON extraction using the requested fields.\n\n### Extract quotes as structured JSON\n\nFor an application, retrieving the page content is only part of the task.\n\nA quotes application might need each quote's text and author as separate fields that can be stored in a database.\n\nThe extraction request specified two fields:\n\n- `author` : the author of the quote.\n- `text` : the quote itself.\n\nThe result contained 10 records, each with both fields populated.\n\nHere's an excerpt from the returned JSON:\n\n```\n{\n  \"quotes\": [\n    {\n      \"author\": \"Albert Einstein\",\n      \"text\": \"“Try not to become a man of success. Rather become a man of value.”\"\n    },\n    {\n      \"author\": \"Andr&eacute; Gide\",\n      \"text\": \"“It is better to be hated for what you are than to be loved for what you are not.”\"\n    }\n  ]\n}\n```\n\nThe complete output contained 10 quote records.\n\nAll 10 had nonempty `author` and `text` fields. Claude Code reported no scraping errors.\n\nThe result demonstrates how structured extraction can turn content from a JavaScript-rendered page into records an application can process.\n\nIt also reveals something worth checking before storing the data.\n\n## Normalize extracted values before using them\n\nOne record returned an encoded author name:\n\n```\nAndr&eacute; Gide\n```\n\nThe intended name is:\n\n```\nAndré Gide\n```\n\nThe JSON structure was valid, and the author field was populated, but the value still needed normalization.\n\nPython's built-in `html` module can decode HTML entities.\n\nRun:\n\n``` python\nfrom html import unescape\n\nauthor = \"Andr&eacute; Gide\"\nprint(unescape(author))\n```\n\nOutput:\n\n```\nAndré Gide\n```\n\nYou can apply the same normalization when processing the full collection of extracted records.\n\nFor application data, also check for missing fields, duplicate records, unexpected types, and values that don't match the source page.\n\nThe quotes example returned 10 structurally complete records. That doesn't independently establish that every extracted value was correct.\n\nIf your application depends on consistent fields, see our guide to [extracting structured data from websites](https://spidra.io/blog/scrape-any-website-structured-data-single-api-call).\n\n## Combine web search and scraping in one workflow\n\nThe previous examples used Search and Scrape separately. You can also combine them in one Claude Code task.\n\nSuppose you're building a reference application for Python HTTP clients.\n\nYou want to collect documentation about request timeouts from Requests, HTTPX, and aiohttp, but you don't have the relevant URLs.\n\nGive Claude Code this prompt:\n\nI'm building a reference for Python HTTP clients.\n\nUse Spidra to find the official documentation for Requests, HTTPX, and aiohttp.\n\nFor each library:\n\nFind its official documentation website.\n\nLocate the page explaining HTTP request timeouts.\n\nScrape the relevant page.\n\nExtract the documented timeout behavior.\n\nInclude the source URL with each record.\n\nReturn the information as structured JSON.\n\nUse only the retrieved documentation. If a value cannot be verified, return `null` instead of guessing.\n\nClaude Code can use Spidra Search to discover relevant documentation pages, followed by Scrape to retrieve and extract their content.\n\nThe resulting records can then be used to generate application code, create a reference file, or populate a development dataset.\n\nThis combined workflow is an additional example to try. It wasn't part of the two completed demonstrations above.\n\n### When should you use Search, Scrape, or Crawl?\n\nChoose the operation based on what you already know about the target.\n\nUse `spidra_search` when you have a query but no URL.\n\nUse `spidra_scrape` when you already know the webpage and need its content or structured fields.\n\nFor several known URLs, `spidra_batch_scrape` can process them together.\n\nIf you need to discover URLs within a website, `spidra_map` can help identify them before scraping. For broader website extraction, `spidra_crawl` can discover and process relevant pages.\n\nYou don't need to crawl an entire documentation website when you only need two pages.\n\nOur guide to [mapping websites before scraping selected pages](https://spidra.io/blog/map-website-batch-scrape-selected-pages) covers this workflow in more detail.\n\n## Fix common MCP and scraping issues\n\n### Claude Code cannot find Spidra\n\nCheck your configured MCP servers:\n\n```\nclaude mcp list\n```\n\nIf Spidra isn't listed, register it again using the setup command.\n\nIf it is listed but disconnected, inspect its status using `/mcp` inside Claude Code.\n\n### Spidra needs authentication\n\nRun:\n\n```\nclaude mcp login spidra\n```\n\nComplete the browser authentication flow.\n\nAvoid placing API keys in Claude Code prompts or committing credentials to a repository.\n\n### Search returns unrelated or nonofficial results\n\nNarrow the query and use available domain filters when necessary.\n\nVerify the source URLs before treating search results as authoritative documentation.\n\n### Scraping returns incomplete content\n\nCheck whether the target page requires JavaScript rendering, browser interactions, or authentication.\n\nSpidra supports browser rendering and interactions, but successful extraction isn't guaranteed for every website.\n\nInspect the returned content and error information before retrying.\n\n### Claude Code answers without using Spidra\n\nMake the tool requirement explicit:\n\nUse Spidra Search for this task. Show the retrieved source URLs and identify anything you couldn't verify.\n\nIf Claude Code cannot access Spidra, check the MCP connection instead of accepting an answer that hasn't been retrieved from the requested source.\n\n## When should you use MCP instead of a direct scraping API?\n\nMCP is useful when Claude Code needs to choose and invoke tools while working on a larger development task.\n\nFor example, you might ask it to discover documentation, retrieve the relevant pages, and create application code using the extracted information.\n\nA direct scraping API may be simpler when the task is predictable.\n\nIf your backend already knows which URLs to retrieve and which fields it needs, calling the API directly avoids requiring an AI assistant to coordinate each request.\n\nYou can use MCP during development to explore a workflow and establish the required data structure, then use the Spidra API or SDK for recurring production operations.\n\nAlso treat retrieved webpage content as untrusted data. Websites may contain instructions intended to influence an AI assistant, so don't let scraped content override your application's instructions or permissions.\n\nBefore scraping third-party websites, review their access restrictions, applicable terms, and privacy requirements. Our [web scraping legal guide](https://spidra.io/blog/is-web-scraping-legal) covers the main considerations.\n\n## Start using Spidra MCP with Claude Code\n\nConnecting Spidra through MCP lets Claude Code discover webpages, retrieve rendered content, and extract structured information as part of a coding workflow.\n\nThe examples in this tutorial show both sides of that process. Search returned relevant documentation and third-party sources for a Python question, while Scrape returned structured quote records from a JavaScript-rendered website.\n\nUse Search when you need to discover pages, Scrape when you know the URL, and validate the returned values before relying on them.\n\n[Create a Spidra account](https://app.spidra.io/) and connect the hosted MCP server to try these examples with websites you're permitted to access.", "url": "https://wpnews.pro/news/how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp", "canonical_source": "https://spidra.io/blog/claude-code-web-search-scraping", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 17:14:39.891702+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools"], "entities": ["Claude Code", "Spidra", "Model Context Protocol", "Anthropic", "Python Requests"], "also_reported_by": [], "takeaway": "Register Spidra as a remote HTTP MCP server in Claude Code with `claude mcp add --transport http spidra https://mcp.spidra.io/mcp`, add `--scope user` to make it available across projects, and authenticate through browser login rather than a manually added API key.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp", "markdown": "https://wpnews.pro/news/how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp.md", "text": "https://wpnews.pro/news/how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp.txt", "jsonld": "https://wpnews.pro/news/how-to-give-claude-code-web-search-and-scraping-capabilities-with-mcp.jsonld"}}