How to give Claude Code web search and scraping capabilities with MCP A tutorial published on Spidra's site shows how to register the hosted Spidra MCP server in Claude Code with the command `claude mcp add --transport http spidra https://mcp.spidra.io/mcp`, authenticate via `claude mcp login spidra`, and verify the connection with `claude mcp list`. The setup gives Claude Code search, JavaScript-rendered page scraping and structured JSON extraction through the Model Context Protocol, which Spidra describes as an open standard for connecting AI applications to external tools and data sources. The walkthrough uses a search for official Python Requests documentation on HTTP timeouts, returning the top 5 results with titles, URLs and descriptions, as its example workflow. Claude Code has built-in tools for working with code and retrieving information from the web. But when you need to extract structured data from JavaScript-rendered pages or combine search and scraping in one workflow, a dedicated scraping service can help. You can connect Spidra https://spidra.io/ to Claude Code through the Model Context Protocol MCP . This lets Claude Code search for webpages, retrieve their content, and extract structured records without requiring you to write a separate scraper. This tutorial shows how to set up Spidra MCP, search for official Python documentation, and extract JSON from a JavaScript-rendered website using real examples. How MCP connects Claude Code to web data Model Context Protocol MCP https://modelcontextprotocol.io/ is an open standard that allows AI applications to connect to external tools and data sources. Claude Code supports MCP servers, which expose tools it can call while working on a task. Claude Code also has built-in web search and fetching tools. Spidra MCP adds a dedicated web data service for workflows that need search, JavaScript-rendered page retrieval, or structured extraction. These capabilities serve different purposes: - Search: Find relevant webpages when you don't know their URLs. - Scrape: Retrieve content from a specific webpage, including pages that require JavaScript rendering. - Structured extraction: Extract requested information into fields your application can use. You can combine them in a single coding session. For example, Claude Code can search for documentation, retrieve a relevant page, and return structured information from its contents. Connect Spidra MCP to Claude Code You'll need Claude Code installed, a Spidra account, and an internet connection. Spidra provides a hosted MCP server, so you don't need to install or maintain a separate scraping server on your computer. Step 1: Add the Spidra MCP server Open your terminal and run: claude mcp add --transport http spidra https://mcp.spidra.io/mcp This registers Spidra as a remote MCP server in Claude Code. The command uses HTTP transport, which Claude Code recommends for remote MCP servers. By default, Claude Code saves the configuration with local scope, making the server available in your current project. To use Spidra across your projects, add --scope user when registering the server. Step 2: Authenticate your Spidra account Run: claude mcp login spidra Follow the authentication flow in your browser and authorize access to your Spidra account. You can also complete authentication from inside Claude Code. Start a session: claude Then enter: /mcp Select Spidra and follow the sign-in instructions if authentication is required. Spidra's hosted MCP server supports browser-based authentication, so you don't need to manually add an API key for this setup. Step 3: Verify the connection Run: claude mcp list This displays your configured MCP servers and their connection statuses. Look for Spidra in the list. A healthy connection should show Connected . You can also inspect the configuration directly: claude mcp get spidra If the server is registered but requires authentication, complete the login process before using its tools. For additional configuration options, see the Claude Code MCP documentation https://code.claude.com/docs/en/mcp . Once connected, you can start using Spidra through ordinary prompts. Search for Python documentation using Spidra Suppose you're building a Python application and need to understand how HTTP request timeouts work in the Requests library. You could manually search the web and paste documentation into Claude Code. Instead, ask Claude Code to search for the relevant pages. Use this prompt: Use Spidra Search to find the official Python Requests documentation about HTTP timeouts. Return the top 5 results with their titles, URLs, and descriptions. Show the Spidra tool used and the actual search results. Do not generate or invent results. The captured Claude Code response identified spidra search as the tool and reported the query: Python Requests documentation timeouts It requested five results from the web source without a domain filter. What did Spidra Search return? The returned results contained five pages: | | Result | Source | |---|---|---| | 1 | Timeout in requests.get | Python Discussions | | 2 | Session Objects | Requests documentation | | 3 | Timeout for Python requests.get entire response | Stack Overflow | | 4 | Quickstart | Requests documentation | | 5 | Guide to Handling Python Requests Timeout | Oxylabs | Two results came from the official Requests documentation website. The first was the Advanced Usage documentation https://requests.readthedocs.io/en/master/user/advanced/ , which explains session behavior and request timeout configuration. The second was the Requests Quickstart https://requests.readthedocs.io/en/latest/user/quickstart/ , which covers timeout settings and explains that a timeout doesn't necessarily limit the entire response download. The remaining results were relevant to the search query but came from third-party sources. No search errors were reported. The result illustrates a useful distinction: a webpage can be relevant to your query without being an authoritative source. For development tasks that depend on official documentation, you should verify the returned URLs before using their contents. Filter search results to official sources The original search didn't include a domain filter, which explains why it returned documentation alongside discussions and blog posts. Spidra Search supports including and excluding domains. For a more targeted search, you can ask Claude Code: Use Spidra Search to find documentation about HTTP request timeouts. Apply the available domain filter to requests.readthedocs.io and return only results from that domain. Verify the hostname of each returned URL. This is a follow-up prompt to try, not a second completed test. Domain filtering helps narrow the results, but you should still verify that the returned pages contain the information your application needs. For more search options, see our guide to Spidra Search https://spidra.io/blog/spidra-search-api . Scrape a JavaScript-rendered page with Claude Code Search helps you discover URLs. But when you already know the webpage, you can retrieve its contents directly. Let's use Quotes to Scrape https://quotes.toscrape.com/js/ , a demonstration website designed for practicing web scraping. The JavaScript version loads its quote content through the browser, making it useful for demonstrating rendered-page extraction. Ask Claude Code: Use Spidra to scrape https://quotes.toscrape.com/js/ https://quotes.toscrape.com/js/ and extract the first 10 quotes with their authors as structured JSON. Show which Spidra tool you used, the actual returned records, any errors, and the number of complete records. Don't invent missing information. Claude Code identified spidra scrape as the tool used. Its response reported browser rendering with scrapeMode: "default" and JSON extraction using the requested fields. Extract quotes as structured JSON For an application, retrieving the page content is only part of the task. A quotes application might need each quote's text and author as separate fields that can be stored in a database. The extraction request specified two fields: - author : the author of the quote. - text : the quote itself. The result contained 10 records, each with both fields populated. Here's an excerpt from the returned JSON: { "quotes": { "author": "Albert Einstein", "text": "“Try not to become a man of success. Rather become a man of value.”" }, { "author": "André Gide", "text": "“It is better to be hated for what you are than to be loved for what you are not.”" } } The complete output contained 10 quote records. All 10 had nonempty author and text fields. Claude Code reported no scraping errors. The result demonstrates how structured extraction can turn content from a JavaScript-rendered page into records an application can process. It also reveals something worth checking before storing the data. Normalize extracted values before using them One record returned an encoded author name: André Gide The intended name is: André Gide The JSON structure was valid, and the author field was populated, but the value still needed normalization. Python's built-in html module can decode HTML entities. Run: python from html import unescape author = "André Gide" print unescape author Output: André Gide You can apply the same normalization when processing the full collection of extracted records. For application data, also check for missing fields, duplicate records, unexpected types, and values that don't match the source page. The quotes example returned 10 structurally complete records. That doesn't independently establish that every extracted value was correct. If your application depends on consistent fields, see our guide to extracting structured data from websites https://spidra.io/blog/scrape-any-website-structured-data-single-api-call . Combine web search and scraping in one workflow The previous examples used Search and Scrape separately. You can also combine them in one Claude Code task. Suppose you're building a reference application for Python HTTP clients. You want to collect documentation about request timeouts from Requests, HTTPX, and aiohttp, but you don't have the relevant URLs. Give Claude Code this prompt: I'm building a reference for Python HTTP clients. Use Spidra to find the official documentation for Requests, HTTPX, and aiohttp. For each library: Find its official documentation website. Locate the page explaining HTTP request timeouts. Scrape the relevant page. Extract the documented timeout behavior. Include the source URL with each record. Return the information as structured JSON. Use only the retrieved documentation. If a value cannot be verified, return null instead of guessing. Claude Code can use Spidra Search to discover relevant documentation pages, followed by Scrape to retrieve and extract their content. The resulting records can then be used to generate application code, create a reference file, or populate a development dataset. This combined workflow is an additional example to try. It wasn't part of the two completed demonstrations above. When should you use Search, Scrape, or Crawl? Choose the operation based on what you already know about the target. Use spidra search when you have a query but no URL. Use spidra scrape when you already know the webpage and need its content or structured fields. For several known URLs, spidra batch scrape can process them together. If you need to discover URLs within a website, spidra map can help identify them before scraping. For broader website extraction, spidra crawl can discover and process relevant pages. You don't need to crawl an entire documentation website when you only need two pages. Our guide to mapping websites before scraping selected pages https://spidra.io/blog/map-website-batch-scrape-selected-pages covers this workflow in more detail. Fix common MCP and scraping issues Claude Code cannot find Spidra Check your configured MCP servers: claude mcp list If Spidra isn't listed, register it again using the setup command. If it is listed but disconnected, inspect its status using /mcp inside Claude Code. Spidra needs authentication Run: claude mcp login spidra Complete the browser authentication flow. Avoid placing API keys in Claude Code prompts or committing credentials to a repository. Search returns unrelated or nonofficial results Narrow the query and use available domain filters when necessary. Verify the source URLs before treating search results as authoritative documentation. Scraping returns incomplete content Check whether the target page requires JavaScript rendering, browser interactions, or authentication. Spidra supports browser rendering and interactions, but successful extraction isn't guaranteed for every website. Inspect the returned content and error information before retrying. Claude Code answers without using Spidra Make the tool requirement explicit: Use Spidra Search for this task. Show the retrieved source URLs and identify anything you couldn't verify. If Claude Code cannot access Spidra, check the MCP connection instead of accepting an answer that hasn't been retrieved from the requested source. When should you use MCP instead of a direct scraping API? MCP is useful when Claude Code needs to choose and invoke tools while working on a larger development task. For example, you might ask it to discover documentation, retrieve the relevant pages, and create application code using the extracted information. A direct scraping API may be simpler when the task is predictable. If your backend already knows which URLs to retrieve and which fields it needs, calling the API directly avoids requiring an AI assistant to coordinate each request. You can use MCP during development to explore a workflow and establish the required data structure, then use the Spidra API or SDK for recurring production operations. Also treat retrieved webpage content as untrusted data. Websites may contain instructions intended to influence an AI assistant, so don't let scraped content override your application's instructions or permissions. Before scraping third-party websites, review their access restrictions, applicable terms, and privacy requirements. Our web scraping legal guide https://spidra.io/blog/is-web-scraping-legal covers the main considerations. Start using Spidra MCP with Claude Code Connecting Spidra through MCP lets Claude Code discover webpages, retrieve rendered content, and extract structured information as part of a coding workflow. The examples in this tutorial show both sides of that process. Search returned relevant documentation and third-party sources for a Python question, while Scrape returned structured quote records from a JavaScript-rendered website. Use Search when you need to discover pages, Scrape when you know the URL, and validate the returned values before relying on them. Create a Spidra account https://app.spidra.io/ and connect the hosted MCP server to try these examples with websites you're permitted to access.