Testing webclaw: A Firecrawl Alternative for Web Extraction in AI Agents

Webclaw, an open-source Rust web data extraction toolkit positioned as a drop-in Firecrawl alternative, has drawn over 2.3k GitHub stars and is available both self-hosted and as a managed cloud service. The project exposes 10+ REST endpoints for scraping, crawling, extraction, search, mapping, batch processing, summarization, research, brand, diff, and lead enrichment, plus Firecrawl-compatible endpoints, and ships as a CLI, a REST API server, Python/TypeScript/Go SDKs, and an MCP server that exposes 10+ tools to Claude, Cursor, Windsurf, Claude Code, Codex, and Antigravity. webclaw outputs pages as Markdown, JSON, plain text, or an LLM-optimized format, renders JavaScript, parses PDF, DOCX, XLSX, and CSV files, tracks content changes via diff, and supports proxy rotation from a proxies.txt file.

webclaw is a modern alternative to traditional web scrapers. Written in Rust and available as open source or a managed cloud service, it makes it easier to feed live web data into AI applications without reinventing the scraping stack every time. In this hands-on post, I’ll put webclaw to the test and share my final take webclaw Explained: What It Is and What It Offers webclaw https://webclaw.io/ is a web data extraction toolkit built in Rust that acts as a drop-in Firecrawl alternative. Its core mission is straightforward: turn any URL into clean, LLM-ready content. What sets it apart is that it’s designed specifically for AI agents and LLM pipelines. The GitHub repository of the project boasts over 2.3k stars https://github.com/0xMassi/webclaw . You can either self-host it or use it via the API through the cloud version. Main Features The core capabilities exposed by WebClaw include: - 10+ REST endpoints : Scrape, crawl, extract, search, map, batch, summarize, research, brand, diff, and lead enrichment endpoints + Firecrawl-compatible endpoints available through the same API interface. - Multiple output formats : Get web pages as Markdown, JSON, plain text, or an LLM-optimized format with links and other metadata. - JavaScript rendering : The CLI version can access pages that require JavaScript rendering. - Document parsing : Extract text from PDFs, DOCX, XLSX, and CSV files. - Content change tracking diff : Compare current snapshots with previous extractions to detect what has changed. - Proxy rotation support : Automatically loads proxies.txt from your working directory for automated proxy rotation https://www.scraping.club/p/use-ipv6-scraping-nyxproxy . - Browser actions : Click, type, scroll, wait, take screenshots, press keys, and execute custom JavaScript before extraction. - Direct integration with AI agents and harnesses : Exposes 10+ tools over stdio for Claude, Cursor, Windsurf, Claude Code, Codex, and Antigravity. Learn more in the official docs https://webclaw.io/docs . Architecture and Products webclaw’s architecture is straightforward, with the same underlying extraction engine for: - webclaw : A CLI for extracting, crawling, summarizing, and tracking changes from the terminal. - webclaw-server : A REST API server that provides access to webclaw through JSON endpoints. - SDKs : Libraries for Python, TypeScript, and Go that simplify integration with the webclaw REST API. - webclaw-mcp : An MCP server that exposes 10+ tools for use with any MCP-compatible AI client. As a result, whether you’re pulling web data through the CLI, REST API, an SDK, or the MCP server, the same core code runs under the hood. This unified architecture means you get consistent, predictable behavior whether you scrape from the terminal, call the REST API directly, integrate webclaw into your scripts, or use it via an AI agent. Let me introduce you to the three webclaw interfaces CLI The webclaw command-line tool is available through the webclaw command. It provides a range of options for extracting, crawling, and processing web content directly from the terminal. Common commands include: - webclaw <url : Extract a single URL to Markdown - webclaw https://docs.example.com --crawl --depth 2 --max-pages 100 : Crawl a website with depth and page limits - webclaw https://example.com --map : Discover URLs from a website’s sitemap.xml and robots.txt https://www.scraping.club/p/understanding-robotstxt-and-its-implications . - webclaw https://example.com --extract-json @schema.json : Extract structured JSON based on a schema using the configured LLM provider Ollama, OpenAI, or Anthropic . - webclaw https://example.com --diff-with snapshot.json : Detect changes since a previous snapshot. - webclaw https://example.com --include "article" --exclude "nav,footer" : Control which page elements are extracted using CSS selectors. - webclaw vertical amazon https://amazon.com/... : Use a site-specific vertical extractor https://webclaw.io/docs/api/vertical , returning typed JSON instead of generic Markdown. - webclaw https://example.com --cloud : Route extraction through the webclaw cloud For the complete list of commands and options, see the webclaw CLI documentation https://webclaw.io/docs/cli . Common use cases include terminal-based web data collection, automation workflows, and one-off scraping tasks. One of the CLI’s main advantages is that you can pipe commands together. For example, you can pipe a JSON response through jq to extract only the fields you need. REST API webclaw comes with an axum-based HTTP server https://docs.rs/axum/latest/axum/ that handles authentication, CORS, gzip compression, and asynchronous job management. The main endpoints https://webclaw.io/docs/api include: The workflow is simple: send a POST request with a URL and your desired options, and webclaw returns the extracted data in the requested format. For users who prefer a managed solution, the cloud version of the APIs is available at api.webclaw.io https://webclaw.io/products/api . This is managed directly by webclaw and offers automatic scaling, proxy rotation, and other premium features out of the box. SDKs webclaw provides official SDKs for Python, TypeScript, and Go. They simplify integration with the cloud REST API, while offering typed clients, built-in error handling, and crawl polling. All SDKs support the full API surface and authenticate using a webclaw API key. In detail, the Python SDK available as the webclaw PyPI package https://pypi.org/project/webclaw/ works like this: python pip install webclaw from webclaw import Webclaw with Webclaw "YOUR WEBCLAW API KEY" as client: Scrape the "https://example.com" page via webclaw's cloud APIs result = client.scrape "https://example.com", formats= "markdown" Print the returned Markdown version of the page print result.markdown The TypeScript and Go SDKs provide equivalent functionality for their respective ecosystems. MCP Server The MCP server exposes 14 tools over STDIO https://modelcontextprotocol.io/specification/2026-07-28/basic/transports/stdio : The easiest way to set up webclaw with Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, or another MCP-compatible client is to run: npx create-webclaw Alternatively, you can configure individual clients manually. For Claude Code, launch: claude mcp add webclaw -- npx -y @webclaw/mcp For Claude Desktop, add the following to your claude desktop config.json configuration: { "mcpServers": { "webclaw": { "command": "npx", "args": "-y", "@webclaw/mcp" , "env": { "WEBCLAW API KEY": "<YOUR WEBCLAW API KEY " } } } } The MCP server uses a local-first architecture. When webclaw detects a bot-protected or JavaScript-heavy page, it automatically falls back to the webclaw cloud API if a WEBCLAW API KEY is configured. Deployment Options webclaw comes with two deployment models: - Self-hosting https://webclaw.io/docs/self-hosting : A single open-source Rust binary that exposes the extraction engine through a local HTTP server. - Cloud API https://webclaw.io/docs/cloud : A fully managed service, where you sign up, create an API key, and start extracting data by calling webclaw’s remote APIs. Explore the pros and cons of each option Self-Hosting webclaw’s full Rust stack is available under the AGPL-3.0 license https://opensource.org/license/agpl-3-0 and can be self-hosted using Cargo installs or via Docker images. The self-hosted version also supports Ollama, allowing you to run the LLM-powered extract and summarize operations entirely on your own hardware. 🎯 Best for : - Users who want unlimited local usage without a credit system. - Sites that don’t require JavaScript rendering and don’t involve bot protection. - Users who need full control over your data and infrastructure. ⬇️ Drawbacks : - You’re responsible for infrastructure, monitoring, maintenance, and scaling. - Synchronous crawls are limited to 500 pages. - You don’t get built-in bot-protection handling, JavaScript rendering, or proxy rotation. Cloud API The webclaw Cloud API is a fully managed service, so there are no servers to run or infrastructure to maintain. Compared to the self-hosting option, it’s suitable for more demanding scraping workloads. 🎯 Best for : - Sites that require JavaScript rendering. - Users who need scalable web data collection and don’t want to manage the infrastructure themselves. - Teams that want usage tracking, billing, and predictable monthly quotas. ⬇️ Drawbacks : - Usage is subject to monthly page quotas based on your plan. - Your data is processed through the webclaw cloud infrastructure. - Monthly rate limits based on your plan. webclaw in Action: Getting Started In this section, I’ll walk through practical examples of using webclaw through its CLI and REST API. Keep in mind that you can achieve the same results through the SDKs or directly from AI agents using MCP integration. Disclaimer : webclaw provided me with a Scale plan with 1,000,000 credits to thoroughly test the product. Installation and Setup On a system with a working Rust toolchain, run this command to install the webclaw CLI: cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli Once the installation is complete, verify it by running: webclaw --version Alternatively, for a simpler setup, you can run webclaw directly with Docker: docker run --rm ghcr.io/0xmassi/webclaw "<TARGET URL " To check that the Docker setup is working, run: docker run --rm ghcr.io/0xmassi/webclaw --version In both cases, the command should return: For more guidance, refer to the “Getting Started” guide https://webclaw.io/docs/getting-started . To use webclaw’s Cloud API features, create an account https://webclaw.io/login . On the “API Keys” page of the dashboard, you’ll find a default API key that was automatically created when your account was set up: Alternatively, click “Create key” and follow the setup wizard to generate a new webclaw API key. Example 1: Scrape a Web Page in Multiple Formats Suppose you want to scrape the “ Quotes to Scrape https://quotes.toscrape.com/ ” website. Achieve that with the webclaw CLI as below: webclaw "https://quotes.toscrape.com/" Note : If you’re a Docker user, remember to replace webclaw with docker run --rm ghcr.io/0xmassi/webclaw . By default, webclaw will return Markdown content: Now, assume you want to access the same page in JSON format. Specify the --format "json" option: webclaw "https://quotes.toscrape.com/" --format "json" This time, you will get a structured JSON response containing the web page in Markdown: Notice how it also contains a links array with all the URLs on the page. Finally, test the special LLM-ready output: webclaw "https://quotes.toscrape.com/" --format "llm" The result will look like this: The produced data format combines the structured metadata of the JSON output with the readability of Markdown. It’s optimized for LLM consumption, removing unnecessary line breaks and whitespace to reduce token usage while keeping the content structured and easy for AI models to process. Note that you can achieve the same result by calling the webclaw Cloud API like this: curl -X POST https://api.webclaw.io/v1/scrape \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" \ -d '{"url":"https://quotes.toscrape.com/","format": "markdown" }' When you pass the output through a JSON beautifier/parser, the result will be: Example 2: Access a Dynamic Web Page Requiring JavaScript Rendering Now, target a webpage that requires JavaScript rendering, such as the JavaScript version of “ https://quotes.toscrape.com/js/ Quote to Scrape https://quotes.toscrape.com/js/ ” https://quotes.toscrape.com/js/ : webclaw "https://quotes.toscrape.com/js/" The result will be: Notice that the returned Markdown contains only the page’s static content. It doesn’t include the content dynamically added by JavaScript. To scrape JavaScript-dependent pages, you must rely on the webclaw Cloud API: curl -X POST https://api.webclaw.io/v1/scrape \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" \ -d '{"url":"https://quotes.toscrape.com/js/"}' This time, the markdown field contains the complete page, including the content rendered dynamically by JavaScript: Note : In my tests, the Cloud API wasn’t able to scrape any pages that the CLI couldn’t already scrape. At the time of testing, this suggests that the anti-bot bypass functionality may not be working as expected. Example 3: Scrape with CSS Selector Filtering Suppose you want to extract only a particular section of a webpage: You can use the --include option to select specific elements or --exclude to filter them out . Here, you only want to extract the prodInfos element. So, write: webclaw "https://world.openfoodfacts.org/product/6111242100992/perly-jaouda" --include " prodInfos" The result will contain only that specific section of the page: You can achieve the same outcome with the Cloud API by using the include selectors field in the request body: curl -X POST https://api.webclaw.io/v1/scrape \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" \ -d '{"url":"https://world.openfoodfacts.org/product/6111242100992/perly-jaouda","include selectors": " prodInfos" }' Now, suppose you want to scrape a webpage with a header and footer, such as a blog post. In this case, you are generally interested only in the main body content. webclaw supports that use case via the --only-main-content option: webclaw "https://world.openfoodfacts.org/product/6111242100992/perly-jaouda" --only-main-content Similarly, you can use the only main content field with the Cloud API: curl -X POST https://api.webclaw.io/v1/scrape \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" \ -d '{"url":"https://world.openfoodfacts.org/product/6111242100992/perly-jaouda","only main content":true}' Example 4: Crawl a Site webclaw supports Breadth-First Search BFS https://www.codecademy.com/article/breadth-first-search-bfs-algorithm crawling, allowing you to discover and scrape multiple pages starting from a single URL. For example, you can crawl up to 10 pages from Quote to Scrape with a maximum depth of 3 : webclaw "https://quotes.toscrape.com/" --crawl --depth 3 --max-pages 10 The command will return the discovered pages along with the Markdown content extracted from each one: You can perform the same crawl through the Cloud API: curl -X POST "https://api.webclaw.io/v1/crawl" \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" \ -d '{"url": "https://quotes.toscrape.com/","max depth": 3,"max pages": 10}' Unlike the CLI command, the Cloud API processes the crawl asynchronously. The initial request returns an ID for the crawl task: You can then use this ID to poll for the crawl status: curl "https://api.webclaw.io/v1/crawl/88e08765-f44a-4e26-a4d6-3552bef983cb" \ -H "Authorization: Bearer <YOUR WEBCLAW API KEY " \ -H "Content-Type: application/json" Once the crawl is complete, the response will look similar to this. The pages array contains the content extracted from all successfully scraped pages: Pricing Options and Plans As of this writing, webclaw’s pricing plans can be summarized as follows: All cloud plans can be billed monthly or annually, with annual billing offering a 20% discount. Important notes : - The credit pool is shared across all API endpoints. - A free tier with 3 runs per day is available indefinitely, with no credit card required. - The Starter plan includes a card-required 7-day free trial, giving you temporary access to the plan before you commit. - Self-hosting provides unlimited local usage with no credit system. The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. Final Thoughts on webclaw webclaw is definitely an interesting solution, and I personally found the CLI easy to use. The API interface is also quite intuitive, and its Firecrawl compatibility is definitely a plus. The project is still young the first commit was made in March 2026 , and this shows in a few areas e.g., unexpected behavior of input parameters or defaults . I also found some inconsistencies between the GitHub page and the official website and documentation. Still, that’s fairly common with fast-evolving projects, I have to say. That’s why technical writers like me still have a job… for now, at least 😉 I especially appreciated the support from Valerio Massimiani https://www.linkedin.com/in/valerio-massimiani/ , the founder of webclaw. He provided me with a Scale plan, remained available to answer my questions, took care of my FAQ requests, and was responsive throughout the process. On the other hand, I found the self-hosting setup quite difficult, especially on Windows and when installing via Cargo. Docker, by comparison, was super easy. I also found the overall anti-bot experience somewhat inconsistent. webclaw was able to bypass challenging sites such as L’Occitane, Temu, and Nike even from the CLI. However, when the CLI failed because of anti-bot protection e.g., on Cloudflare-protected sites, Hermès, Canada Goose, and others , the cloud version failed as well. At least, in my tests… That being said, the project has plenty of potential and is clearly moving forward quickly. I’ll definitely keep an eye on it and see how it develops, especially given how interesting the overall concept is and how easy it is to integrate webclaw with local AI agents. Did you like this article? Share it with someone who might find it useful and get a discount on paid plans. FAQ Disclaimer : The following FAQ questions were answered by Valerio Massimiani, the founder of the project. How does webclaw differ from Firecrawl? webclaw is written in Rust and skips the browser for most pages, so it’s faster and cheaper to run. Output is LLM-shaped by default, and an MCP server lets agents call it directly. Firecrawl-compatible /v2 endpoints mean existing Firecrawl code can point at it unchanged. Can I use webclaw with my existing web scraping code? Yes. There are REST endpoints plus official Python, TypeScript, and Go SDKs, so it drops into existing pipelines. If you already run Firecrawl, the /v2 compatibility layer covers scrape, crawl, map, and search, so most code moves by changing the base URL and key. Does webclaw come with official agent skills? Yes. npx skills add 0xMassi/webclaw-skill installs two: a webclaw skill covering the MCP tools, and a lead-enrichment skill that runs discovery and enrichment over /v1/extract . Both work in Claude Code and any agent that reads the skills format. Does webclaw support proxy integration? Yes. Self-hosted, pass --proxy for a single upstream or --proxy-file to rotate a pool per request; both also read from environment variables. HTTP and SOCKS5 are supported. On the managed API, proxying is handled for you and needs no configuration. What’s the difference between the webclaw CLI and the REST API? Same extraction engine, different entry point. The CLI runs locally, needs no key for most sites, and suits one-off scrapes and scripts. The REST API adds authentication, async jobs for crawl and research, and managed handling of protected pages. The output is identical. How does the LLM optimization actually work? The llm format strips navigation, footers, scripts, and tracking markup, then flattens what remains into compact prose with headings and links preserved. The result is a fraction of the original token count. Schema-based extraction goes further and returns typed JSON you can validate. Is my data private with the webclaw Cloud API? Content is fetched per request and returned to you. Results may be cached briefly so repeat requests for the same URL are faster. Public pages aren’t private data, but don’t route credentials through the API. Self-host if you need full control of the pipeline.