{"slug": "show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents", "title": "Show HN: Webcrawl, an open-source web crawling API and MCP server for agents", "summary": "Brainfish AI released Webcrawl, an open-source web crawling API and MCP server that lets AI coding agents such as Claude Code, Cursor, Codex, Gemini and Windsurf search, scrape, map, crawl and monitor websites without an API key or account. The tool installs eight Agent Skills via `npx skills add brainfish-ai/webcrawl-anything`, registers the `@webcrawl/mcp` server through `npx webcrawl setup`, and returns structured Markdown across targets including JavaScript SPAs, PDFs and Office documents, YouTube transcripts, product and travel listings, CVE records with CVSS v3.1/v4.0 scores and CISA KEV flags, GitHub repositories, ArXiv papers and llms.txt files. Webcrawl runs locally using `impit` TLS impersonation and a local Chrome instance, with an optional cloud stealth browser at webcrawl.app and optional SERPER_API_KEY for shopping extraction.", "body_md": "**Give your AI agent the whole web. Free, local, no API key.**\n\nSearch, scrape, map, crawl and monitor any website, SPA, PDF, YouTube video, GitHub repo, ArXiv paper, product page or CVE, from Claude Code, Cursor, Codex or your terminal.\n\n<sub>Every frame above is real output, recorded with `node scripts/record-demo.mjs`. No API key, no account.</sub>\n\n```\nnpx skills add brainfish-ai/webcrawl-anything\n```\n\nThat installs all eight Webcrawl [Agent Skills](#agent-skills-any-coding-agent) into Claude Code, Cursor, Codex, Gemini, Windsurf or any agent that reads `SKILL.md`. Only want one? Pick it by name:\n\n```\nnpx skills add brainfish-ai/webcrawl-anything --skill webcrawl-deal-watch\n```\n\nTo also register the `@webcrawl/mcp` server, so your agent can call the crawl, search and monitor tools directly:\n\n```\nnpx webcrawl setup\n```\n\nRestart your agent and paste one of these:\n\n```\nIs CVE-2024-3094 in anything I depend on? Look it up, give me the CVSS score, CISA KEV\nstatus and fixed versions, then check my package-lock.json and link the evidence.\nMap https://docs.stripe.com/webhooks, crawl every page under it into Markdown, and\nwrite me a one-page cheat sheet with the source URL under each section.\nWatch https://news.ycombinator.com every hour and alert me when a story about MCP\nreaches the front page. Show me the monitor you created.\n```\n\nMore reproducible prompts, each with the command it runs: **[docs/demos.md](https://github.com/brainfish-ai/webcrawl-anything/blob/main/docs/demos.md)**.\n\nNo agent? Use the CLI directly: `npx webcrawl search \"react 19 compiler\"`.\n\n| Target | Returns | Requirements | \n|---|---|---|\n| **Websites & JS SPAs** (`web` ) | Clean Markdown, semantic headings, tables, code blocks, links, OpenGraph/JSON-LD, and SHA-256 content hashes | Zero-key local ( `impit` TLS impersonation + local Chrome) or[`webcrawl.app`](https://webcrawl.app) cloud stealth browsers | \n| **PDF & Office Docs** (`pdf-office` ) | Page-numbered Markdown, tables, slide notes, and automatic OCR on scanned PDF pages ( `pagesOcr` + honest`pagesSkipped` ) | Built-in PDF inspector plus `pdf-parse` ,`mammoth` ,`officeparser` , and`tesseract.js` | \n| **YouTube & Media** (`youtube-media` ) | Timestamped transcript Markdown, video title/channel/duration metadata, and spoken audio transcription | Built-in zero-key YouTube multi-client caption extractor; optional audio transcription provider for raw MP3/MP4 | \n| **Shopping & Products** (`shopping` ) | Multi-retailer price comparison, currency-normalized offers, stock status, ratings, review counts, and shipping | Zero-key local HTML/JSON-LD product extractor ( `-c shopping` ); optional`SERPER_API_KEY` | \n| **Flights & Hotels** (`travel` ) | Structured flight routes ( `origin` ,`destination` ,`airline` ,`stops` ,`price` ) and hotel nightly rates, stars, and guest ratings | Zero-key local search + schema.org extractor ( `-c flights` /`-c hotels` ) | \n| **Real Estate & Local Business** (`real-estate` ) | Property listings ( `bedrooms` ,`bathrooms` ,`parking` ,`suburb` ,`portal` ) and local business hours, phone, address, and reviews | Zero-key local extractor ( `-c real-estate --sub buy\\|rent\\|sold` /`-c business` ) | \n| **Security & CVEs** (`cve` ) | Authoritative CVE & GHSA records with CVSS v3.1/v4.0 scores, CWEs, affected/fixed package versions, and CISA KEV flags | Built-in live NVD 2.0 + OSV.dev + CISA KEV enrichment ( `-c security --sub cve` ) | \n| **GitHub Repos, Issues & Releases** (`github` ) | Structured repo metadata (stars, forks, license, topics, language), README Markdown, or issue/PR/release discussion threads | Built-in zero-key HTML & GitHub extractor ( `npx webcrawl scrape https://github.com/owner/repo` ) | \n| **ArXiv & Academic Papers** (`arxiv` ) | Paper title, authors, abstract, primary category, submission date, direct PDF link, and full parsed paper Markdown | Built-in ArXiv abstract & PDF inspector ( `npx webcrawl scrape https://arxiv.org/abs/...` ) | \n| **llms.txt & Sitemaps** (`llms-txt` ) | Structured documentation manifest, section hierarchy, and crawlable doc URLs | Built-in ( `npx webcrawl scrape https://docs.example.com/llms.txt` or`npx webcrawl map` ) | \n\nRun `npx webcrawl targets` at any time to inspect all registered targets from your terminal.\n\nOne command registers the `@webcrawl/mcp` server and installs the `webcrawl-anything` Agent Skill (`SKILL.md`) for Claude Code, Cursor, and Codex:\n\n```\nnpx webcrawl setup\n```\n\nOr target specific coding agents:\n\n```\nnpx webcrawl setup --agent claude --agent cursor --agent codex --agent gemini --agent windsurf\n```\n\nCheck your local engine, stealth browser, and parser readiness:\n\n```\nnpx webcrawl doctor\n```\n\nInstall webcrawl's skills into Claude Code, Cursor, Codex, or any agent that supports the [Agent Skills](https://agentskills.io/specification) spec:\n\n```\nnpx skills add brainfish-ai/webcrawl-anything\n```\n\nOr install a single skill, for example the Black Friday price-drop watcher:\n\n```\nnpx skills add brainfish-ai/webcrawl-anything --skill webcrawl-deal-watch\n```\n\n| Skill | What it does | \n|---|---|\n| **`webcrawl-ticket-watch`** | Watch StubHub/Ticketmaster/SeatGeek event pages for price drops & seat availability — get a push alert on your phone the moment tickets hit your target. | \n| **`webcrawl-jobs-leads`** | Job-market intelligence: crawl company careers pages and Greenhouse/Lever boards, normalize postings, track new listings daily, rank fit. | \n| **`webcrawl-deal-watch`** | Black Friday / price-drop monitor for your personal watchlist — pings your phone (free ntfy.sh push) when an item hits target price. | \n| **`webcrawl-crawl`** | Any URL → clean Markdown: scrape, crawl, map, extract, categorized search. The core skill. | \n| **`webcrawl-protected-sites`** | Read sites that challenge automated clients (DataDome/Cloudflare/AWS-WAF): TLS-impersonate → proxy → real-browser tier ladder, zero config to start. | \n| **`webcrawl-learn-topic`** | Deep-research any topic: search → crawl authoritative sources → cited digest with a reading list. | \n| **`webcrawl-research-corpus`** | Build RAG-ready Markdown corpora from docs sites: llms.txt, sitemaps, path scoping, hash dedupe. | \n| **`webcrawl-real-estate`** | Search & monitor property listings with the built-in extractor — new-listing and price-reduction alerts. | \n\nSee [docs/skills.md](https://github.com/brainfish-ai/webcrawl-anything/blob/main/docs/skills.md) for details and trigger phrases per skill.\n\nWebcrawl Anything runs **out-of-the-box locally with zero API keys** using the built-in `impit` TLS-impersonated fetcher, document/media parsers, BM25 + semantic ranker, and vertical extractors:\n\n```\n# Scrape any website, GitHub repo, ArXiv paper, PDF, or YouTube video into clean Markdown\nnpx webcrawl scrape https://github.com/torvalds/linux\nnpx webcrawl scrape https://arxiv.org/abs/1706.03762\nnpx webcrawl scrape https://www.youtube.com/watch?v=dQw4w9WgXcQ\n\n# Search the live web or structured verticals\nnpx webcrawl search \"react 19 compiler\" -n 5\nnpx webcrawl search \"sony wh-1000xm5\" -c shopping\nnpx webcrawl search \"boutique hotels in shibuya\" -c hotels\nnpx webcrawl search \"3 bedroom house in marrickville\" -c real-estate --sub buy\nnpx webcrawl search \"CVE-2024-3094 xz utils\" -c security --sub cve\n\n# Discover every URL on a domain (sitemap.xml + robots.txt + link graph)\nnpx webcrawl map https://docs.stripe.com --filter webhook -n 50\n\n# Crawl up to 25 pages in parallel with target evidence\nnpx webcrawl crawl https://docs.stripe.com/webhooks -n 5\n\n# Extract structured JSON from any page\nnpx webcrawl extract https://news.ycombinator.com --prompt \"top 5 stories with points\"\n```\n\nOnce `npx webcrawl setup` has registered the MCP server and skill, ask Claude Code, Cursor, or Codex directly:\n\n- *\"Search for CVE-2024-3094 in the security category and show me the CVSS score, CISA KEV status, and fixed versions.\"*\n- *\"Scrape `https://arxiv.org/abs/2305.18290` and summarize the DPO objective and experimental results.\"*\n- *\"Compare retail prices and stock availability for the Apple Mac Mini M4 across US retailers using `web_search` with `category: shopping`.\"*\n- *\"Map `https://modelcontextprotocol.io`, pick the specification pages, and crawl them into Markdown.\"*\n- *\"Parse the PDF at `https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf` and verify whether any pages required OCR.\"*\n\n### Local Engine vs. [`webcrawl.app`](https://webcrawl.app) Cloud\n\n`webcrawl.app`\nWebcrawl Anything uses the **exact same TypeScript engine (`src/`)** locally and in the hosted [` webcrawl.app`](https://webcrawl.app) cloud API. You can start with zero keys locally and upgrade to cloud with one command whenever you need managed browsers or sub-50ms global cache hits:\n\n| Capability | Open-Source Local ( `--mode local` ) | [`webcrawl.app`](https://webcrawl.app) Cloud (`--mode cloud` ) | \n|---|---|---|\n| **API Key Required** | **None (Zero-Key)** | `WEBCRAWL_API_KEY` (`npx webcrawl login` ) | \n| **Core Engine & Target Plugins** | 100% shared ( `src/engine` ,`src/targets` ,`src/extract` ) | 100% shared | \n| **PDF, Office, YouTube & CVE Parsers** | Built-in local execution | Hosted + background async workers ( `/v1/runs/:id` ) | \n| **JavaScript-heavy pages** | Local `impit` TLS + local Chrome | Managed browser fleet + Cloudflare rendering | \n| **Search Fan-Out** | Built-in DuckDuckGo/Bing plus optional API keys | Managed multi-provider fan-out | \n| **Global Content Cache** | Local in-memory LRU | Sub-50ms global SHA-256 content-addressed cache | \n\nStatic pages never launch Chrome. JavaScript pages use a CDP-driven local Chrome session with a readiness wait, a hard timeout, and fallback flags. If rendering cannot happen, the scrape is `success: false` (`needsJs`). Extract strips script and style tags and supports nested schemas. Search fans out across keyless sources and fills up to `n`. See [docs/engine.md](https://github.com/brainfish-ai/webcrawl-anything/blob/main/docs/engine.md).\n\nTo connect your CLI and MCP server to [`webcrawl.app`](https://webcrawl.app) cloud:\n\n```\nnpx webcrawl login\nnpx webcrawl whoami\n```\n\nBy default, `npx webcrawl setup` configures `WEBCRAWL_MCP_SCHEMA_PROFILE=compact` so MCP tool schemas use minimal token overhead inside Claude Code, Cursor, and local LLM context windows while keeping full server-side validation.\n\nTo switch to verbose tool parameter descriptions:\n\n```\nnpx webcrawl setup --schema-profile full\nwebcrawl/\n├── src/\n│   ├── targets/        # Pluggable Target & Extractor Registry (web, pdf-office, youtube, github, arxiv, llms-txt, shopping, travel, real-estate, cve)\n│   ├── engine/         # Multi-tier fetcher (impit TLS impersonation → stealth browser → HTML-to-Markdown + SHA-256 cache)\n│   ├── parse/          # Native PDF inspector + OCR fallback, DOCX/XLSX/PPTX parser, YouTube caption extractor\n│   ├── extract/        # Vertical structured extractors (shopping, flights, hotels, real-estate, business, CVE/NVD/OSV)\n│   ├── search/         # Multi-provider search fan-out + vertical pipelines\n│   ├── ranker/         # Hybrid BM25 + semantic vector + RRF + MMR diversity reranker\n│   └── local-engine.ts # In-process zero-key backend powering CLI & MCP locally\n├── packages/\n│   ├── cli/            # `webcrawl` CLI (`npx webcrawl search|scrape|map|crawl|extract|rerank|targets|setup|doctor`)\n│   ├── mcp/            # `@webcrawl/mcp` stdio & Streamable HTTP MCP server\n│   └── client/         # `@webcrawl/client` zero-dependency TypeScript SDK\n└── api/\n    └── index.ts        # Production `webcrawl.app` API powered by the exact same `src/` engine\n```\n\nEvery community improvement to a target plugin (`src/targets/`), vertical extractor (` src/extract/`), or document parser (` src/parse/`) immediately powers both local CLI/MCP users and the hosted [`webcrawl.app`](https://webcrawl.app) platform.\n\nBuilt something with webcrawl, like a price tracker, a docs corpus, a research agent, or a monitor that saved you a refresh? Open a [Show your build](https://github.com/brainfish-ai/webcrawl-anything/discussions/categories/show-and-tell) discussion with the prompt you used and a screenshot. We feature the best ones here and in release notes.\n\nAdd the badge to your project:\n\n```\n[![Built with webcrawl](https://img.shields.io/badge/built%20with-webcrawl-0a0a0a)](https://github.com/brainfish-ai/webcrawl-anything)\n```\n\nWant to add a new target (e.g., Reddit threads, Hacker News discussions, SEC EDGAR filings, npm/PyPI packages, or a regional real-estate/shopping portal)?\n\nSee **[CONTRIBUTING.md](https://github.com/brainfish-ai/webcrawl-anything/blob/main/CONTRIBUTING.md)** for the 3-step guide to registering a `CrawlTargetPlugin` in `src/targets/` with a fixture test in `src/__tests__/targets.test.ts`.\n\nNew here? Start with a [good first issue](https://github.com/brainfish-ai/webcrawl-anything/issues?q=is%3Aopen+label%3A%22good+first+issue%22) or a [target plugin request](https://github.com/brainfish-ai/webcrawl-anything/issues?q=is%3Aopen+label%3Atarget-plugin).\n\nEvery merged PR is credited by name in the release notes.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents", "canonical_source": "https://github.com/brainfish-ai/webcrawl-anything", "published_at": "2026-11-13 13:33:42+00:00", "updated_at": "2026-11-13 13:51:28.913323+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-crawlers"], "entities": ["Brainfish AI", "Webcrawl", "@webcrawl/mcp", "Claude Code", "Cursor", "Codex", "Gemini", "Windsurf"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents", "markdown": "https://wpnews.pro/news/show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents.md", "text": "https://wpnews.pro/news/show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-webcrawl-an-open-source-web-crawling-api-and-mcp-server-for-agents.jsonld"}}