{"slug": "show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server", "title": "Show HN: PyScrappy, self-healing web scraping selectors plus an MCP server", "summary": "PyScrappy, an AI-native web scraping toolkit that converts websites into structured, LLM-ready data, has been released as a Python library and MCP server, featuring self-healing selectors, JS rendering, and 20+ built-in scrapers. The toolkit, installable via pip, includes an MCP server for AI agents and a built-in agent for local models like Ollama, supporting tool calling.", "body_md": "PyScrappy is an AI-native web scraping toolkit that turns websites into structured, LLM-ready data. Use it as a Python library or expose it as an MCP server for AI agents.\n\n📖 **Documentation:** [pyscrappy.vercel.app](https://pyscrappy.vercel.app)\n\n**Generic scraper**— give it any URL, get back structured text, links, images, tables, and metadata** LLM-ready output**—`.to_markdown()`\n\nturns any result into clean Markdown; also`.to_json()`\n\nand`.to_dataframe()`\n\n**MCP server**— expose the scrapers as tools for AI agents (Claude, Cursor, local LLMs, …)** JS rendering**— optional Playwright backend for JavaScript-heavy sites** Custom selectors**— pass CSS selectors to extract exactly what you need** Chainable**— navigate HTML directly with CSS/XPath,`Selector`\n\n`find_all`\n\n,`find_by_text`\n\n, and`find_similar`\n\n(Scrapy/BeautifulSoup-style)**Adaptive (self-healing) selectors**— remember an element and relocate it by similarity when a site changes its markup, so scrapers don't silently break** Concurrent scraping**—`scrape_many`\n\n/`scrape_all`\n\nrun scrapes in parallel**Proxy & scraping-API support**— route through a proxy or ScraperAPI/ScrapeOps for blocked sites** TLS-fingerprint impersonation**—`impersonate=\"chrome\"`\n\ngets past anti-bot filters that block plain clients (optional`curl_cffi`\n\nbackend)**Command-line extract**—`pyscrappy extract <url> out.md`\n\nscrapes a URL straight to a file, no code**Retry & rate-limiting**— built-in exponential backoff and per-domain rate limiting** Type-safe**— full type hints,`py.typed`\n\nmarker**20+ built-in scrapers**— Wikipedia, IMDB, stocks, news, GitHub, Amazon/IKEA, YouTube, and[more](#built-in-scrapers)\n\n```\npip install pyscrappy\n```\n\n**Optional extras:**\n\n```\n# Browser support (for JS-rendered pages)\npip install 'pyscrappy[browser]'\nplaywright install chromium\n\n# DataFrame support\npip install 'pyscrappy[dataframe]'\n\n# MCP server (use PyScrappy's scrapers as AI-agent tools)\npip install 'pyscrappy[mcp]'\n\n# Stealth (TLS-fingerprint impersonation to bypass anti-bot filters)\npip install 'pyscrappy[stealth]'\n\n# Everything\npip install 'pyscrappy[all]'\n```\n\nPyScrappy ships an [MCP server](#mcp-server-use-pyscrappy-from-an-ai-agent) that\nexposes its scrapers as tools, so an agent (Claude, Cursor, an OpenAI agent, a\nlocal LLM) can pull structured web data from any URL and hand it straight to the\nmodel:\n\n```\nAI agent  ──MCP tool call──▶  PyScrappy  ──fetch + extract──▶  Any website\n   ▲                                                                │\n   └──────────────  clean Markdown / JSON  ◀───────────────────────┘\npip install 'pyscrappy[mcp]'\nclaude mcp add pyscrappy pyscrappy-mcp\n```\n\nThen just ask: *\"use pyscrappy to summarize the latest headlines from bbc.com.\"*\nSee [MCP server](#mcp-server-use-pyscrappy-from-an-ai-agent) for the full setup\nand tool list.\n\nOllama can't talk MCP on its own, so normally you'd run a host (Goose, Cline, …) in between. PyScrappy skips that with a built-in agent that talks to Ollama directly and lets a local model call the scrapers as tools:\n\n```\npip install 'pyscrappy[mcp]'                 # needs Python 3.10+\npyscrappy chat --model qwen2.5 \"what's the current AAPL quote?\"\n```\n\nIt exposes the same 22 tools as the MCP server. The only requirement is a model\nthat supports **tool calling** (Llama 3.1, Qwen 2.5, Mistral, …); how well it\n*picks* the right tool is up to the model. Point it at a remote Ollama with\n`--host`\n\n, and pass `-v`\n\nto see each tool call.\n\nPyScrappy ships an optional [Model Context Protocol](https://modelcontextprotocol.io)\nserver, so an AI agent (e.g. Claude) can call PyScrappy's scrapers as tools and\nget structured web data back.\n\n```\npip install 'pyscrappy[mcp]'\n```\n\nThe MCP extra installs the standalone `fastmcp`\n\npackage and requires Python 3.10\nor newer. On Python 3.9 the core scraping library still works, but the MCP server\nis unavailable.\n\nThis installs the `pyscrappy-mcp`\n\ncommand. It uses stdio by default for local MCP\nclients; Streamable HTTP and legacy SSE are available for remote deployments:\n\n```\npyscrappy-mcp          # stdio (default)\npyscrappy-mcp --http   # Streamable HTTP\npyscrappy-mcp --sse    # legacy SSE\n```\n\nYou can also run the stdio server with `python -m pyscrappy.mcp`\n\n.\n\n```\nclaude mcp add pyscrappy pyscrappy-mcp\n```\n\nAdd to your `claude_desktop_config.json`\n\nand restart the app:\n\n```\n{\n  \"mcpServers\": {\n    \"pyscrappy\": {\n      \"command\": \"pyscrappy-mcp\"\n    }\n  }\n}\n```\n\nTip:Claude Desktop does not inherit your shell`PATH`\n\n. If`pyscrappy-mcp`\n\nis not found, use the absolute path to the command (e.g. the one printed by`which pyscrappy-mcp`\n\n).\n\nThe server exposes **20+ tools**. The most common ones are ** scrape_url** (any\nURL → text, links, images, tables, metadata),\n\n**,**\n\n`scrape_wikipedia`\n\n**,**\n\n`scrape_stock`\n\n**, and**\n\n`scrape_news`\n\n**— plus many more covering image/YouTube/LinkedIn/Hacker News/book search, weather, crypto, currency, dictionary, Amazon/Newegg/IKEA/SoundCloud, IMDB, and Zomato/Uber Eats.**\n\n`search_github`\n\nTo see the full, live list, ask the agent to call the ** list_available_scrapers**\ntool, or from a shell:\n\n``` python\npython -c \"from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))\"\n```\n\nThe `lookup_movie`\n\ntool needs a free [OMDb](https://www.omdbapi.com/apikey.aspx) API\nkey. Pass it to the server through your MCP client config, e.g. for Claude Desktop:\n\n```\n{\n  \"mcpServers\": {\n    \"pyscrappy\": {\n      \"command\": \"pyscrappy-mcp\",\n      \"env\": { \"OMDB_API_KEY\": \"your-key\" }\n    }\n  }\n}\n```\n\nOnce registered, just ask the agent naturally, e.g. *\"use pyscrappy to get the\nlatest headlines from bbc.co.uk and the AAPL stock quote.\"*\n\nPyScrappy ships **24 built-in scrapers**, and every one that works without a\nproxy is also exposed as an [MCP tool](#mcp-server-use-pyscrappy-from-an-ai-agent).\n\nA few of them:\n\n— scrape any URL with auto-extraction (text, links, images, tables, metadata)`GenericScraper`\n\n**Data / research**—,`WikipediaScraper`\n\n(Yahoo Finance),`StockScraper`\n\n(RSS/Atom),`NewsScraper`\n\n,`GitHubScraper`\n\n, plus weather, crypto, currency, dictionary, image, LinkedIn-jobs, and book search`HackerNewsScraper`\n\n**E-commerce**—,`AmazonScraper`\n\n`NeweggScraper`\n\n,`IKEAScraper`\n\n**Social / media / food**—, SoundCloud, Zomato, Uber Eats (Instagram / Twitter / Spotify also ship, but are blocked and need a proxy)`YouTubeScraper`\n\n…and many more. To see the full, live list:\n\n``` python\npython -c \"from pyscrappy import list_scrapers; print(', '.join(sorted(list_scrapers())))\"\n```\n\n** IMDBScraper** (\n\n`lookup_movie`\n\n) is the one exception that needs a key — a free\n[OMDb](https://www.omdbapi.com/apikey.aspx)\n\n`OMDB_API_KEY`\n\n(see the\n[MCP config](#available-tools)above for how to pass it).\n\nPyScrappy is extensible: you can add your own scrapers, and third parties can\nship them as standalone `pyscrappy-<name>`\n\npackages. A registered scraper works\neverywhere a built-in does, including the MCP server and the `pyscrappy chat`\n\nagent, with no change to PyScrappy core.\n\n**In your own code** — register with the decorator:\n\n``` python\nfrom pyscrappy import BaseScraper, register_scraper, get_scraper\nfrom pyscrappy.core.models import ScrapeResult, ScrapeMetadata\n\n@register_scraper(\"reddit\")\nclass RedditScraper(BaseScraper):\n    def scrape(self, subreddit: str, **kwargs) -> ScrapeResult:\n        data = self.fetch_and_parse(f\"https://old.reddit.com/r/{subreddit}/.json\")\n        # ... build a list of dicts ...\n        return ScrapeResult(data=[...], metadata=ScrapeMetadata(scraper=\"reddit\"))\n\nget_scraper(\"reddit\")().scrape(subreddit=\"python\")\n```\n\n**As a distributable package** — advertise an entry point in your\n`pyproject.toml`\n\n, and PyScrappy discovers it once your package is installed:\n\n```\n[project.entry-points.\"pyscrappy.scrapers\"]\nreddit = \"pyscrappy_reddit:RedditScraper\"\n```\n\nAfter `pip install pyscrappy-reddit`\n\n, the scraper shows up in\n`list_scrapers()`\n\n, and an AI agent can call it via the `scrape_with`\n\nMCP tool —\nno core change required.\n\n**First-class MCP tools (optional).** Add an `mcp_tools`\n\nmapping and your scraper\nbecomes a dedicated, typed MCP tool instead of only being reachable through the\ngeneric `scrape_with`\n\n— its schema is derived from the method signature, so\nagents get proper named arguments:\n\n```\n@register_scraper(\"reddit\")\nclass RedditScraper(BaseScraper):\n    mcp_tools = {\"search_reddit\": \"scrape\"}   # tool name -> method\n\n    def scrape(self, subreddit: str, sort: str = \"hot\") -> ScrapeResult:\n        ...\n```\n\nSee the [plugin template](/mldsveda/PyScrappy/blob/main/plugin-template) for a complete, copyable starting\npoint, and the [plugin guide](https://pyscrappy.vercel.app/docs/plugins/) for\nthe full walkthrough.\n\n``` python\nfrom pyscrappy import scrape\n\nresult = scrape(\"https://en.wikipedia.org/wiki/Web_scraping\")\n\nprint(result.to_markdown())   # feed straight to an LLM\n# ...or result.to_json() / result.to_dataframe()\n```\n\nPrefer raw fields? Every result is a `ScrapeResult`\n\nwith `.data`\n\n(a list of\ndicts):\n\n```\nprint(result.data[0][\"metadata\"][\"title\"])\nprint(result.data[0][\"text\"][\"word_count\"])\npython\nfrom pyscrappy import GenericScraper\n\nwith GenericScraper() as gs:\n    result = gs.scrape(\n        url=\"https://news.ycombinator.com\",\n        selectors={\"title\": \".titleline a\", \"score\": \".score\"},\n    )\n    for item in result.data:\n        print(item[\"title\"], item.get(\"score\", \"\"))\n```\n\nWhen you want to traverse markup directly (Scrapy/BeautifulSoup-style) rather than\nget back structured dicts, use `Selector`\n\n:\n\n``` python\nfrom pyscrappy import Selector\n\npage = Selector(html)                             # or navigate any HTML string\npage.css(\".title::text\").getall()                 # CSS with ::text / ::attr(name)\npage.xpath(\"//a/@href\").getall()                   # XPath (elements, text(), @attr)\npage.find_all(\"h2\", class_=\"title\")                # BeautifulSoup-style search\npage.find_by_text(\"Add to cart\", tag=\"button\")     # search by text content\n\nfirst = page.css(\".product\")[0]\nfirst.css(\".price::text\").get()                    # chainable\nfirst.find_similar()                               # sibling elements shaped like this one\n```\n\n`css()`\n\n/ `xpath()`\n\nreturn a `SelectorList`\n\nwith `.get()`\n\n/ `.getall()`\n\n/ `.text()`\n\n.\n`find_similar()`\n\nlocates elements with the same tag and overlapping classes, handy\nfor pulling every card/row once you've found one.\n\nA hard-coded CSS selector silently breaks the day a site changes its markup. Adaptive selectors survive that: save a fingerprint of the element the first time, and if the selector later matches nothing, relocate it by structural and textual similarity instead of returning empty.\n\n``` python\nfrom pyscrappy import Selector\n\n# First run: match normally and remember this element under an id.\npage = Selector(html_v1, url=\"https://shop.example.com\")\nprice = page.css(\".price\", auto_save=True, adaptive_id=\"price\").get()\n\n# Later, after a redesign renamed \".price\" — heal instead of breaking:\npage = Selector(html_v2, url=\"https://shop.example.com\")\nresult = page.css(\".price\", adaptive=True, adaptive_id=\"price\")\nprint(result.get(), \"→ confidence:\", result.adaptive_confidence)\n```\n\nHow the relocation decides — and where it's stronger than a naive similarity match:\n\n**Weighted signals, not a flat average.** A stable`id`\n\n/`data-*`\n\nhook counts far more than a sibling-tag list, so weak signals can't outvote strong ones.**Anchor-relative.** It remembers the nearest stable ancestor (an id'd /`data-*`\n\ncontainer) and depth, so it survives layout reshuffles that move absolute positions.**Volatility-aware text.** Prices, dates, and counts are down-weighted, so healing stays reliable on exactly the fields that change most between scrapes.**Confidence-scored.**`SelectorList.adaptive_confidence`\n\n(0-100) tells you how sure the relocation was;`threshold=`\n\nsets the minimum to accept.\n\nFingerprints persist in a small JSON store (`~/.pyscrappy/adaptive.json`\n\nby\ndefault, or `$PYSCRAPPY_HOME`\n\n), namespaced by site so the same `adaptive_id`\n\non\ntwo sites never collides. Adaptive is entirely opt-in: without `adaptive=True`\n\n, a\nbroken selector still just returns empty, exactly as before.\n\nEvery built-in scraper follows the same pattern — instantiate, `scrape(...)`\n\n,\nread `result.data`\n\n(or `.to_dataframe()`\n\n/ `.to_markdown()`\n\n):\n\n``` python\nfrom pyscrappy import WikipediaScraper\n\nwith WikipediaScraper() as ws:\n    result = ws.scrape(query=\"Python (programming language)\", mode=\"summary\")\n    print(result.data[0][\"text\"])\n```\n\nEach scraper has its own arguments (Wikipedia, stocks, IMDB, news, YouTube,\nAmazon/Newegg/IKEA, Uber Eats, and more — see the [full list](#built-in-scrapers)).\nFor per-scraper arguments and examples, see the\n[documentation](https://pyscrappy.vercel.app/docs/scrapers/).\n\nScrape a URL straight to a file without writing any code — the output format is inferred from the file extension:\n\n```\npyscrappy extract https://example.com out.md      # clean Markdown\npyscrappy extract https://example.com out.json    # structured JSON\npyscrappy extract https://example.com out.txt     # extracted page text\npyscrappy extract https://example.com out.html    # raw fetched HTML\n\n# Narrow to elements matching a CSS selector, or render JS first:\npyscrappy extract https://example.com items.txt --css-selector \".product\"\npyscrappy extract https://example.com page.md --render-js\npython\nfrom pyscrappy import ScraperConfig, GenericScraper\n\nconfig = ScraperConfig(\n    timeout=20.0,            # request timeout in seconds\n    max_retries=3,           # retry failed requests\n    rate_limit=2.0,          # seconds between requests per domain\n    proxy=\"http://...\",      # proxy URL, or a list to rotate through\n    scraper_api=None,        # route via a scraping-API service (see below)\n    headless=True,           # browser runs headless\n    render_js=\"auto\",        # auto-detect if JS rendering is needed\n    cache_ttl=0,             # response cache TTL in seconds (0 = disabled)\n    impersonate=None,        # e.g. \"chrome\" to spoof a browser's TLS fingerprint (see below)\n)\n\nwith GenericScraper(config) as gs:\n    result = gs.scrape(url=\"https://example.com\")\n```\n\nSome sites (e.g. eBay, Instagram, Twitter/X, Spotify) block direct automated requests. PyScrappy supports two ways to get through them.\n\n**A proxy** (or a rotating list) — applies to both the HTTP and browser backends:\n\n``` python\nfrom pyscrappy import ScraperConfig, AmazonScraper\n\n# Single proxy\nconfig = ScraperConfig(proxy=\"http://user:pass@host:port\")\n\n# Rotating list (one picked per request)\nconfig = ScraperConfig(proxy=[\"http://p1:8080\", \"http://p2:8080\"])\n```\n\n**A scraping-API service** (ScraperAPI, ScrapeOps, ScrapingBee) — routes requests\nthrough the service, which handles proxies and anti-bot challenges for you:\n\n```\nconfig = ScraperConfig(scraper_api={\n    \"provider\": \"scraperapi\",   # or \"scrapeops\", \"scrapingbee\"\n    \"api_key\": \"YOUR_KEY\",\n    \"render_js\": True,           # optional\n})\n\n# Now any scraper works through the service, unchanged:\nwith AmazonScraper(config) as scraper:\n    result = scraper.scrape(query=\"laptop\")\n```\n\nThis is the reliable way to use the scrapers marked \"needs proxy\" above.\n\n**TLS-fingerprint impersonation** — many anti-bot systems block a plain HTTP\nclient by its TLS/JA3 fingerprint before serving any content. Set `impersonate`\n\nto mimic a real browser's fingerprint and get past that class of block without a\nheadless browser:\n\n``` python\nfrom pyscrappy import ScraperConfig, GenericScraper\n\n# needs the optional extra:  pip install 'pyscrappy[stealth]'\nconfig = ScraperConfig(impersonate=\"chrome\")   # or \"chrome124\", \"safari\", \"firefox\"\n\nwith GenericScraper(config) as gs:\n    result = gs.scrape(\"https://example.com\")\n```\n\nImpersonation currently applies to the **synchronous** path only; setting it on\nan async client raises a clear error. All the usual retry, rate-limiting,\ncaching, and robots handling still apply.\n\nScraping is I/O-bound, so running several scrapes at once parallelizes the\nnetwork waits. `scrape_many`\n\nruns one scraper over many inputs; `scrape_all`\n\nruns a mix of scrapers together. Both preserve input order.\n\n``` python\nfrom pyscrappy import scrape_many, scrape_all, AmazonScraper, WikipediaScraper, NewsScraper\n\n# One scraper, many queries, concurrently:\nresults = scrape_many(AmazonScraper, [{\"query\": \"laptop\"}, {\"query\": \"phone\"}])\n\n# Different scrapers at once:\nresults = scrape_all([\n    lambda: WikipediaScraper().scrape(query=\"Python\"),\n    lambda: NewsScraper().scrape(feed_url=\"https://rss.nytimes.com/services/xml/rss/nyt/World.xml\"),\n])\n```\n\nSet `cache_ttl`\n\nto a positive number of seconds to cache successful GET\nresponses. Repeated requests for the same URL (and query params) within the TTL\nare served from cache, skipping both the network and the rate limiter. Caching\nis **disabled by default** (`cache_ttl=0`\n\n).\n\n``` python\nfrom pyscrappy import WikipediaScraper\nfrom pyscrappy import ScraperConfig\n\nconfig = ScraperConfig(cache_ttl=300)   # cache for 5 minutes\n\nwith WikipediaScraper(config) as ws:\n    ws.scrape(query=\"Python\")   # fetched over the network\n    ws.scrape(query=\"Python\")   # served from cache\n```\n\nThe cache is in memory and shared across scraper instances in the same process\n(so it also speeds up repeated calls through the MCP server), and is cleared\nwhen the process exits. Call `HttpClient.clear_cache()`\n\nto empty it manually.\n\n**Required:** `httpx`\n\n, `beautifulsoup4`\n\n, `lxml`\n\n**Optional:** `playwright`\n\n(JS rendering), `pandas`\n\n(DataFrames), `fastmcp`\n\n(MCP server, Python 3.10+)\n\nAll contributions welcome. See [Issues](https://github.com/mldsveda/PyScrappy/issues).\n\n**This package is for educational and research purposes.**", "url": "https://wpnews.pro/news/show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server", "canonical_source": "https://github.com/mldsveda/PyScrappy", "published_at": "2026-08-16 07:46:40+00:00", "updated_at": "2026-08-16 08:10:57.413044+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "artificial-intelligence"], "entities": ["PyScrappy", "MCP", "Claude", "Cursor", "Ollama", "Playwright", "ScraperAPI", "ScrapeOps"], "alternates": {"html": "https://wpnews.pro/news/show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server", "markdown": "https://wpnews.pro/news/show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server.md", "text": "https://wpnews.pro/news/show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server.txt", "jsonld": "https://wpnews.pro/news/show-hn-pyscrappy-self-healing-web-scraping-selectors-plus-an-mcp-server.jsonld"}}