Turn any website into clean Markdown for your RAG pipeline (no boilerplate, with token counts)

A developer published an Apify Actor called LLM Web Scraper that converts web pages into clean Markdown with metadata and approximate token counts, aimed at feeding RAG pipelines without nav bars, cookie banners and other boilerplate. The scraper supports same-domain crawling with page and depth caps, and can be called via the Apify Python client or through the Apify MCP server as a tool in Claude, ChatGPT or Cursor. The writeup includes a sample Markdown-aware chunking function that splits on H1-H3 headings and stores URL and title as chunk metadata.

If you've built a RAG pipeline, you know the boring part isn't the embeddings. It's getting clean text out of web pages. Raw HTML is full of nav bars, cookie banners, footers and "related posts", and all of it ends up in your vector database and in your prompts. Here's the setup I use now: give a list of URLs or one docs site to a scraper that returns only the main content as Markdown , plus metadata and an approximate token count per page. Then chunk and embed. I ran it on two pages: the Wikipedia article on RAG and the Apify docs page for Actors. This is the real output Markdown trimmed : { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented generation", "title": "Retrieval-augmented generation - Wikipedia", "date": "2023-11-05", "sitename": "Wikimedia Foundation, Inc.", "wordCount": 225, "tokensApprox": 375, "method": "http", "success": true, "markdown": "---\ntitle: Retrieval-augmented generation - Wikipedia\n...\n---\n Retrieval-augmented generation RAG is a technique that ..." } A few things that matter for RAG: tokensApprox method maxCharsPerPage The scraper is an Apify Actor, so you can call it with the official client and get the dataset back in one call: python from apify client import ApifyClient client = ApifyClient "YOUR APIFY TOKEN" run = client.actor "get anything/llm-web-scraper" .call run input={ "startUrls": {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented generation"}, {"url": "https://docs.apify.com/platform/actors"}, , "outputFormat": "markdown", or "text" / "both" "includeLinks": False, cleaner chunks for embeddings "useBrowser": "auto", browser only when HTTP isn't enough } pages = list client.dataset run "defaultDatasetId" .iterate items for p in pages: print p "title" , p "tokensApprox" , "tokens" Markdown headings make natural chunk boundaries. A minimal splitter: python import re def chunk markdown page, max chars=2000 : parts = re.split r"\n ?= {1,3} ", page "markdown" split before H1-H3 chunks, buf = , "" for part in parts: if len buf + len part max chars and buf: chunks.append buf buf = "" buf += part + "\n" if buf: chunks.append buf return {"text": c, "url": page "url" , "title": page "title" } for c in chunks docs = c for p in pages if p "success" for c in chunk markdown p - embed text , store url + title as metadata pgvector, Qdrant, Pinecone, Weaviate... Set crawl: true and it follows same-domain links from each start URL, with caps so a run can't run away: { "startUrls": { "url": "https://docs.example.com/" } , "crawl": true, "maxPages": 200, "maxDepth": 3, "outputFormat": "markdown" } That gives you one dataset row per page, ready to chunk the same way. It's an easy way to give an assistant a product's documentation. You don't have to write code at all. Through the Apify MCP server, Claude, ChatGPT, Cursor or any MCP client can call the scraper as a tool: { "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get anything/llm-web-scraper" } } } Then just ask: "Read these three URLs and summarise what each says about pricing." Step-by-step setup: Connect any Apify scraper to Claude or ChatGPT in 2 minutes https://dev.to/get anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he . You pay per page converted see the Pricing tab on the Actor page . There's no subscription, and the Apify free plan's monthly credit covers plenty of testing. Try it: LLM Web Scraper on Apify https://apify.com/get anything/llm-web-scraper . If you need the search step too query → top results → their pages as Markdown , I wrote that up separately: the AI Web Search Scraper https://apify.com/get anything/ai-web-search-scraper . Questions or a site it doesn't handle well? Drop a comment, I read them all.