{"slug": "turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with", "title": "Turn any website into clean Markdown for your RAG pipeline (no boilerplate, with token counts)", "summary": "A developer published an Apify Actor called LLM Web Scraper that converts web pages into clean Markdown with metadata and approximate token counts, aimed at feeding RAG pipelines without nav bars, cookie banners and other boilerplate. The scraper supports same-domain crawling with page and depth caps, and can be called via the Apify Python client or through the Apify MCP server as a tool in Claude, ChatGPT or Cursor. The writeup includes a sample Markdown-aware chunking function that splits on H1-H3 headings and stores URL and title as chunk metadata.", "body_md": "If you've built a RAG pipeline, you know the boring part isn't the embeddings. It's getting **clean text** out of web pages. Raw HTML is full of nav bars, cookie banners, footers and \"related posts\", and all of it ends up in your vector database and in your prompts.\n\nHere's the setup I use now: give a list of URLs (or one docs site) to a scraper that returns **only the main content as Markdown**, plus metadata and an approximate token count per page. Then chunk and embed.\n\nI ran it on two pages: the Wikipedia article on RAG and the Apify docs page for Actors. This is the real output (Markdown trimmed):\n\n```\n{\n  \"url\": \"https://en.wikipedia.org/wiki/Retrieval-augmented_generation\",\n  \"title\": \"Retrieval-augmented generation - Wikipedia\",\n  \"date\": \"2023-11-05\",\n  \"sitename\": \"Wikimedia Foundation, Inc.\",\n  \"wordCount\": 225,\n  \"tokensApprox\": 375,\n  \"method\": \"http\",\n  \"success\": true,\n  \"markdown\": \"---\\ntitle: Retrieval-augmented generation - Wikipedia\\n...\\n---\\n**Retrieval-augmented generation** (**RAG**) is a technique that ...\"\n}\n```\n\nA few things that matter for RAG:\n\n`tokensApprox`` method``maxCharsPerPage`\nThe scraper is an Apify Actor, so you can call it with the official client and get the dataset back in one call:\n\n``` python\nfrom apify_client import ApifyClient\n\nclient = ApifyClient(\"YOUR_APIFY_TOKEN\")\n\nrun = client.actor(\"get_anything/llm-web-scraper\").call(run_input={\n    \"startUrls\": [\n        {\"url\": \"https://en.wikipedia.org/wiki/Retrieval-augmented_generation\"},\n        {\"url\": \"https://docs.apify.com/platform/actors\"},\n    ],\n    \"outputFormat\": \"markdown\",   # or \"text\" / \"both\"\n    \"includeLinks\": False,        # cleaner chunks for embeddings\n    \"useBrowser\": \"auto\",         # browser only when HTTP isn't enough\n})\n\npages = list(client.dataset(run[\"defaultDatasetId\"]).iterate_items())\nfor p in pages:\n    print(p[\"title\"], p[\"tokensApprox\"], \"tokens\")\n```\n\nMarkdown headings make natural chunk boundaries. A minimal splitter:\n\n``` python\nimport re\n\ndef chunk_markdown(page, max_chars=2000):\n    parts = re.split(r\"\\n(?=#{1,3} )\", page[\"markdown\"])   # split before H1-H3\n    chunks, buf = [], \"\"\n    for part in parts:\n        if len(buf) + len(part) > max_chars and buf:\n            chunks.append(buf)\n            buf = \"\"\n        buf += part + \"\\n\"\n    if buf:\n        chunks.append(buf)\n    return [{\"text\": c, \"url\": page[\"url\"], \"title\": page[\"title\"]} for c in chunks]\n\ndocs = [c for p in pages if p[\"success\"] for c in chunk_markdown(p)]\n# -> embed `text`, store `url` + `title` as metadata (pgvector, Qdrant, Pinecone, Weaviate...)\n```\n\nSet `crawl: true` and it follows same-domain links from each start URL, with caps so a run can't run away:\n\n```\n{\n  \"startUrls\": [{ \"url\": \"https://docs.example.com/\" }],\n  \"crawl\": true,\n  \"maxPages\": 200,\n  \"maxDepth\": 3,\n  \"outputFormat\": \"markdown\"\n}\n```\n\nThat gives you one dataset row per page, ready to chunk the same way. It's an easy way to give an assistant a product's documentation.\n\nYou don't have to write code at all. Through the Apify MCP server, Claude, ChatGPT, Cursor or any MCP client can call the scraper as a tool:\n\n```\n{ \"mcpServers\": { \"apify\": { \"url\": \"https://mcp.apify.com/?tools=get_anything/llm-web-scraper\" } } }\n```\n\nThen just ask: *\"Read these three URLs and summarise what each says about pricing.\"* Step-by-step setup: [Connect any Apify scraper to Claude or ChatGPT in 2 minutes](https://dev.to/get_anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he).\n\nYou pay per page converted (see the Pricing tab on the Actor page). There's no subscription, and the Apify free plan's monthly credit covers plenty of testing.\n\n**Try it:** [LLM Web Scraper on Apify](https://apify.com/get_anything/llm-web-scraper). If you need the *search* step too (query → top results → their pages as Markdown), I wrote that up separately: the [AI Web Search Scraper](https://apify.com/get_anything/ai-web-search-scraper).\n\nQuestions or a site it doesn't handle well? Drop a comment, I read them all.", "url": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with", "canonical_source": "https://dev.to/get_anything/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with-token-counts-20h1", "published_at": "2026-10-30 06:38:15+00:00", "updated_at": "2026-10-30 06:52:50.376324+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "agent-protocols", "ai-infrastructure", "developer-tools"], "entities": ["Apify", "LLM Web Scraper", "Apify MCP server", "Claude", "ChatGPT", "Cursor", "Wikipedia"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with", "markdown": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with.md", "text": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with.txt", "jsonld": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-no-boilerplate-with.jsonld"}}