Turn any website into clean Markdown for your RAG pipeline (no boilerplate, with token counts)

If you've built a RAG pipeline, you know the boring part isn't the embeddings. It's getting clean text out of web pages. Raw HTML is full of nav bars, cookie banners, footers and "related posts", and all of it ends up in your vector database and in your prompts.

Here's the setup I use now: give a list of URLs (or one docs site) to a scraper that returns only the main content as Markdown, plus metadata and an approximate token count per page. Then chunk and embed.

I ran it on two pages: the Wikipedia article on RAG and the Apify docs page for Actors. This is the real output (Markdown trimmed):

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "date": "2023-11-05",
  "sitename": "Wikimedia Foundation, Inc.",
  "wordCount": 225,
  "tokensApprox": 375,
  "method": "http",
  "success": true,
  "markdown": "---\ntitle: Retrieval-augmented generation - Wikipedia\n...\n---\n**Retrieval-augmented generation** (**RAG**) is a technique that ..."
}

A few things that matter for RAG:

tokensApprox`` method``maxCharsPerPage The scraper is an Apify Actor, so you can call it with the official client and get the dataset back in one call:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("get_anything/llm-web-scraper").call(run_input={
    "startUrls": [
        {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"},
        {"url": "https://docs.apify.com/platform/actors"},
    ],
    "outputFormat": "markdown",   # or "text" / "both"
    "includeLinks": False,        # cleaner chunks for embeddings
    "useBrowser": "auto",         # browser only when HTTP isn't enough
})

pages = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for p in pages:
    print(p["title"], p["tokensApprox"], "tokens")

Markdown headings make natural chunk boundaries. A minimal splitter:

import re

def chunk_markdown(page, max_chars=2000):
    parts = re.split(r"\n(?=#{1,3} )", page["markdown"])   # split before H1-H3
    chunks, buf = [], ""
    for part in parts:
        if len(buf) + len(part) > max_chars and buf:
            chunks.append(buf)
            buf = ""
        buf += part + "\n"
    if buf:
        chunks.append(buf)
    return [{"text": c, "url": page["url"], "title": page["title"]} for c in chunks]

docs = [c for p in pages if p["success"] for c in chunk_markdown(p)]

Set crawl: true and it follows same-domain links from each start URL, with caps so a run can't run away:

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "crawl": true,
  "maxPages": 200,
  "maxDepth": 3,
  "outputFormat": "markdown"
}

That gives you one dataset row per page, ready to chunk the same way. It's an easy way to give an assistant a product's documentation.

You don't have to write code at all. Through the Apify MCP server, Claude, ChatGPT, Cursor or any MCP client can call the scraper as a tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get_anything/llm-web-scraper" } } }

Then just ask: "Read these three URLs and summarise what each says about pricing." Step-by-step setup: Connect any Apify scraper to Claude or ChatGPT in 2 minutes.

You pay per page converted (see the Pricing tab on the Actor page). There's no subscription, and the Apify free plan's monthly credit covers plenty of testing.

Try it: LLM Web Scraper on Apify. If you need the search step too (query → top results → their pages as Markdown), I wrote that up separately: the AI Web Search Scraper.

Questions or a site it doesn't handle well? Drop a comment, I read them all.