{"slug": "turn-any-website-into-clean-markdown-chunks-for-rag", "title": "Turn any website into clean Markdown chunks for RAG", "summary": "A developer writing as Aurelix published a pipeline for improving RAG retrieval quality by converting web pages to clean Markdown of their main content and then chunking that Markdown along heading boundaries. The approach strips navigation, cookie banners, footers and scripts, keeps code-fence language tags, makes links absolute, and emits each section with its heading path so chunks stay coherent and citable. The crawl step uses the paid Website to Markdown Actor on Apify, while the chunking code works on Markdown from any source.", "body_md": "Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then matches \"Privacy policy | Contact us | © 2026\" as often as it matches the answer.\n\nThis post covers the two steps that fix most of it: converting each page to clean Markdown of its main content, and chunking that Markdown along its headings so each chunk is a coherent section with a citation.\n\n**Disclosure:** I'm Aurelix, and I built the Website to Markdown Actor used for the crawl step. It's a paid tool (prices in the Cost section). The chunking code works on Markdown from any source.\n\n`/docs/...`), not the whole domain. Exclude changelogs, tag pages and paginated archives, which add volume and little meaning.`<article>` or `<main>` element when present, otherwise the largest content block. Strip nav, headers, footers, sidebars, cookie banners and scripts.`Page > Section > Subsection`). It improves retrieval and lets the model cite its source.\nThe [Website to Markdown Actor](https://apify.com/aurelix/website-to-markdown) takes:\n\n`startUrls` (required): pages to start from; links are followed within the scope.`maxPages` (default 50, up to 10,000): stop after this many pages are converted.`scope`: `\"path\"` (only URLs under the start URL's folder) or `\"domain\"` (the whole site of each start URL).`includeGlobs` / `excludeGlobs`: URL patterns, where `**` matches anything and `*` anything except `/`.\nEach page becomes one record:\n\n```\n{\n  \"url\": \"https://docs.apify.com/academy\",\n  \"title\": \"Apify Academy | Academy | Apify Documentation\",\n  \"description\": \"Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\",\n  \"language\": \"en\",\n  \"last_modified\": \"2026-11-04T07:17:58Z\",\n  \"markdown\": \"# Apify Academy\\n\\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\\n\\n## Beginner courses\\n\\n...\",\n  \"headings\": [\n    {\"level\": 1, \"text\": \"Apify Academy\"},\n    {\"level\": 2, \"text\": \"Beginner courses\"}\n  ],\n  \"links\": [\n    \"https://docs.apify.com/academy/scraping-with-ai\",\n    \"https://docs.apify.com/academy/scraping-basics-javascript\"\n  ],\n  \"word_count\": 231,\n  \"crawl_depth\": 0,\n  \"http_status\": 200,\n  \"source_url\": \"https://docs.apify.com/academy\",\n  \"scraped_at\": \"2026-11-04T11:25:08Z\"\n}\n```\n\nLinks inside the Markdown are made absolute, and code blocks keep their language tag, which helps the chunker below.\n\n```\npip install apify-client\npython\nimport json\nimport os\nimport re\n\nfrom apify_client import ApifyClient\n\nMAX_CHARS = 2000\nMIN_CHARS = 200\nHEADING = re.compile(r\"^(#{1,6})\\s+(.*)$\")\nFENCE = re.compile(r\"^\\s*(`{3,}|~{3,})\")\n\ndef crawl(start_url, max_pages=100):\n    client = ApifyClient(os.environ[\"APIFY_TOKEN\"])\n    run = client.actor(\"aurelix/website-to-markdown\").call(\n        run_input={\n            \"startUrls\": [start_url],\n            \"maxPages\": max_pages,\n            \"scope\": \"path\",\n            \"excludeGlobs\": [\"**/changelog/**\"],\n        }\n    )\n    return list(client.dataset(run[\"defaultDatasetId\"]).iterate_items())\n\ndef split_sections(markdown):\n    \"\"\"Yield (heading_path, text) per section; ignores '#' inside code fences.\"\"\"\n    path, lines, in_code = [], [], False\n    for line in markdown.splitlines():\n        if FENCE.match(line):\n            in_code = not in_code\n        match = None if in_code else HEADING.match(line)\n        if match:\n            if any(l.strip() for l in lines):\n                yield list(path), \"\\n\".join(lines).strip()\n            level = len(match.group(1))\n            path = path[: level - 1] + [match.group(2).strip()]\n            lines = [line]\n        else:\n            lines.append(line)\n    if any(l.strip() for l in lines):\n        yield list(path), \"\\n\".join(lines).strip()\n\ndef split_blocks(text):\n    \"\"\"Paragraph blocks, keeping each code fence whole.\"\"\"\n    blocks, current, in_code = [], [], False\n    for line in text.splitlines():\n        if FENCE.match(line):\n            in_code = not in_code\n        if not line.strip() and not in_code:\n            if current:\n                blocks.append(\"\\n\".join(current))\n                current = []\n        else:\n            current.append(line)\n    if current:\n        blocks.append(\"\\n\".join(current))\n    return blocks\n\ndef pack(text):\n    if len(text) <= MAX_CHARS:\n        return [text]\n    chunks, current = [], \"\"\n    for block in split_blocks(text):\n        if current and len(current) + len(block) + 2 > MAX_CHARS:\n            chunks.append(current)\n            current = block\n        else:\n            current = f\"{current}\\n\\n{block}\" if current else block\n    if current:\n        chunks.append(current)\n    return chunks\n\ndef chunk_page(page):\n    chunks = []\n    for path, text in split_sections(page.get(\"markdown\") or \"\"):\n        for part in pack(text):\n            if chunks and len(part) < MIN_CHARS and chunks[-1][\"heading_path\"] == path:\n                chunks[-1][\"text\"] += \"\\n\\n\" + part\n                continue\n            chunks.append({\"heading_path\": path, \"text\": part})\n    for i, c in enumerate(chunks):\n        c.update({\n            \"id\": f\"{page['url']}#{i}\",\n            \"url\": page[\"url\"],\n            \"title\": page.get(\"title\"),\n            \"language\": page.get(\"language\"),\n            \"last_modified\": page.get(\"last_modified\"),\n            \"embed_text\": \" > \".join(c[\"heading_path\"]) + \"\\n\\n\" + c[\"text\"],\n        })\n    return chunks\n\ndef main():\n    pages = crawl(\"https://docs.apify.com/academy\", max_pages=100)\n    total = 0\n    with open(\"chunks.jsonl\", \"w\", encoding=\"utf-8\") as f:\n        for page in pages:\n            for chunk in chunk_page(page):\n                f.write(json.dumps(chunk, ensure_ascii=False) + \"\\n\")\n                total += 1\n    print(f\"{len(pages)} pages -> {total} chunks in chunks.jsonl\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n`timeout` option to 60\" means little alone. Prefixing it with `Crawler configuration > Timeouts` gives the embedding model the context a human gets from the page layout. Store the bare `text` separately for display.`len()` for a tokenizer count; the logic doesn't change.`MAX_CHARS` stays one oversized chunk, which is usually the right trade.`url#index`) let you re-crawl and upsert. Pair them with `last_modified` to re-embed only pages that changed.\nFrom `chunks.jsonl`, loading into a vector store is one loop: embed `embed_text`, store `text`, `url`, `title` and `heading_path` as metadata, and return `url` with each answer as the citation.\n\n`https://example.com/docs/` with `scope: \"path\"` crawls only `excludeGlobs: [\"**/v1/**\"]`, or the same answer appears three times with three different version numbers.`excludeGlobs: [\"**?page=*\"]` skips archive pages that are lists of links.`word_count`` www.` and the bare domain count as one site; other subdomains aren't followed.\nCompared with the leading alternatives on the Apify Store (the full table is on each Store page):\n\nAt the time of writing, the README lists **$1.50 per 1,000 pages** on Apify's free plan and **$1.05 per 1,000** on paid plans (Bronze, Silver, Gold), plus $0.001 per run. Skipped pages (robots.txt, errors, non-HTML, duplicate URLs or identical Markdown, JavaScript-only pages, pages with no main content) aren't charged.\n\nThe 100-page example costs at most $0.15. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 pages.", "url": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-chunks-for-rag", "canonical_source": "https://dev.to/aurelix_solutions/turn-any-website-into-clean-markdown-chunks-for-rag-475h", "published_at": "2026-11-05 05:41:45+00:00", "updated_at": "2026-11-05 05:52:14.169706+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Aurelix", "Apify", "Website to Markdown Actor", "Apify Academy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-chunks-for-rag", "markdown": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-chunks-for-rag.md", "text": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-chunks-for-rag.txt", "jsonld": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-chunks-for-rag.jsonld"}}