Turn any website into clean Markdown chunks for RAG

A developer writing as Aurelix published a pipeline for improving RAG retrieval quality by converting web pages to clean Markdown of their main content and then chunking that Markdown along heading boundaries. The approach strips navigation, cookie banners, footers and scripts, keeps code-fence language tags, makes links absolute, and emits each section with its heading path so chunks stay coherent and citable. The crawl step uses the paid Website to Markdown Actor on Apify, while the chunking code works on Markdown from any source.

Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then matches "Privacy policy | Contact us | © 2026" as often as it matches the answer. This post covers the two steps that fix most of it: converting each page to clean Markdown of its main content, and chunking that Markdown along its headings so each chunk is a coherent section with a citation. Disclosure: I'm Aurelix, and I built the Website to Markdown Actor used for the crawl step. It's a paid tool prices in the Cost section . The chunking code works on Markdown from any source. /docs/... , not the whole domain. Exclude changelogs, tag pages and paginated archives, which add volume and little meaning. <article or <main element when present, otherwise the largest content block. Strip nav, headers, footers, sidebars, cookie banners and scripts. Page Section Subsection . It improves retrieval and lets the model cite its source. The Website to Markdown Actor https://apify.com/aurelix/website-to-markdown takes: startUrls required : pages to start from; links are followed within the scope. maxPages default 50, up to 10,000 : stop after this many pages are converted. scope : "path" only URLs under the start URL's folder or "domain" the whole site of each start URL . includeGlobs / excludeGlobs : URL patterns, where matches anything and anything except / . Each page becomes one record: { "url": "https://docs.apify.com/academy", "title": "Apify Academy | Academy | Apify Documentation", "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.", "language": "en", "last modified": "2026-11-04T07:17:58Z", "markdown": " Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n Beginner courses\n\n...", "headings": {"level": 1, "text": "Apify Academy"}, {"level": 2, "text": "Beginner courses"} , "links": "https://docs.apify.com/academy/scraping-with-ai", "https://docs.apify.com/academy/scraping-basics-javascript" , "word count": 231, "crawl depth": 0, "http status": 200, "source url": "https://docs.apify.com/academy", "scraped at": "2026-11-04T11:25:08Z" } Links inside the Markdown are made absolute, and code blocks keep their language tag, which helps the chunker below. pip install apify-client python import json import os import re from apify client import ApifyClient MAX CHARS = 2000 MIN CHARS = 200 HEADING = re.compile r"^ {1,6} \s+ . $" FENCE = re.compile r"^\s {3,}|~{3,} " def crawl start url, max pages=100 : client = ApifyClient os.environ "APIFY TOKEN" run = client.actor "aurelix/website-to-markdown" .call run input={ "startUrls": start url , "maxPages": max pages, "scope": "path", "excludeGlobs": " /changelog/ " , } return list client.dataset run "defaultDatasetId" .iterate items def split sections markdown : """Yield heading path, text per section; ignores ' ' inside code fences.""" path, lines, in code = , , False for line in markdown.splitlines : if FENCE.match line : in code = not in code match = None if in code else HEADING.match line if match: if any l.strip for l in lines : yield list path , "\n".join lines .strip level = len match.group 1 path = path : level - 1 + match.group 2 .strip lines = line else: lines.append line if any l.strip for l in lines : yield list path , "\n".join lines .strip def split blocks text : """Paragraph blocks, keeping each code fence whole.""" blocks, current, in code = , , False for line in text.splitlines : if FENCE.match line : in code = not in code if not line.strip and not in code: if current: blocks.append "\n".join current current = else: current.append line if current: blocks.append "\n".join current return blocks def pack text : if len text <= MAX CHARS: return text chunks, current = , "" for block in split blocks text : if current and len current + len block + 2 MAX CHARS: chunks.append current current = block else: current = f"{current}\n\n{block}" if current else block if current: chunks.append current return chunks def chunk page page : chunks = for path, text in split sections page.get "markdown" or "" : for part in pack text : if chunks and len part < MIN CHARS and chunks -1 "heading path" == path: chunks -1 "text" += "\n\n" + part continue chunks.append {"heading path": path, "text": part} for i, c in enumerate chunks : c.update { "id": f"{page 'url' } {i}", "url": page "url" , "title": page.get "title" , "language": page.get "language" , "last modified": page.get "last modified" , "embed text": " ".join c "heading path" + "\n\n" + c "text" , } return chunks def main : pages = crawl "https://docs.apify.com/academy", max pages=100 total = 0 with open "chunks.jsonl", "w", encoding="utf-8" as f: for page in pages: for chunk in chunk page page : f.write json.dumps chunk, ensure ascii=False + "\n" total += 1 print f"{len pages } pages - {total} chunks in chunks.jsonl" if name == " main ": main timeout option to 60" means little alone. Prefixing it with Crawler configuration Timeouts gives the embedding model the context a human gets from the page layout. Store the bare text separately for display. len for a tokenizer count; the logic doesn't change. MAX CHARS stays one oversized chunk, which is usually the right trade. url index let you re-crawl and upsert. Pair them with last modified to re-embed only pages that changed. From chunks.jsonl , loading into a vector store is one loop: embed embed text , store text , url , title and heading path as metadata, and return url with each answer as the citation. https://example.com/docs/ with scope: "path" crawls only excludeGlobs: " /v1/ " , or the same answer appears three times with three different version numbers. excludeGlobs: " ?page= " skips archive pages that are lists of links. word count www. and the bare domain count as one site; other subdomains aren't followed. Compared with the leading alternatives on the Apify Store the full table is on each Store page : At the time of writing, the README lists $1.50 per 1,000 pages on Apify's free plan and $1.05 per 1,000 on paid plans Bronze, Silver, Gold , plus $0.001 per run. Skipped pages robots.txt, errors, non-HTML, duplicate URLs or identical Markdown, JavaScript-only pages, pages with no main content aren't charged. The 100-page example costs at most $0.15. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 pages.