Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then matches "Privacy policy | Contact us | © 2026" as often as it matches the answer.
This post covers the two steps that fix most of it: converting each page to clean Markdown of its main content, and chunking that Markdown along its headings so each chunk is a coherent section with a citation.
Disclosure: I'm Aurelix, and I built the Website to Markdown Actor used for the crawl step. It's a paid tool (prices in the Cost section). The chunking code works on Markdown from any source.
/docs/...), not the whole domain. Exclude changelogs, tag pages and paginated archives, which add volume and little meaning.<article> or <main> element when present, otherwise the largest content block. Strip nav, headers, footers, sidebars, cookie banners and scripts.Page > Section > Subsection). It improves retrieval and lets the model cite its source.
The Website to Markdown Actor takes:
startUrls (required): pages to start from; links are followed within the scope.maxPages (default 50, up to 10,000): stop after this many pages are converted.scope: "path" (only URLs under the start URL's folder) or "domain" (the whole site of each start URL).includeGlobs / excludeGlobs: URL patterns, where ** matches anything and * anything except /.
Each page becomes one record:
{
"url": "https://docs.apify.com/academy",
"title": "Apify Academy | Academy | Apify Documentation",
"description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
"language": "en",
"last_modified": "2026-11-04T07:17:58Z",
"markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n...",
"headings": [
{"level": 1, "text": "Apify Academy"},
{"level": 2, "text": "Beginner courses"}
],
"links": [
"https://docs.apify.com/academy/scraping-with-ai",
"https://docs.apify.com/academy/scraping-basics-javascript"
],
"word_count": 231,
"crawl_depth": 0,
"http_status": 200,
"source_url": "https://docs.apify.com/academy",
"scraped_at": "2026-11-04T11:25:08Z"
}
Links inside the Markdown are made absolute, and code blocks keep their language tag, which helps the chunker below.
pip install apify-client
python
import json
import os
import re
from apify_client import ApifyClient
MAX_CHARS = 2000
MIN_CHARS = 200
HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
def crawl(start_url, max_pages=100):
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/website-to-markdown").call(
run_input={
"startUrls": [start_url],
"maxPages": max_pages,
"scope": "path",
"excludeGlobs": ["**/changelog/**"],
}
)
return list(client.dataset(run["defaultDatasetId"]).iterate_items())
def split_sections(markdown):
"""Yield (heading_path, text) per section; ignores '#' inside code fences."""
path, lines, in_code = [], [], False
for line in markdown.splitlines():
if FENCE.match(line):
in_code = not in_code
match = None if in_code else HEADING.match(line)
if match:
if any(l.strip() for l in lines):
yield list(path), "\n".join(lines).strip()
level = len(match.group(1))
path = path[: level - 1] + [match.group(2).strip()]
lines = [line]
else:
lines.append(line)
if any(l.strip() for l in lines):
yield list(path), "\n".join(lines).strip()
def split_blocks(text):
"""Paragraph blocks, keeping each code fence whole."""
blocks, current, in_code = [], [], False
for line in text.splitlines():
if FENCE.match(line):
in_code = not in_code
if not line.strip() and not in_code:
if current:
blocks.append("\n".join(current))
current = []
else:
current.append(line)
if current:
blocks.append("\n".join(current))
return blocks
def pack(text):
if len(text) <= MAX_CHARS:
return [text]
chunks, current = [], ""
for block in split_blocks(text):
if current and len(current) + len(block) + 2 > MAX_CHARS:
chunks.append(current)
current = block
else:
current = f"{current}\n\n{block}" if current else block
if current:
chunks.append(current)
return chunks
def chunk_page(page):
chunks = []
for path, text in split_sections(page.get("markdown") or ""):
for part in pack(text):
if chunks and len(part) < MIN_CHARS and chunks[-1]["heading_path"] == path:
chunks[-1]["text"] += "\n\n" + part
continue
chunks.append({"heading_path": path, "text": part})
for i, c in enumerate(chunks):
c.update({
"id": f"{page['url']}#{i}",
"url": page["url"],
"title": page.get("title"),
"language": page.get("language"),
"last_modified": page.get("last_modified"),
"embed_text": " > ".join(c["heading_path"]) + "\n\n" + c["text"],
})
return chunks
def main():
pages = crawl("https://docs.apify.com/academy", max_pages=100)
total = 0
with open("chunks.jsonl", "w", encoding="utf-8") as f:
for page in pages:
for chunk in chunk_page(page):
f.write(json.dumps(chunk, ensure_ascii=False) + "\n")
total += 1
print(f"{len(pages)} pages -> {total} chunks in chunks.jsonl")
if __name__ == "__main__":
main()
timeout option to 60" means little alone. Prefixing it with Crawler configuration > Timeouts gives the embedding model the context a human gets from the page layout. Store the bare text separately for display.len() for a tokenizer count; the logic doesn't change.MAX_CHARS stays one oversized chunk, which is usually the right trade.url#index) let you re-crawl and upsert. Pair them with last_modified to re-embed only pages that changed.
From chunks.jsonl, into a vector store is one loop: embed embed_text, store text, url, title and heading_path as metadata, and return url with each answer as the citation.
https://example.com/docs/ with scope: "path" crawls only excludeGlobs: ["**/v1/**"], or the same answer appears three times with three different version numbers.excludeGlobs: ["**?page=*"] skips archive pages that are lists of links.word_count`` www. and the bare domain count as one site; other subdomains aren't followed.
Compared with the leading alternatives on the Apify Store (the full table is on each Store page):
At the time of writing, the README lists $1.50 per 1,000 pages on Apify's free plan and $1.05 per 1,000 on paid plans (Bronze, Silver, Gold), plus $0.001 per run. Skipped pages (robots.txt, errors, non-HTML, duplicate URLs or identical Markdown, JavaScript-only pages, pages with no main content) aren't charged.
The 100-page example costs at most $0.15. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 pages.