Turn Any Docs Site Into Clean Markdown for RAG (Python, No Dependencies)

If you're building a RAG chatbot or giving an AI agent "the docs", the first problem is boring: web pages are 80% navigation, footers, cookie banners, and script tags. Feed that raw HTML to an embedding model and your chunks fill up with "Skip to main content" and menu links.

What you want is the main content as Markdown: headings kept (they make great chunk boundaries), code blocks fenced, lists intact, and everything else gone.

Standard library only. It keeps <main> (or role="main"), drops nav/ header/ footer/ aside/ script and ARIA navigation landmarks, and converts headings, paragraphs, lists, inline code, and <pre> blocks:

"""Turn a docs page into clean Markdown for RAG, with the standard library only.
Usage: python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md"""
import re
import sys
import urllib.request
from html.parser import HTMLParser

SKIP = {"script", "style", "nav", "header", "footer", "aside", "form", "noscript", "svg"}
HEADINGS = {"h1": "# ", "h2": "## ", "h3": "### ", "h4": "#### "}

class ToMarkdown(HTMLParser):
    def __init__(self):
        super().__init__()
        self.out, self.skip, self.pre = [], 0, False

    def handle_starttag(self, tag, attrs):
        role = dict(attrs).get("role", "")
        if tag in SKIP or role in ("navigation", "banner", "contentinfo"):
            self.skip += 1
        elif self.skip:
            return
        elif tag in HEADINGS:
            self.out.append("\n\n" + HEADINGS[tag])
        elif tag in ("p", "div", "section"):
            self.out.append("\n\n")
        elif tag == "li":
            self.out.append("\n- ")
        elif tag == "pre":
            self.pre = True
            self.out.append("\n\n```

\n")
        elif tag == "code" and not self.pre:
            self.out.append("`")

    def handle_endtag(self, tag):
        if tag in SKIP:
            self.skip = max(0, self.skip - 1)
        elif self.skip:
            return
        elif tag == "pre":
            self.pre = False
            self.out.append("\n```\n")
        elif tag == "code" and not self.pre:
            self.out.append("`")

    def handle_data(self, data):
        if not self.skip:
            self.out.append(data if self.pre else re.sub(r"\s+", " ", data))

url = sys.argv[1]
req = urllib.request.Request(url, headers={"User-Agent": "docs-to-markdown"})
html = urllib.request.urlopen(req, timeout=30).read().decode("utf-8", "replace")
main = re.search(r"<main\b.*?</main>|<div[^>]+role=\"main\".*", html, re.S | re.I)  # prefer the main content
parser = ToMarkdown()
parser.feed(main.group(0) if main else html)
md = re.sub(r"\n{3,}", "\n\n", "".join(parser.out).replace("¶", "")).strip()
print(md)
print(f"\n<!-- {len(md.split())} words from {url} -->", file=sys.stderr)
``{% endraw %}{% raw %}`

Real run (October 2026):

``` shell
$ python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md
<!-- 3810 words from https://docs.python.org/3/library/json.html -->
``{% endraw %}{% raw %}`

The output starts like this, with no menus and no breadcrumbs:

``` python

Source code: Lib/json/__init__.py

JSON (JavaScript Object Notation), specified by RFC 7159 ...
``{% endraw %}{% raw %}`

## Why Markdown and not plain text?

- **Headings become chunk boundaries.** Split on `##` and each chunk is about one topic, which retrieves far better than fixed 500-token windows.
- **Code stays fenced.** Your model can tell code from prose.
- **It's compact.** Markdown is much smaller than the HTML it came from, so you spend fewer tokens per page.

## What this script doesn't do (and what bites at scale)

- **Tables and links.** It flattens them. For docs heavy on parameter tables, convert `<table>` to pipe tables and keep `[text](url)` links.
- **Crawling.** One page is easy. A whole docs site needs the sitemap, link-following that stays in scope (only `/docs/v2/`, not the blog), and deduplication.
- **robots.txt.** Be a good citizen and check it before crawling. Also, Python's `urllib.robotparser` mishandles `*` wildcards and rewrites `Disallow: /?` as `Disallow: /`, so test it against real files.
- **JavaScript-rendered docs.** Plain HTTP sees an empty shell. Those need a headless browser.

## The no-code version

I packaged a fuller version (tables, links, nested lists, sitemap plus in-scope crawling, robots.txt, about 2 seconds for 8 pages) as [Website to Markdown on Apify](https://apify.com/kramer_labs/website-to-markdown). Give it a start URL like `https://docs.python.org/3/library/` and get one Markdown record per page, ready for LangChain, LlamaIndex, or your vector store. It costs $2 per 1,000 pages, and failed pages and pages robots.txt disallows are free.

Either way, the script above is yours to copy.

*More free scripts and tools from [Kramer Labs](https://kramer-labs.netlify.app/).*