Turn Any Docs Site Into Clean Markdown for RAG (Python, No Dependencies)

A developer published a dependency-free Python script that converts documentation pages into clean Markdown for retrieval-augmented generation pipelines, using only the standard library's HTMLParser and urllib. The tool keeps the main content region, strips navigation, headers, footers, scripts and ARIA landmarks, and preserves headings, lists, inline code and fenced code blocks so headings can serve as chunk boundaries. In a test run against Python's json module documentation, it produced roughly 3,810 words of Markdown with no menus or breadcrumbs.

If you're building a RAG chatbot or giving an AI agent "the docs", the first problem is boring: web pages are 80% navigation, footers, cookie banners, and script tags. Feed that raw HTML to an embedding model and your chunks fill up with "Skip to main content" and menu links. What you want is the main content as Markdown : headings kept they make great chunk boundaries , code blocks fenced, lists intact, and everything else gone. Standard library only. It keeps <main or role="main" , drops nav / header / footer / aside / script and ARIA navigation landmarks, and converts headings, paragraphs, lists, inline code, and <pre blocks: """Turn a docs page into clean Markdown for RAG, with the standard library only. Usage: python3 docs to markdown.py https://docs.python.org/3/library/json.html json.md""" import re import sys import urllib.request from html.parser import HTMLParser SKIP = {"script", "style", "nav", "header", "footer", "aside", "form", "noscript", "svg"} HEADINGS = {"h1": " ", "h2": " ", "h3": " ", "h4": " "} class ToMarkdown HTMLParser : def init self : super . init self.out, self.skip, self.pre = , 0, False def handle starttag self, tag, attrs : role = dict attrs .get "role", "" if tag in SKIP or role in "navigation", "banner", "contentinfo" : self.skip += 1 elif self.skip: return elif tag in HEADINGS: self.out.append "\n\n" + HEADINGS tag elif tag in "p", "div", "section" : self.out.append "\n\n" elif tag == "li": self.out.append "\n- " elif tag == "pre": self.pre = True self.out.append "\n\n \n" elif tag == "code" and not self.pre: self.out.append " " def handle endtag self, tag : if tag in SKIP: self.skip = max 0, self.skip - 1 elif self.skip: return elif tag == "pre": self.pre = False self.out.append "\n \n" elif tag == "code" and not self.pre: self.out.append " " def handle data self, data : if not self.skip: self.out.append data if self.pre else re.sub r"\s+", " ", data url = sys.argv 1 req = urllib.request.Request url, headers={"User-Agent": "docs-to-markdown"} html = urllib.request.urlopen req, timeout=30 .read .decode "utf-8", "replace" main = re.search r"<main\b. ?</main |<div ^ +role=\"main\". ", html, re.S | re.I prefer the main content parser = ToMarkdown parser.feed main.group 0 if main else html md = re.sub r"\n{3,}", "\n\n", "".join parser.out .replace "¶", "" .strip print md print f"\n< -- {len md.split } words from {url} -- ", file=sys.stderr {% endraw %}{% raw %} Real run October 2026 : shell $ python3 docs to markdown.py https://docs.python.org/3/library/json.html json.md < -- 3810 words from https://docs.python.org/3/library/json.html -- {% endraw %}{% raw %} The output starts like this, with no menus and no breadcrumbs: python json — JSON encoder and decoder Source code: Lib/json/ init .py JSON JavaScript Object Notation , specified by RFC 7159 ... {% endraw %}{% raw %} Why Markdown and not plain text? - Headings become chunk boundaries. Split on and each chunk is about one topic, which retrieves far better than fixed 500-token windows. - Code stays fenced. Your model can tell code from prose. - It's compact. Markdown is much smaller than the HTML it came from, so you spend fewer tokens per page. What this script doesn't do and what bites at scale - Tables and links. It flattens them. For docs heavy on parameter tables, convert <table to pipe tables and keep text url links. - Crawling. One page is easy. A whole docs site needs the sitemap, link-following that stays in scope only /docs/v2/ , not the blog , and deduplication. - robots.txt. Be a good citizen and check it before crawling. Also, Python's urllib.robotparser mishandles wildcards and rewrites Disallow: /? as Disallow: / , so test it against real files. - JavaScript-rendered docs. Plain HTTP sees an empty shell. Those need a headless browser. The no-code version I packaged a fuller version tables, links, nested lists, sitemap plus in-scope crawling, robots.txt, about 2 seconds for 8 pages as Website to Markdown on Apify https://apify.com/kramer labs/website-to-markdown . Give it a start URL like https://docs.python.org/3/library/ and get one Markdown record per page, ready for LangChain, LlamaIndex, or your vector store. It costs $2 per 1,000 pages, and failed pages and pages robots.txt disallows are free. Either way, the script above is yours to copy. More free scripts and tools from Kramer Labs https://kramer-labs.netlify.app/ .