{"slug": "turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies", "title": "Turn Any Docs Site Into Clean Markdown for RAG (Python, No Dependencies)", "summary": "A developer published a dependency-free Python script that converts documentation pages into clean Markdown for retrieval-augmented generation pipelines, using only the standard library's HTMLParser and urllib. The tool keeps the main content region, strips navigation, headers, footers, scripts and ARIA landmarks, and preserves headings, lists, inline code and fenced code blocks so headings can serve as chunk boundaries. In a test run against Python's json module documentation, it produced roughly 3,810 words of Markdown with no menus or breadcrumbs.", "body_md": "If you're building a RAG chatbot or giving an AI agent \"the docs\", the first problem is boring: web pages are 80% navigation, footers, cookie banners, and script tags. Feed that raw HTML to an embedding model and your chunks fill up with \"Skip to main content\" and menu links.\n\nWhat you want is the **main content as Markdown**: headings kept (they make great chunk boundaries), code blocks fenced, lists intact, and everything else gone.\n\nStandard library only. It keeps `<main>` (or `role=\"main\"`), drops `nav`/` header`/` footer`/` aside`/` script` and ARIA navigation landmarks, and converts headings, paragraphs, lists, inline code, and `<pre>` blocks:\n\n```\n\"\"\"Turn a docs page into clean Markdown for RAG, with the standard library only.\nUsage: python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md\"\"\"\nimport re\nimport sys\nimport urllib.request\nfrom html.parser import HTMLParser\n\nSKIP = {\"script\", \"style\", \"nav\", \"header\", \"footer\", \"aside\", \"form\", \"noscript\", \"svg\"}\nHEADINGS = {\"h1\": \"# \", \"h2\": \"## \", \"h3\": \"### \", \"h4\": \"#### \"}\n\nclass ToMarkdown(HTMLParser):\n    def __init__(self):\n        super().__init__()\n        self.out, self.skip, self.pre = [], 0, False\n\n    def handle_starttag(self, tag, attrs):\n        role = dict(attrs).get(\"role\", \"\")\n        if tag in SKIP or role in (\"navigation\", \"banner\", \"contentinfo\"):\n            self.skip += 1\n        elif self.skip:\n            return\n        elif tag in HEADINGS:\n            self.out.append(\"\\n\\n\" + HEADINGS[tag])\n        elif tag in (\"p\", \"div\", \"section\"):\n            self.out.append(\"\\n\\n\")\n        elif tag == \"li\":\n            self.out.append(\"\\n- \")\n        elif tag == \"pre\":\n            self.pre = True\n            self.out.append(\"\\n\\n```\n\n\\n\")\n        elif tag == \"code\" and not self.pre:\n            self.out.append(\"`\")\n\n    def handle_endtag(self, tag):\n        if tag in SKIP:\n            self.skip = max(0, self.skip - 1)\n        elif self.skip:\n            return\n        elif tag == \"pre\":\n            self.pre = False\n            self.out.append(\"\\n```\\n\")\n        elif tag == \"code\" and not self.pre:\n            self.out.append(\"`\")\n\n    def handle_data(self, data):\n        if not self.skip:\n            self.out.append(data if self.pre else re.sub(r\"\\s+\", \" \", data))\n\nurl = sys.argv[1]\nreq = urllib.request.Request(url, headers={\"User-Agent\": \"docs-to-markdown\"})\nhtml = urllib.request.urlopen(req, timeout=30).read().decode(\"utf-8\", \"replace\")\nmain = re.search(r\"<main\\b.*?</main>|<div[^>]+role=\\\"main\\\".*\", html, re.S | re.I)  # prefer the main content\nparser = ToMarkdown()\nparser.feed(main.group(0) if main else html)\nmd = re.sub(r\"\\n{3,}\", \"\\n\\n\", \"\".join(parser.out).replace(\"¶\", \"\")).strip()\nprint(md)\nprint(f\"\\n<!-- {len(md.split())} words from {url} -->\", file=sys.stderr)\n``{% endraw %}{% raw %}`\n\nReal run (October 2026):\n\n``` shell\n$ python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md\n<!-- 3810 words from https://docs.python.org/3/library/json.html -->\n``{% endraw %}{% raw %}`\n\nThe output starts like this, with no menus and no breadcrumbs:\n\n``` python\n# `json` — JSON encoder and decoder\n\nSource code: Lib/json/__init__.py\n\nJSON (JavaScript Object Notation), specified by RFC 7159 ...\n``{% endraw %}{% raw %}`\n\n## Why Markdown and not plain text?\n\n- **Headings become chunk boundaries.** Split on `##` and each chunk is about one topic, which retrieves far better than fixed 500-token windows.\n- **Code stays fenced.** Your model can tell code from prose.\n- **It's compact.** Markdown is much smaller than the HTML it came from, so you spend fewer tokens per page.\n\n## What this script doesn't do (and what bites at scale)\n\n- **Tables and links.** It flattens them. For docs heavy on parameter tables, convert `<table>` to pipe tables and keep `[text](url)` links.\n- **Crawling.** One page is easy. A whole docs site needs the sitemap, link-following that stays in scope (only `/docs/v2/`, not the blog), and deduplication.\n- **robots.txt.** Be a good citizen and check it before crawling. Also, Python's `urllib.robotparser` mishandles `*` wildcards and rewrites `Disallow: /?` as `Disallow: /`, so test it against real files.\n- **JavaScript-rendered docs.** Plain HTTP sees an empty shell. Those need a headless browser.\n\n## The no-code version\n\nI packaged a fuller version (tables, links, nested lists, sitemap plus in-scope crawling, robots.txt, about 2 seconds for 8 pages) as [Website to Markdown on Apify](https://apify.com/kramer_labs/website-to-markdown). Give it a start URL like `https://docs.python.org/3/library/` and get one Markdown record per page, ready for LangChain, LlamaIndex, or your vector store. It costs $2 per 1,000 pages, and failed pages and pages robots.txt disallows are free.\n\nEither way, the script above is yours to copy.\n\n*More free scripts and tools from [Kramer Labs](https://kramer-labs.netlify.app/).*\n```\n\n", "url": "https://wpnews.pro/news/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies", "canonical_source": "https://dev.to/kramer_labs/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies-1ldm", "published_at": "2026-10-31 17:17:17+00:00", "updated_at": "2026-10-31 17:22:22.588588+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence", "ai-agents"], "entities": ["Python", "HTMLParser", "urllib", "RFC 7159"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies", "markdown": "https://wpnews.pro/news/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies.md", "text": "https://wpnews.pro/news/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies.txt", "jsonld": "https://wpnews.pro/news/turn-any-docs-site-into-clean-markdown-for-rag-python-no-dependencies.jsonld"}}