HTML to clean Markdown chunks in Python, and spotting what really changed
A developer published a Python script that converts HTML pages into clean, sectioned Markdown chunks using only the standard library, aimed at search indexes and RAG pipelines that need stable section names. The script builds a lightweight DOM with html.parser, extracts main content via //role="main"/ fallbacks, and skips noise such as scripts, nav, footers and Sphinx headerlink permalinks. A demonstration diffed the Python 3.13 and 3.14 json module docs to show which sections changed, so only affected chunks need re-embedding.
If you feed web pages to a search index or a retrieval-augmented generation RAG pipeline, you usually want three things: only the main content, as plain Markdown, split into sections that have stable names. When the page is updated, you also want to know which sections changed, so you re-embed a few chunks instead of the whole page. This post builds that with Python's standard library only. The output is CommonMark, plus GitHub Flavored Markdown pipe tables for HTML tables. The script: The real run below uses two versions of the same page: the json module documentation for Python 3.13 and for Python 3.14, fetched on 27 September 2026 UTC . docs.python.org/robots.txt disallows /dev , /release and end-of-life versions such as /3.9/ ; /3.13/ and /3.14/ are allowed. That gives a genuine before and after without waiting for a page to change. html.parser is an event parser: it reports start tags, end tags and text. Building a tree from those events takes a stack. Void elements such as