` and `
` when a `
` starts, for example. For well-formed pages, such as most documentation sites, that is fine. For messy HTML, parse with `html5lib` or `lxml` and keep the rendering part of this post.\n\nThe script looks for ``, then ``, then any element with `role=\"main\"`, then ``. Inside it, whole elements are skipped by tag (scripts, styles, navigation, footers, forms) and by class.\n\n*Excerpt of `page_to_chunks.py`, lines 20-27:*\n\n```\nUSER_AGENT = \"html-to-chunks-example/1.0 (tutorial script; one request per page)\"\nVOID = {\"area\", \"base\", \"br\", \"col\", \"embed\", \"hr\", \"img\", \"input\", \"link\", \"meta\", \"source\", \"track\", \"wbr\"}\nSKIP = {\"script\", \"style\", \"noscript\", \"template\", \"svg\", \"nav\", \"footer\", \"form\",\n \"button\", \"iframe\", \"aside\"}\nNOISE_CLASSES = {\"headerlink\"} # e.g. the pilcrow permalinks Sphinx adds after headings\nBLOCKS = {\"p\", \"div\", \"section\", \"article\", \"main\", \"blockquote\", \"pre\", \"ul\", \"ol\", \"table\",\n \"dl\", \"dt\", \"dd\", \"hr\", \"figure\", \"h1\", \"h2\", \"h3\", \"h4\", \"h5\", \"h6\"} | SKIP\n```\n\n*Excerpt of `page_to_chunks.py`, lines 64-78:*\n\n``` python\ndef main_content(root):\n for test in (lambda n: n[\"tag\"] == \"main\", lambda n: n[\"tag\"] == \"article\",\n lambda n: n[\"attrs\"].get(\"role\") == \"main\", lambda n: n[\"tag\"] == \"body\"):\n hit = find(root, test)\n if hit:\n return hit\n return root\n\ndef ignored(node):\n return node[\"tag\"] in SKIP or bool(NOISE_CLASSES & set(node[\"attrs\"].get(\"class\", \"\").split()))\n\ndef collapse(text):\n return re.sub(r\"\\s+\", \" \", text).strip()\n```\n\nThe class list is where site-specific cleanup goes. Sphinx, the documentation generator behind the Python docs, puts a `¶` permalink after every heading with the class `headerlink`. Without that one entry, every heading in the output would end in `¶` and every chunk name would carry it.\n\nInline content becomes one line: bold and italics, inline code, image alt text, and links. Links are made absolute with `urljoin`, because a relative link is useless once the text leaves the page. Links to anchors on the same page (`#...`) keep only their text.\n\n*Excerpt of `page_to_chunks.py`, lines 81-108:*\n\n``` python\ndef text_of(node):\n \"\"\"Raw text with whitespace kept (for `) become plain text. In Sphinx output they hold function signatures, and the `*` in a signature such as `json.dump(obj, fp, *, ...)` would break Markdown emphasis if it were wrapped in `**`.\n\nTables become pipe tables. Pipes inside cells are escaped, short rows are padded, and `colspan`/` rowspan` are ignored, so complex tables lose their structure. That is a known limit.\n\n*Excerpt of `page_to_chunks.py`, lines 125-137:*\n\n``` python\ndef render_table(node, base):\n rows = []\n for row in find_all(node, \"tr\"):\n cells = [collapse(inline(cell, base)).replace(\"|\", \"\\\\|\") for cell in row[\"children\"]\n if isinstance(cell, dict) and cell[\"tag\"] in (\"th\", \"td\")]\n if cells:\n rows.append(cells)\n if not rows:\n return \"\"\n width = max(len(row) for row in rows)\n rows = [row + [\"\"] * (width - len(row)) for row in rows]\n lines = [\"| \" + \" | \".join(rows[0]) + \" |\", \"|\" + \" --- |\" * width]\n return \"\\n\".join(lines + [\"| \" + \" | \".join(row) + \" |\" for row in rows[1:]])\n```\n\nEach heading up to level 3 starts a new chunk. The chunk's id is its heading path, such as `Guide > Setup`. Deeper headings stay inside their parent's chunk. When two sections share a path, the second gets `#2`.\n\nEvery chunk gets two hashes:\n\n`sha256` of the exact text, to answer \"is this byte-for-byte the same?\",`fingerprint`, the hash of the text with link targets removed and whitespace collapsed, to answer \"did the words change?\".\nThe real run shows why the second one matters.\n\n*Excerpt of `page_to_chunks.py`, lines 197-229:*\n\n``` python\ndef prose(text):\n \"\"\"Text with link targets and whitespace differences removed.\"\"\"\n return collapse(re.sub(r\"\\]\\([^)]*\\)\", \"]\", text))\n\ndef fingerprint(text):\n return hashlib.sha256(prose(text).encode(\"utf-8\")).hexdigest()[:16]\n\ndef chunk(blocks, max_level=3):\n \"\"\"Group blocks under their nearest heading (h1..h{max_level}); one chunk per section.\"\"\"\n chunks, path, body, seen = [], [], [], {}\n\n def emit():\n text = \"\\n\\n\".join(body).strip()\n body.clear()\n if text:\n name = \" > \".join(path) or \"(top)\"\n seen[name] = seen.get(name, 0) + 1\n chunk_id = name if seen[name] == 1 else f\"{name} #{seen[name]}\"\n chunks.append({\"id\": chunk_id, \"words\": len(text.split()),\n \"sha256\": hashlib.sha256(text.encode(\"utf-8\")).hexdigest()[:16],\n \"fingerprint\": fingerprint(text), \"text\": text})\n\n for block in blocks:\n heading = re.match(r\"(#{1,6}) (.*)\", block)\n if heading and len(heading.group(1)) <= max_level:\n emit()\n del path[len(heading.group(1)) - 1:]\n path.append(heading.group(2))\n body.append(block)\n emit()\n return chunks\n```\n\nSections are matched by id. A renamed heading changes the id, so it first shows up as one removed and one added section. `compare()` pairs those up when their text is at least 60% similar according to `difflib.SequenceMatcher`, and reports them as renamed. The 0.6 threshold is a judgment call; the test below also checks that a strict threshold turns pairing off.\n\n*Excerpt of `page_to_chunks.py`, lines 232-253:*\n\n``` python\ndef compare(old, new, rename_threshold=0.6):\n \"\"\"Classify chunks by id: added, removed, renamed, changed (prose), links-only, unchanged.\"\"\"\n before = {c[\"id\"]: c for c in old}\n after = {c[\"id\"]: c for c in new}\n result = {\"added\": sorted(after.keys() - before.keys()),\n \"removed\": sorted(before.keys() - after.keys()),\n \"renamed\": [], \"changed\": [], \"links-only\": [], \"unchanged\": []}\n for old_id in list(result[\"removed\"]): # a renamed heading looks like removed + added\n scored = [(difflib.SequenceMatcher(None, prose(before[old_id][\"text\"]),\n prose(after[new_id][\"text\"])).ratio(), new_id)\n for new_id in result[\"added\"]]\n if scored and max(scored)[0] >= rename_threshold:\n ratio, new_id = max(scored)\n result[\"removed\"].remove(old_id)\n result[\"added\"].remove(new_id)\n result[\"renamed\"].append(f\"{old_id} -> {new_id} (similarity {ratio:.3f})\")\n for chunk_id in [c[\"id\"] for c in new if c[\"id\"] in before]:\n a, b = before[chunk_id], after[chunk_id]\n kind = (\"unchanged\" if a[\"sha256\"] == b[\"sha256\"] else\n \"links-only\" if a[\"fingerprint\"] == b[\"fingerprint\"] else \"changed\")\n result[kind].append(chunk_id)\n return result\n```\n\nOne request per page, a descriptive `User-Agent`, `robots.txt` read first, and at least a one-second pause, or the site's `Crawl-delay` if it sets one. If `robots.txt` disallows the page, the script stops. One trap: `urllib.robotparser` ends a group at a blank line, while [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) does not, and `docs.python.org/robots.txt` has a blank line before its end-of-life rules. Parsed as is, `/3.9/` comes out allowed, so `robots_rules()` drops blank lines first. The standard parser still differs from RFC 9309 in other ways; for example, it applies the first matching rule rather than the longest one.\n\n*Excerpt of `page_to_chunks.py`, lines 256-281:*\n\n``` python\ndef robots_rules(text):\n \"\"\"Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.\"\"\"\n rules = urllib.robotparser.RobotFileParser()\n rules.parse([line for line in text.splitlines() if line.strip()])\n return rules\n\ndef fetch(url):\n parts = urllib.parse.urlsplit(url)\n try:\n text = get(f\"{parts.scheme}://{parts.netloc}/robots.txt\")\n except urllib.error.HTTPError as err:\n if err.code not in (404, 410):\n raise\n text = \"\" # no robots.txt: nothing is disallowed\n rules = robots_rules(text)\n if not rules.can_fetch(USER_AGENT, url):\n sys.exit(f\"robots.txt disallows {url}; stopping\")\n time.sleep(max(1, rules.crawl_delay(USER_AGENT) or 0))\n return get(url)\n\ndef get(url):\n request = urllib.request.Request(url, headers={\"User-Agent\": USER_AGENT})\n with urllib.request.urlopen(request, timeout=30) as response:\n return response.read().decode(response.headers.get_content_charset() or \"utf-8\", \"replace\")\n```\n\nTwelve `unittest` tests cover the parser, the renderer, the comparison and the `robots.txt` parsing. Two of them:\n\n*Excerpt of `test_page_to_chunks.py`, lines 13-18:*\n\n``` python\n def test_main_content_only_and_noise_removed(self):\n html = \"\"\"
), minus ignored elements.\"\"\"\n if isinstance(node, str):\n return node\n return \"\" if ignored(node) else \"\".join(text_of(child) for child in node[\"children\"])\n\ndef inline(node, base):\n \"\"\"Render inline content as Markdown on one line.\"\"\"\n if isinstance(node, str):\n return node\n if ignored(node):\n return \"\"\n tag, attrs = node[\"tag\"], node[\"attrs\"]\n if tag == \"br\":\n return \" \"\n if tag == \"img\":\n return attrs.get(\"alt\", \"\")\n inner = \"\".join(inline(child, base) for child in node[\"children\"])\n if tag == \"code\":\n return f\"`{collapse(text_of(node))}`\" if collapse(text_of(node)) else \"\"\n if tag in (\"strong\", \"b\", \"em\", \"i\") and collapse(inner):\n mark = \"**\" if tag in (\"strong\", \"b\") else \"*\"\n return f\"{mark}{collapse(inner)}{mark}\"\n href = attrs.get(\"href\", \"\")\n if tag == \"a\" and collapse(inner) and href and not href.startswith((\"#\", \"javascript:\")):\n return f\"[{collapse(inner)}]({urllib.parse.urljoin(base, href)})\"\n return inner\n```\n\n`render()` walks the children of a node. Text and inline elements are collected into a \"run\" until a block element appears; then the run is flushed as a paragraph. Headings, code blocks, lists, tables and definition lists each get their own branch. Code blocks keep their whitespace; everything else is collapsed.\n\n*Excerpt of `page_to_chunks.py`, lines 149-185:*\n\n``` python\ndef render(node, base, out):\n \"\"\"Append Markdown blocks (strings) for the children of `node` to `out`.\"\"\"\n run = []\n\n def flush():\n text = collapse(\"\".join(run))\n run.clear()\n if text:\n out.append(text)\n\n for child in node[\"children\"]:\n if isinstance(child, str) or child[\"tag\"] not in BLOCKS:\n run.append(inline(child, base))\n continue\n flush()\n tag = child[\"tag\"]\n if ignored(child) or tag == \"hr\":\n continue\n if re.fullmatch(r\"h[1-6]\", tag):\n title = collapse(inline(child, base))\n if title:\n out.append(\"#\" * int(tag[1]) + \" \" + title)\n elif tag == \"pre\":\n out.append(\"```\n\n\\n\" + text_of(child).strip(\"\\n\") + \"\\n\n\n```\")\n elif tag in (\"ul\", \"ol\"):\n render_list(child, base, out)\n elif tag == \"table\":\n out.append(render_table(child, base))\n elif tag == \"dt\":\n out.append(collapse(text_of(child)))\n elif tag == \"blockquote\":\n quoted = []\n render(child, base, quoted)\n out.extend(\"> \" + block.replace(\"\\n\", \"\\n> \") for block in quoted)\n else:\n render(child, base, out)\n flush()\n```\n\nDefinition terms (`\n
\"\"\"\n self.assertEqual(md(html), [\"# Title\", \"Hello **bold** world.\"])\n```\n\n*Excerpt of `test_page_to_chunks.py`, lines 59-73:*\n\n``` python\n def test_compare(self):\n old = chunk([\"# A\", \"same\", \"# B\", \"[x](https://v1)\", \"# C\", \"old text\", \"# D\", \"gone\"])\n new = chunk([\"# A\", \"same\", \"# B\", \"[x](https://v2)\", \"# C\", \"new text\", \"# E\", \"fresh\"])\n self.assertEqual(compare(old, new), {\"added\": [\"E\"], \"removed\": [\"D\"], \"renamed\": [],\n \"changed\": [\"C\"], \"links-only\": [\"B\"], \"unchanged\": [\"A\"]})\n\n def test_renamed_heading_is_paired(self):\n body = \"The json module can be run from the shell to validate and pretty-print input.\"\n old = chunk([\"# Command Line Interface\", body])\n new = chunk([\"# Command-line interface\", body + \" Also as python -m json.\"])\n result = compare(old, new)\n self.assertEqual((result[\"added\"], result[\"removed\"]), ([], []))\n self.assertEqual(len(result[\"renamed\"]), 1)\n self.assertTrue(result[\"renamed\"][0].startswith(\"Command Line Interface -> Command-line interface (\"))\n self.assertEqual(compare(old, new, rename_threshold=0.99)[\"renamed\"], [])\n```\n\n*Excerpt of `test-log.txt`, lines 14-17:*\n\n```\n----------------------------------------------------------------------\nRan 12 tests in 0.002s\n\nOK\n```\n\nConvert both versions:\n\n``` bash\n$ python page_to_chunks.py convert https://docs.python.org/3.13/library/json.html json-3.13.md json-3.13.json\nhttps://docs.python.org/3.13/library/json.html at 2026-09-27T13:07:49Z: 110,495 characters of HTML -> 30,089 characters of Markdown in 12 chunks\n$ python page_to_chunks.py convert https://docs.python.org/3.14/library/json.html json-3.14.md json-3.14.json\nhttps://docs.python.org/3.14/library/json.html at 2026-09-27T13:07:51Z: 112,591 characters of HTML -> 30,296 characters of Markdown in 12 chunks\n```\n\nAbout 110,000 characters of HTML became about 30,000 characters of Markdown: navigation, sidebars, scripts, styles and tags are gone. The 3.14 page splits into these twelve chunks:\n\n``` bash\n$ python list_chunks.py json-3.14.json\n 530 words 7748786f6be31bbd `json` — JSON encoder and decoder\n 1092 words 9e39ef8ad501eb91 `json` — JSON encoder and decoder > Basic Usage\n 1061 words 16519e3a65bd9a4f `json` — JSON encoder and decoder > Encoders and Decoders\n 50 words b697d5281fe94b36 `json` — JSON encoder and decoder > Exceptions\n 120 words 33131999719582ed `json` — JSON encoder and decoder > Standard Compliance and Interoperability\n 194 words de4623597eb4dc0c `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Character Encodings\n 102 words b48c164cc407d4a2 `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Infinite and NaN Number Values\n 79 words 065b393d5fffcd15 `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Repeated Names Within an Object\n 86 words f97d8636fc31a9fc `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Top-level Non-Object, Non-Array Values\n 133 words 9f09bd8d5c282095 `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Implementation Limitations\n 138 words a6742132e3ec15c4 `json` — JSON encoder and decoder > Command-line interface\n 129 words e253fc9726be8808 `json` — JSON encoder and decoder > Command-line interface > Command-line options\n```\n\nA piece of the Markdown, the decoding table from the 3.14 page, shows how an HTML table comes out:\n\n*Excerpt of `json-3.14.md`, lines 245-254:*\n\n```\n| JSON | Python |\n| --- | --- |\n| object | dict |\n| array | list |\n| string | str |\n| number (int) | int |\n| number (real) | float |\n| true | True |\n| false | False |\n| null | None |\n```\n\nNow compare the two versions:\n\n``` bash\n$ python page_to_chunks.py compare json-3.13.json json-3.14.json\nhttps://docs.python.org/3.13/library/json.html -> https://docs.python.org/3.14/library/json.html\nadded 0\nremoved 0\nrenamed 2\n `json` — JSON encoder and decoder > Command Line Interface -> `json` — JSON encoder and decoder > Command-line interface (similarity 0.758)\n `json` — JSON encoder and decoder > Command Line Interface > Command line options -> `json` — JSON encoder and decoder > Command-line interface > Command-line options (similarity 0.996)\nchanged 1\n `json` — JSON encoder and decoder\nlinks-only 6\n `json` — JSON encoder and decoder > Basic Usage\n `json` — JSON encoder and decoder > Encoders and Decoders\n `json` — JSON encoder and decoder > Exceptions\n `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Character Encodings\n `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Top-level Non-Object, Non-Array Values\n `json` — JSON encoder and decoder > Standard Compliance and Interoperability > Implementation Limitations\nunchanged 3\n```\n\nReading that result:\n\n`/3.13/...`; in the 3.14 page they point to `/3.14/...`. A plain byte hash marks all six as changed. The fingerprint, which ignores link targets, shows that the words are identical. The check below confirms it: in each of the six, swapping `/3.13/` for `/3.14/` gives exactly the 3.14 text.\n\n``` bash\n$ python check_links_only.py\nBasic Usage: 50 links, 49 differ; identical once /3.13/ is replaced by /3.14/: True\nEncoders and Decoders: 19 links, 18 differ; identical once /3.13/ is replaced by /3.14/: True\nExceptions: 1 links, 1 differ; identical once /3.13/ is replaced by /3.14/: True\nStandard Compliance and Interoperability > Character Encodings: 3 links, 3 differ; identical once /3.13/ is replaced by /3.14/: True\nStandard Compliance and Interoperability > Top-level Non-Object, Non-Array Values: 4 links, 2 differ; identical once /3.13/ is replaced by /3.14/: True\nStandard Compliance and Interoperability > Implementation Limitations: 2 links, 2 differ; identical once /3.13/ is replaced by /3.14/: True\nunchanged: Standard Compliance and Interoperability | links: 2\nunchanged: Standard Compliance and Interoperability > Infinite and NaN Number Values | links: 0\nunchanged: Standard Compliance and Interoperability > Repeated Names Within an Object | links: 0\n```\n\n`python -m json`, with `python -m json.tool` kept for backwards compatibility. A renamed section can still have changed content, so re-embed it.\nFor a RAG index, that means re-embedding three chunks (one changed, two renamed) out of twelve, and only updating the stored link targets of six others.\n\n`html5lib` or `lxml` for parsing if the output looks wrong.`html.parser`: `difflib.SequenceMatcher`: `hashlib`: `urllib.robotparser`: \n\n```\n\"\"\"Turn an HTML page into clean Markdown chunks, and compare two chunk sets.\n\nUsage:\n python page_to_chunks.py convert URL out.md out.json\n python page_to_chunks.py compare old.json new.json\n\"\"\"\nimport difflib\nimport hashlib\nimport json\nimport re\nimport sys\nimport time\nimport urllib.error\nimport urllib.parse\nimport urllib.request\nimport urllib.robotparser\nfrom datetime import datetime, timezone\nfrom html.parser import HTMLParser\n\nUSER_AGENT = \"html-to-chunks-example/1.0 (tutorial script; one request per page)\"\nVOID = {\"area\", \"base\", \"br\", \"col\", \"embed\", \"hr\", \"img\", \"input\", \"link\", \"meta\", \"source\", \"track\", \"wbr\"}\nSKIP = {\"script\", \"style\", \"noscript\", \"template\", \"svg\", \"nav\", \"footer\", \"form\",\n \"button\", \"iframe\", \"aside\"}\nNOISE_CLASSES = {\"headerlink\"} # e.g. the pilcrow permalinks Sphinx adds after headings\nBLOCKS = {\"p\", \"div\", \"section\", \"article\", \"main\", \"blockquote\", \"pre\", \"ul\", \"ol\", \"table\",\n \"dl\", \"dt\", \"dd\", \"hr\", \"figure\", \"h1\", \"h2\", \"h3\", \"h4\", \"h5\", \"h6\"} | SKIP\n\nclass TreeBuilder(HTMLParser):\n \"\"\"Build a small tree of {\"tag\", \"attrs\", \"children\"} dicts; text stays as plain strings.\"\"\"\n\n def __init__(self):\n super().__init__(convert_charrefs=True)\n self.root = {\"tag\": \"#root\", \"attrs\": {}, \"children\": []}\n self.stack = [self.root]\n\n def handle_starttag(self, tag, attrs):\n node = {\"tag\": tag, \"attrs\": {k: v or \"\" for k, v in attrs}, \"children\": []}\n self.stack[-1][\"children\"].append(node)\n if tag not in VOID:\n self.stack.append(node)\n\n def handle_endtag(self, tag):\n for depth in range(len(self.stack) - 1, 0, -1): # close the nearest open match, if any\n if self.stack[depth][\"tag\"] == tag:\n del self.stack[depth:]\n break\n\n def handle_data(self, data):\n self.stack[-1][\"children\"].append(data)\n\ndef find(node, test):\n if isinstance(node, dict):\n if test(node):\n return node\n for child in node[\"children\"]:\n hit = find(child, test)\n if hit:\n return hit\n return None\n\ndef main_content(root):\n for test in (lambda n: n[\"tag\"] == \"main\", lambda n: n[\"tag\"] == \"article\",\n lambda n: n[\"attrs\"].get(\"role\") == \"main\", lambda n: n[\"tag\"] == \"body\"):\n hit = find(root, test)\n if hit:\n return hit\n return root\n\ndef ignored(node):\n return node[\"tag\"] in SKIP or bool(NOISE_CLASSES & set(node[\"attrs\"].get(\"class\", \"\").split()))\n\ndef collapse(text):\n return re.sub(r\"\\s+\", \" \", text).strip()\n\ndef text_of(node):\n \"\"\"Raw text with whitespace kept (for Title¶
\nHello bold\n world.
), minus ignored elements.\"\"\"\n if isinstance(node, str):\n return node\n return \"\" if ignored(node) else \"\".join(text_of(child) for child in node[\"children\"])\n\ndef inline(node, base):\n \"\"\"Render inline content as Markdown on one line.\"\"\"\n if isinstance(node, str):\n return node\n if ignored(node):\n return \"\"\n tag, attrs = node[\"tag\"], node[\"attrs\"]\n if tag == \"br\":\n return \" \"\n if tag == \"img\":\n return attrs.get(\"alt\", \"\")\n inner = \"\".join(inline(child, base) for child in node[\"children\"])\n if tag == \"code\":\n return f\"`{collapse(text_of(node))}`\" if collapse(text_of(node)) else \"\"\n if tag in (\"strong\", \"b\", \"em\", \"i\") and collapse(inner):\n mark = \"**\" if tag in (\"strong\", \"b\") else \"*\"\n return f\"{mark}{collapse(inner)}{mark}\"\n href = attrs.get(\"href\", \"\")\n if tag == \"a\" and collapse(inner) and href and not href.startswith((\"#\", \"javascript:\")):\n return f\"[{collapse(inner)}]({urllib.parse.urljoin(base, href)})\"\n return inner\n\ndef render_list(node, base, out):\n number = 0\n for item in node[\"children\"]:\n if not isinstance(item, dict) or item[\"tag\"] != \"li\":\n continue\n number += 1\n marker = f\"{number}.\" if node[\"tag\"] == \"ol\" else \"-\"\n parts = []\n render(item, base, parts)\n if parts:\n indent = \"\\n\" + \" \" * (len(marker) + 1)\n out.append(marker + \" \" + indent.join(part.replace(\"\\n\", indent) for part in parts))\n\ndef render_table(node, base):\n rows = []\n for row in find_all(node, \"tr\"):\n cells = [collapse(inline(cell, base)).replace(\"|\", \"\\\\|\") for cell in row[\"children\"]\n if isinstance(cell, dict) and cell[\"tag\"] in (\"th\", \"td\")]\n if cells:\n rows.append(cells)\n if not rows:\n return \"\"\n width = max(len(row) for row in rows)\n rows = [row + [\"\"] * (width - len(row)) for row in rows]\n lines = [\"| \" + \" | \".join(rows[0]) + \" |\", \"|\" + \" --- |\" * width]\n return \"\\n\".join(lines + [\"| \" + \" | \".join(row) + \" |\" for row in rows[1:]])\n\ndef find_all(node, tag):\n for child in node[\"children\"]:\n if isinstance(child, dict):\n if child[\"tag\"] == tag:\n yield child\n elif child[\"tag\"] != \"table\": # do not descend into nested tables\n yield from find_all(child, tag)\n\ndef render(node, base, out):\n \"\"\"Append Markdown blocks (strings) for the children of `node` to `out`.\"\"\"\n run = []\n\n def flush():\n text = collapse(\"\".join(run))\n run.clear()\n if text:\n out.append(text)\n\n for child in node[\"children\"]:\n if isinstance(child, str) or child[\"tag\"] not in BLOCKS:\n run.append(inline(child, base))\n continue\n flush()\n tag = child[\"tag\"]\n if ignored(child) or tag == \"hr\":\n continue\n if re.fullmatch(r\"h[1-6]\", tag):\n title = collapse(inline(child, base))\n if title:\n out.append(\"#\" * int(tag[1]) + \" \" + title)\n elif tag == \"pre\":\n out.append(\"```\n\n\\n\" + text_of(child).strip(\"\\n\") + \"\\n\n\n```\")\n elif tag in (\"ul\", \"ol\"):\n render_list(child, base, out)\n elif tag == \"table\":\n out.append(render_table(child, base))\n elif tag == \"dt\":\n out.append(collapse(text_of(child)))\n elif tag == \"blockquote\":\n quoted = []\n render(child, base, quoted)\n out.extend(\"> \" + block.replace(\"\\n\", \"\\n> \") for block in quoted)\n else:\n render(child, base, out)\n flush()\n\ndef to_markdown(html, base):\n builder = TreeBuilder()\n builder.feed(html)\n builder.close()\n blocks = []\n render(main_content(builder.root), base, blocks)\n return [block for block in blocks if block]\n\ndef prose(text):\n \"\"\"Text with link targets and whitespace differences removed.\"\"\"\n return collapse(re.sub(r\"\\]\\([^)]*\\)\", \"]\", text))\n\ndef fingerprint(text):\n return hashlib.sha256(prose(text).encode(\"utf-8\")).hexdigest()[:16]\n\ndef chunk(blocks, max_level=3):\n \"\"\"Group blocks under their nearest heading (h1..h{max_level}); one chunk per section.\"\"\"\n chunks, path, body, seen = [], [], [], {}\n\n def emit():\n text = \"\\n\\n\".join(body).strip()\n body.clear()\n if text:\n name = \" > \".join(path) or \"(top)\"\n seen[name] = seen.get(name, 0) + 1\n chunk_id = name if seen[name] == 1 else f\"{name} #{seen[name]}\"\n chunks.append({\"id\": chunk_id, \"words\": len(text.split()),\n \"sha256\": hashlib.sha256(text.encode(\"utf-8\")).hexdigest()[:16],\n \"fingerprint\": fingerprint(text), \"text\": text})\n\n for block in blocks:\n heading = re.match(r\"(#{1,6}) (.*)\", block)\n if heading and len(heading.group(1)) <= max_level:\n emit()\n del path[len(heading.group(1)) - 1:]\n path.append(heading.group(2))\n body.append(block)\n emit()\n return chunks\n\ndef compare(old, new, rename_threshold=0.6):\n \"\"\"Classify chunks by id: added, removed, renamed, changed (prose), links-only, unchanged.\"\"\"\n before = {c[\"id\"]: c for c in old}\n after = {c[\"id\"]: c for c in new}\n result = {\"added\": sorted(after.keys() - before.keys()),\n \"removed\": sorted(before.keys() - after.keys()),\n \"renamed\": [], \"changed\": [], \"links-only\": [], \"unchanged\": []}\n for old_id in list(result[\"removed\"]): # a renamed heading looks like removed + added\n scored = [(difflib.SequenceMatcher(None, prose(before[old_id][\"text\"]),\n prose(after[new_id][\"text\"])).ratio(), new_id)\n for new_id in result[\"added\"]]\n if scored and max(scored)[0] >= rename_threshold:\n ratio, new_id = max(scored)\n result[\"removed\"].remove(old_id)\n result[\"added\"].remove(new_id)\n result[\"renamed\"].append(f\"{old_id} -> {new_id} (similarity {ratio:.3f})\")\n for chunk_id in [c[\"id\"] for c in new if c[\"id\"] in before]:\n a, b = before[chunk_id], after[chunk_id]\n kind = (\"unchanged\" if a[\"sha256\"] == b[\"sha256\"] else\n \"links-only\" if a[\"fingerprint\"] == b[\"fingerprint\"] else \"changed\")\n result[kind].append(chunk_id)\n return result\n\ndef robots_rules(text):\n \"\"\"Parse robots.txt. urllib.robotparser ends a group at a blank line; RFC 9309 does not.\"\"\"\n rules = urllib.robotparser.RobotFileParser()\n rules.parse([line for line in text.splitlines() if line.strip()])\n return rules\n\ndef fetch(url):\n parts = urllib.parse.urlsplit(url)\n try:\n text = get(f\"{parts.scheme}://{parts.netloc}/robots.txt\")\n except urllib.error.HTTPError as err:\n if err.code not in (404, 410):\n raise\n text = \"\" # no robots.txt: nothing is disallowed\n rules = robots_rules(text)\n if not rules.can_fetch(USER_AGENT, url):\n sys.exit(f\"robots.txt disallows {url}; stopping\")\n time.sleep(max(1, rules.crawl_delay(USER_AGENT) or 0))\n return get(url)\n\ndef get(url):\n request = urllib.request.Request(url, headers={\"User-Agent\": USER_AGENT})\n with urllib.request.urlopen(request, timeout=30) as response:\n return response.read().decode(response.headers.get_content_charset() or \"utf-8\", \"replace\")\n\ndef main(args):\n if args[:1] == [\"convert\"] and len(args) == 4:\n url, md_file, json_file = args[1:]\n html = fetch(url)\n captured = datetime.now(timezone.utc).strftime(\"%Y-%m-%dT%H:%M:%SZ\")\n blocks = to_markdown(html, url)\n chunks = chunk(blocks)\n with open(md_file, \"w\", encoding=\"utf-8\") as handle:\n handle.write(\"\\n\\n\".join(blocks) + \"\\n\")\n with open(json_file, \"w\", encoding=\"utf-8\") as handle:\n json.dump({\"url\": url, \"captured_at\": captured, \"chunks\": chunks}, handle, indent=2)\n print(f\"{url} at {captured}: {len(html):,} characters of HTML -> \"\n f\"{sum(len(b) for b in blocks):,} characters of Markdown in {len(chunks)} chunks\")\n elif args[:1] == [\"compare\"] and len(args) == 3:\n with open(args[1], encoding=\"utf-8\") as a, open(args[2], encoding=\"utf-8\") as b:\n old, new = json.load(a), json.load(b)\n result = compare(old[\"chunks\"], new[\"chunks\"])\n print(f\"{old['url']} -> {new['url']}\")\n for kind, ids in result.items():\n print(f\"{kind:<10} {len(ids):>3}\" + \"\".join(f\"\\n {i}\" for i in ids if kind != \"unchanged\"))\n else:\n sys.exit(__doc__)\n\nif __name__ == \"__main__\":\n main(sys.argv[1:])\npython\nimport unittest\n\nfrom page_to_chunks import chunk, compare, fingerprint, robots_rules, to_markdown\n\nBASE = \"https://example.org/docs/page.html\"\n\ndef md(html):\n return to_markdown(html, BASE)\n\nclass Markdown(unittest.TestCase):\n def test_main_content_only_and_noise_removed(self):\n html = \"\"\"\n Title¶
\n Hello bold\n world.
\"\"\"\n self.assertEqual(md(html), [\"# Title\", \"Hello **bold** world.\"])\n\n def test_links_become_absolute_and_anchors_become_text(self):\n html = 'Call json.dumps( ):
>>> x = 1\\n y
One
- a
- b
- Two
JSON Python object
dict a|b
- json.dump(obj, fp)¶
\"\n \"Serialize.
Quoted
First
Second
outside\"), [\"First\", \"Second\"])\n\nclass Chunks(unittest.TestCase):\n def test_heading_paths_levels_and_duplicates(self):\n blocks = [\"intro\", \"# Guide\", \"a\", \"## Setup\", \"b\", \"#### Deep\", \"c\", \"## Setup\", \"d\", \"### Notes\", \"e\"]\n result = chunk(blocks)\n self.assertEqual([c[\"id\"] for c in result],\n [\"(top)\", \"Guide\", \"Guide > Setup\", \"Guide > Setup #2\", \"Guide > Setup > Notes\"])\n self.assertEqual(result[2][\"text\"], \"## Setup\\n\\nb\\n\\n#### Deep\\n\\nc\")\n\n def test_fingerprint_ignores_link_targets(self):\n self.assertEqual(fingerprint(\"See [docs](https://x/3.13/a.html).\"),\n fingerprint(\"See docs.\"))\n self.assertNotEqual(fingerprint(\"See [docs](u).\"), fingerprint(\"See [the docs](u).\"))\n\n def test_compare(self):\n old = chunk([\"# A\", \"same\", \"# B\", \"[x](https://v1)\", \"# C\", \"old text\", \"# D\", \"gone\"])\n new = chunk([\"# A\", \"same\", \"# B\", \"[x](https://v2)\", \"# C\", \"new text\", \"# E\", \"fresh\"])\n self.assertEqual(compare(old, new), {\"added\": [\"E\"], \"removed\": [\"D\"], \"renamed\": [],\n \"changed\": [\"C\"], \"links-only\": [\"B\"], \"unchanged\": [\"A\"]})\n\n def test_renamed_heading_is_paired(self):\n body = \"The json module can be run from the shell to validate and pretty-print input.\"\n old = chunk([\"# Command Line Interface\", body])\n new = chunk([\"# Command-line interface\", body + \" Also as python -m json.\"])\n result = compare(old, new)\n self.assertEqual((result[\"added\"], result[\"removed\"]), ([], []))\n self.assertEqual(len(result[\"renamed\"]), 1)\n self.assertTrue(result[\"renamed\"][0].startswith(\"Command Line Interface -> Command-line interface (\"))\n self.assertEqual(compare(old, new, rename_threshold=0.99)[\"renamed\"], [])\n\nclass Robots(unittest.TestCase):\n def test_blank_line_does_not_end_the_group(self):\n rules = robots_rules(\"User-agent: *\\nDisallow: /dev\\n\\n# EOL versions\\nDisallow: /3.9/\\n\")\n self.assertFalse(rules.can_fetch(\"page-to-chunks\", \"https://x/3.9/library/json.html\"))\n self.assertFalse(rules.can_fetch(\"page-to-chunks\", \"https://x/dev/\"))\n self.assertTrue(rules.can_fetch(\"page-to-chunks\", \"https://x/3.14/library/json.html\"))\n\nif __name__ == \"__main__\":\n unittest.main()\n```\n\nRun the tests with `python -m unittest -v test_page_to_chunks`.\n\n*This article and its code were drafted by an AI assistant at the account owner's request. The code was run against docs.python.org on 27 September 2026 at 13:07:49 and 13:07:51 UTC, and the tests passed the same day. The blank-line handling for `robots.txt` was added later the same day after a review; it makes the same `robots.txt` decisions for both pages, so it does not change the run shown above.*", "url": "https://wpnews.pro/news/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed", "canonical_source": "https://dev.to/abdulwhab95/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed-1ibm", "published_at": "2026-09-27 18:49:20+00:00", "updated_at": "2026-09-27 19:01:09.956215+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "natural-language-processing"], "entities": ["Python", "Sphinx", "html.parser", "html5lib", "lxml", "GitHub Flavored Markdown", "CommonMark"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed", "markdown": "https://wpnews.pro/news/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed.md", "text": "https://wpnews.pro/news/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed.txt", "jsonld": "https://wpnews.pro/news/html-to-clean-markdown-chunks-in-python-and-spotting-what-really-changed.jsonld"}}