{"slug": "merkle-trees-and-inclusion-proofs-in-python-from-scratch", "title": "Merkle Trees and Inclusion Proofs in Python From Scratch", "summary": "A developer demonstrated building a Merkle tree from scratch in Python to generate and verify inclusion proofs for AI retrieval provenance, letting anyone confirm a source document was part of the set a model saw without re-running the pipeline. The implementation uses domain separation with distinct leaf (0x00) and internal node (0x01) prefixes to prevent second-preimage attacks, and promotes lone odd nodes unchanged to the next level. Proofs require only O(log n) sibling hashes, each tagged with its left/right position, so a verifier trusting the root can recompute the path and confirm membership.", "body_md": "I kept running into the same question while building AI retrieval systems: how do you prove, after the fact, that a particular source document was actually part of the set the model saw? You want a compact receipt that anyone can check without re-running your whole pipeline. Merkle trees are the classic answer, and once I built one by hand I realized the idea is much simpler than the cryptography reputation suggests.\n\nIn this tutorial I will build a Merkle tree from scratch in Python, generate an inclusion proof for a single leaf, and verify that proof standalone. I will use retrieval provenance as the running example, but the technique applies to any situation where you want to commit to a set of items and later prove membership: transaction logs, file backups, certificate transparency, and so on.\n\nA Merkle tree hashes your data in a binary tree. The leaves are hashes of your items. Each internal node is the hash of its two children concatenated. The single hash at the top, the Merkle root, is a fingerprint of the entire set. Change any item, and the root changes.\n\nThe magic is the inclusion proof. To prove a leaf is in the tree, you do not need the whole tree. You only need the sibling hashes along the path from that leaf up to the root. That is O(log n) hashes. A verifier who trusts the root can recompute their way up and confirm they arrive at the same root.\n\nBefore any code, two things separate a toy Merkle tree from a correct one.\n\nFirst, domain separation. If you hash leaves and internal nodes the same way, an attacker can present an internal node as if it were a leaf. This is the classic second-preimage weakness. The fix is to prefix a different byte before hashing leaves versus internal nodes. I use `0x00` for leaves and `0x01` for internal nodes.\n\nSecond, odd-node handling. When a level has an odd number of nodes, one node has no sibling to pair with. Different systems handle this differently. I promote the lone node unchanged to the next level. This is simple and deterministic, though I will flag a caveat about it later.\n\nLet me start with the two hashing primitives.\n\n``` python\nimport hashlib\n\nLEAF_PREFIX = b\"\\x00\"\nNODE_PREFIX = b\"\\x01\"\n\ndef hash_leaf(data: bytes) -> bytes:\n    return hashlib.sha256(LEAF_PREFIX + data).digest()\n\ndef hash_node(left: bytes, right: bytes) -> bytes:\n    return hashlib.sha256(NODE_PREFIX + left + right).digest()\n```\n\nNow the tree. I build it level by level, keeping every level so I can generate proofs later.\n\n``` python\nclass MerkleTree:\n    def __init__(self, items: list[bytes]):\n        if not items:\n            raise ValueError(\"cannot build a Merkle tree over zero items\")\n        self.leaves = [hash_leaf(item) for item in items]\n        self.levels = self._build(self.leaves)\n\n    def _build(self, leaves: list[bytes]) -> list[list[bytes]]:\n        levels = [leaves]\n        current = leaves\n        while len(current) > 1:\n            nxt = []\n            for i in range(0, len(current), 2):\n                if i + 1 < len(current):\n                    nxt.append(hash_node(current[i], current[i + 1]))\n                else:\n                    # odd node: promote it unchanged\n                    nxt.append(current[i])\n            levels.append(nxt)\n            current = nxt\n        return levels\n\n    @property\n    def root(self) -> bytes:\n        return self.levels[-1][0]\n```\n\nThe `levels` list holds the leaf hashes at index 0, then each successive parent level, ending with a single-element level that holds the root.\n\nA proof is the list of sibling hashes on the way up, each tagged with whether it sits on the left or right. The verifier needs the side so it concatenates in the correct order.\n\n``` php\n    def proof(self, index: int) -> list[tuple[str, bytes]]:\n        if not 0 <= index < len(self.leaves):\n            raise IndexError(\"leaf index out of range\")\n        path = []\n        for level in self.levels[:-1]:\n            is_right_node = index % 2\n            if is_right_node:\n                sibling_index = index - 1\n                path.append((\"left\", level[sibling_index]))\n            else:\n                sibling_index = index + 1\n                if sibling_index < len(level):\n                    path.append((\"right\", level[sibling_index]))\n                # else: promoted odd node, no sibling at this level\n            index //= 2\n        return path\n```\n\nWhen our node is on the right, its sibling is to the left, and vice versa. If our node is a left node at the end of an odd level, it has no sibling and was promoted, so we add nothing for that level. We then move up by halving the index.\n\nThis is the part that makes Merkle trees useful. Verification needs only three things: the original item, the proof, and the trusted root. No tree, no other items.\n\n``` php\ndef verify_proof(item: bytes, proof: list[tuple[str, bytes]], root: bytes) -> bool:\n    computed = hash_leaf(item)\n    for side, sibling in proof:\n        if side == \"left\":\n            computed = hash_node(sibling, computed)\n        else:\n            computed = hash_node(computed, computed if False else sibling)\n    return computed == root\n```\n\nLet me clean that verify up, since the ternary is noise:\n\n``` php\ndef verify_proof(item: bytes, proof: list[tuple[str, bytes]], root: bytes) -> bool:\n    computed = hash_leaf(item)\n    for side, sibling in proof:\n        if side == \"left\":\n            computed = hash_node(sibling, computed)\n        else:\n            computed = hash_node(computed, sibling)\n    return computed == root\n```\n\nThe verifier hashes the item as a leaf, then walks the proof, folding in each sibling on the correct side, and checks whether it lands on the root.\n\nNow the concrete use case. Say a RAG system retrieved five source chunks to answer a question. We want a receipt proving that one specific chunk, say the one the answer cited, was in that retrieval set.\n\n```\nsources = [\n    b\"doc:handbook#p12 :: refunds within 30 days\",\n    b\"doc:handbook#p13 :: return shipping is prepaid\",\n    b\"doc:policy#p4    :: no refunds on final-sale items\",\n    b\"doc:faq#q9       :: exchanges allowed within 60 days\",\n    b\"doc:handbook#p14 :: store credit never expires\",\n]\n\ntree = MerkleTree(sources)\nroot = tree.root  # publish or log this as the retrieval commitment\n\ncited_index = 2\ncited_source = sources[cited_index]\nreceipt = tree.proof(cited_index)\n\nprint(\"root:\", root.hex())\nprint(\"verified:\", verify_proof(cited_source, receipt, root))\n\n# Tamper check: a source that was never retrieved must fail\nfake = b\"doc:policy#p4    :: refunds on final-sale items are fine\"\nprint(\"tampered:\", verify_proof(fake, receipt, root))\n```\n\nRunning this prints `verified: True` and `tampered: False`. The tampered line proves the point: flip a single word in the source and the proof no longer reconstructs the root. If you log the root at retrieval time, anyone holding a source and its receipt can later confirm it was genuinely part of that set, without access to your database or the other four chunks. That is exactly the property I wanted for retrieval provenance in my project answerproof.\n\nThe odd-node promotion I used is simple, but it is worth knowing its weakness. Because a promoted node is carried up unchanged, a tree over an odd count can, in adversarial constructions, share structure with a differently shaped tree, which is the root of the well known CVE-2012-2459 style ambiguity in some Merkle implementations. For a trusted producer logging its own retrieval sets this is fine. If you need to defend against a malicious tree builder, use a scheme that removes the ambiguity: duplicate the lone node instead of promoting it, or better, prepend the leaf count to what you hash and reject proofs whose length does not match the committed size. Domain separation, which I did include, already blocks the leaf-versus-node confusion, so we are only talking about the shape ambiguity here.\n\nThat is a complete, correct Merkle tree in well under a hundred lines: domain-separated leaf and node hashing, level-by-level construction with odd-node promotion, logarithmic inclusion proofs, and standalone verification that needs nothing but the item, the receipt, and the root. The technique is general, but retrieval provenance made it click for me: a tiny receipt that survives long after the pipeline that produced it.\n\nIf you want to see this idea wired into a real retrieval-answer flow, the provenance work lives in github.com/AgentPostmortem/answerproof. Clone the snippets above, break something on purpose, and watch the root refuse to lie.", "url": "https://wpnews.pro/news/merkle-trees-and-inclusion-proofs-in-python-from-scratch", "canonical_source": "https://dev.to/royalpinto007/merkle-trees-and-inclusion-proofs-in-python-from-scratch-25lo", "published_at": "2026-09-27 09:30:31+00:00", "updated_at": "2026-09-27 10:01:20.897147+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "developer-tools"], "entities": ["Python", "SHA-256"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/merkle-trees-and-inclusion-proofs-in-python-from-scratch", "markdown": "https://wpnews.pro/news/merkle-trees-and-inclusion-proofs-in-python-from-scratch.md", "text": "https://wpnews.pro/news/merkle-trees-and-inclusion-proofs-in-python-from-scratch.txt", "jsonld": "https://wpnews.pro/news/merkle-trees-and-inclusion-proofs-in-python-from-scratch.jsonld"}}