{"slug": "finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for", "title": "Finding near-duplicate documents with MinHash and LSH before you index them for RAG", "summary": "A developer demonstrated a MinHash and locality-sensitive hashing (LSH) pipeline for detecting near-duplicate documents before indexing them for RAG, using 5-word shingles, 128-permutation MinHash signatures built with salted blake2b hashes, and 32-band LSH bucketing to surface candidate pairs. In a four-document test, the approach flagged a re-exported refund policy as a near-copy while distinguishing a version 2 that changed 14 days to 30 and 5 business days to 7, which measured 66% similar. The developer's guidance: deduplicate copies, never deduplicate versions.", "body_md": "Document collections are full of **near-duplicates**: the same policy exported twice, a page with a new footer, version 1 and version 2 of a contract. Indexed for RAG, they **fill the top-k results with copies of one passage** and crowd out everything else. **MinHash** finds them efficiently: split each document into overlapping word **shingles**, and compare documents by **Jaccard similarity** (shared shingles ÷ all shingles). MinHash compresses each document into a short **signature** whose matching positions estimate Jaccard, and **LSH banding** finds candidate pairs without comparing every document to every other. One warning from my test: **two versions of a policy with different numbers are 66% similar**. Deduplicate copies; never deduplicate versions.\n\nA shingle is a run of k consecutive words. Two documents that share most of their 5-word runs are near-copies; a reworded document shares few.\n\n``` python title=\"minhash.py: shingles\"\n\nimport re\n\ndef shingles(text: str, k: int = 5) -> set[str]:\n\n    words = re.findall(r\"\\w+\", text.lower())\n\n    return {\" \".join(words[i:i + k]) for i in range(max(1, len(words) - k + 1))}\n\ndef jaccard(a: set, b: set) -> float:\n\n    return len(a & b) / len(a | b) if a | b else 1.0\n\n```\nExact Jaccard needs every pair of documents compared shingle by shingle: fine for a hundred documents, too slow for a million.\n\n## MinHash: a signature that estimates Jaccard\n\nHash every shingle with a seeded hash function and keep the **minimum**. The chance that two documents have the same minimum equals their Jaccard similarity. Repeat with 128 different seeds, and the fraction of matching positions estimates it:\n\n``` python title=\"minhash.py: signatures\"\nimport hashlib\n\ndef _h(seed: int, s: str) -> int:\n    return int.from_bytes(hashlib.blake2b(s.encode(), digest_size=8, salt=seed.to_bytes(8, \"little\")).digest(), \"little\")\n\ndef minhash(sh: set[str], num_perm: int = 128) -> list[int]:\n    return [min(_h(seed, s) for s in sh) for seed in range(num_perm)]\n\ndef estimate(sig_a: list[int], sig_b: list[int]) -> float:\n    return sum(x == y for x, y in zip(sig_a, sig_b)) / len(sig_a)\n```\n\n`blake2b` with a salt gives deterministic seeded hashes from the standard library. Python's built-in `hash()` is randomised per process, so signatures stored in one run wouldn't match the next.\n\nSplit each 128-number signature into **32 bands of 4**. Two documents that are identical in **any** band land in the same bucket and become a candidate pair; only candidates are compared.\n\n``` python title=\"minhash.py: LSH\"\n\nfrom collections import defaultdict\n\ndef near_duplicates(docs: dict[str, str], threshold: float = 0.8, bands: int = 32) -> list[tuple[str, str, float]]:\n\n    \"\"\"LSH: split each signature into bands; documents sharing any identical band become candidates.\"\"\"\n\n    sigs = {doc_id: minhash(shingles(text)) for doc_id, text in docs.items()}\n\n    rows = len(next(iter(sigs.values()))) // bands\n\n    buckets = defaultdict(set)\n\n    for doc_id, sig in sigs.items():\n\n        for b in range(bands):\n\n            buckets[(b, tuple(sig[b * rows:(b + 1) * rows]))].add(doc_id)\n\n    candidates = {tuple(sorted(pair)) for ids in buckets.values() for pair in _pairs(ids)}\n\n    found = [(a, b, estimate(sigs[a], sigs[b])) for a, b in candidates]\n\n    return sorted([f for f in found if f[2] >= threshold], key=lambda f: -f[2])\n\ndef _pairs(ids):\n\n    ids = sorted(ids)\n\n    return [(a, b) for i, a in enumerate(ids) for b in ids[i + 1:]]\n\n```\nMore bands with fewer rows catch lower similarities but produce more candidates to check. With 32 bands of 4, the chance a pair becomes a candidate is `1 - (1 - s⁴)³²`: almost certain at 0.8 similarity, about 87% at 0.5, and about 5% at 0.2.\n\n## What did the test show?\n\nFour documents: a refund policy, the same policy with a \"Last updated\" line, a **version 2** where 14 days became 30 and 5 business days became 7, and an unrelated shipping page.\n\n| Pair                    | Exact Jaccard | MinHash estimate |\n| ----------------------- | ------------- | ---------------- |\n| Policy vs its copy      | 0.92          | 0.93             |\n| Policy v1 vs v2         | 0.66          | 0.62             |\n| Policy vs shipping page | 0.00          | 0.00             |\n\nAt a 0.8 threshold, only the copy is found, which is right. At 0.5, **version 1 and version 2 are flagged as duplicates**. Dropping one of them by similarity alone could leave the outdated refund period in the index. That's the case to design for.\n\n## How should you act on duplicates?\n\n* **Copies** (above \\~0.9): keep one, preferably the canonical URL or newest file, and record the others as aliases so citations still resolve.\n* **Versions** (same document, different content): don't drop either blindly. Keep the **current** one in the index and archive the rest, using dates or version metadata, not similarity.\n* **Boilerplate** shared across many documents (headers, disclaimers) inflates similarity; strip it before shingling.\n\nDeduplication changes what retrieval returns, so measure before and after with the metrics in [how to evaluate a RAG system](https://aliahmed.si/blog/rag-retrieval-evaluation-metrics/). How you split documents matters too; see [RAG chunking strategies](https://aliahmed.si/blog/rag-chunking-strategies/).\n\n## Summary\n\n* Near-duplicates waste a RAG index and crowd out diverse results.\n* Shingle documents, estimate Jaccard with **MinHash**, and find candidates with **LSH banding**.\n* Use deterministic hashes, not Python's randomised `hash()`.\n* Drop **copies**; resolve **versions** with metadata, because similar text can carry different facts.\n\nRelated: [hybrid search with BM25 and RRF](https://aliahmed.si/blog/hybrid-search-bm25-reciprocal-rank-fusion/) and [embeddings and vector search, explained](https://aliahmed.si/blog/embeddings-vector-search-explained/).\n\n---\n\nWritten by Ali Ahmed · [Original article](https://aliahmed.si/blog/minhash-near-duplicate-documents-rag/)\n```\n\n", "url": "https://wpnews.pro/news/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for", "canonical_source": "https://dev.to/quantumbreakz/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for-rag-580c", "published_at": "2026-11-19 17:37:22+00:00", "updated_at": "2026-11-19 17:43:36.177350+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "artificial-intelligence"], "entities": ["MinHash", "LSH", "blake2b", "Python"], "also_reported_by": [], "takeaway": "With 32 LSH bands of 4 rows, a pair's chance of becoming a candidate is 1-(1-s⁴)³² — near-certain at 0.8 Jaccard but only ~5% at 0.2 — and a policy version changing 14 days to 30 still scores 66% similar, so deduplicate copies but not versions.", "recommendation": {"advice": "Run MinHash plus 32-band LSH over 5-word shingles before indexing RAG documents to drop near-copies, but review pairs above ~0.66 similarity by hand so versioned content isn't merged.", "playbook": "rag-retrieval", "playbook_url": "https://wpnews.pro/playbooks/rag-retrieval"}, "alternates": {"html": "https://wpnews.pro/news/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for", "markdown": "https://wpnews.pro/news/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for.md", "text": "https://wpnews.pro/news/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for.txt", "jsonld": "https://wpnews.pro/news/finding-near-duplicate-documents-with-minhash-and-lsh-before-you-index-them-for.jsonld"}}