Building a Perceptual Hash Pipeline for Video Deduplication in Python ViralVidVault built a perceptual hash pipeline in Python to deduplicate near-identical videos that evade SHA-256. The system samples frames, computes DCT-based 64-bit hashes, and queries millions of fingerprints using SQLite and a Go helper. 'A single viral clip never stays a single file,' the team notes. A single viral clip never stays a single file. Within hours of a video trending, our crawlers pull the "same" clip back a dozen times: re-encoded at a lower bitrate, letterboxed for a vertical feed, watermarked by an aggregator, trimmed by three seconds, or mirrored with a 2% zoom to dodge platform fingerprinting. A byte-for-byte SHA-256 sees twelve unique videos. Our users see one clip spammed across their feed. For a European viral-video discovery product where feed quality is the value proposition, that is an existential bug, not a cosmetic one. At ViralVidVault https://viralvidvault.com we lean on perceptual hashing to collapse those near-duplicates before anything reaches the ranking model. This post is the actual approach we run in production: how to turn frames into hashes that survive re-encoding, how to aggregate per-frame hashes into a per-video fingerprint, and how to query millions of them without a GPU or a dedicated vector database. The heavy lifting is Python, the storage and query layer is PHP 8.4 on SQLite in WAL mode, and the hot comparison path is a small Go helper. No proprietary services, nothing that ships a raw frame off our origin. Cryptographic hashes are designed to be maximally sensitive: flip one bit of input and roughly half the output bits change. That is exactly what you want for integrity checking and exactly what you do not want for deduplication. Two visually identical frames that differ only because one went through an extra H.264 pass will produce completely unrelated SHA-256 digests. There is no notion of "close." Perceptual hashing inverts the goal. A perceptual hash pHash is a short fixed-length code where visually similar inputs produce similar codes, and "similar" is measured with Hamming distance — the number of differing bits. Two frames that a human would call the same clip land within a handful of bits of each other. The properties we actually need: The most reliable variant in our testing is the DCT-based hash. You resize a frame to a small square, take a 2D discrete cosine transform, keep only the low-frequency coefficients where the perceptually meaningful structure lives , and threshold each coefficient against the median. The result is a 64-bit integer. The first stage reads a video, samples frames at a fixed cadence, and computes one 64-bit pHash per sampled frame. Sampling every couple of seconds is enough — viral clips are short, and consecutive frames are near-identical anyway, so hashing every frame is wasted work. python import cv2 import numpy as np def dct phash frame: np.ndarray, hash size: int = 8, highfreq factor: int = 4 - int: """Return a 64-bit DCT perceptual hash for a single BGR frame.""" img size = hash size highfreq factor 32x32 by default gray = cv2.cvtColor frame, cv2.COLOR BGR2GRAY resized = cv2.resize gray, img size, img size , interpolation=cv2.INTER AREA dct = cv2.dct np.float32 resized low = dct :hash size, :hash size .flatten Exclude the DC term index 0 from the threshold; it only carries brightness. med = np.median low 1: bits = low med value = 0 for bit in bits: value = value << 1 | int bit return value def iter keyframes path: str, every seconds: float = 2.0 : """Yield timestamp seconds, frame sampled at a fixed cadence.""" cap = cv2.VideoCapture path fps = cap.get cv2.CAP PROP FPS or 25.0 step = max 1, int round fps every seconds idx = 0 try: while True: ok, frame = cap.read if not ok: break if idx % step == 0: yield idx / fps, frame idx += 1 finally: cap.release def video signature path: str - list tuple float, int : """Compute the temporal pHash signature of a video.""" return ts, dct phash frame for ts, frame in iter keyframes path A 40-second clip sampled every two seconds yields ~20 hashes: a compact temporal fingerprint that is cheap to store and cheap to compare. Two things matter here. First, INTER AREA interpolation is deliberate — it is the correct resampling filter for downscaling and avoids the ringing artifacts that bilinear introduces, which would otherwise perturb the DCT. Second, excluding the DC coefficient from the median threshold is what buys brightness invariance; the DC term is essentially the average luminance, so a video that got brightened by an aggregator would otherwise flip a predictable set of bits. A per-frame hash is not enough on its own. Viral content is full of shared segments — the same reaction meme spliced into a hundred different compilations, the same stock intro card. If you dedupe on a single frame you will merge videos that share one common shot but are otherwise unrelated. The fix is to treat the video as a set of frame hashes and ask a temporal question: what fraction of one video's frames have a near-match somewhere in the other video? Comparing every frame of every candidate against every frame of every other video is O n² and does not scale. Within a single pairwise comparison the frame counts are small, so brute force is fine there. The scaling problem is finding candidate videos in the first place, and for that we build a BK-tree — a metric tree that indexes items by Hamming distance and lets us pull "everything within k bits of this hash" in sub-linear time. php def hamming a: int, b: int - int: return a ^ b .bit count Python 3.10+ popcount class BKTree: """Burkhard-Keller tree over 64-bit hashes with Hamming distance.""" def init self - None: self.root: tuple int, dict int, object | None = None def add self, value: int - None: if self.root is None: self.root = value, {} return node, children = self.root while True: d = hamming value, node if d == 0: return exact duplicate already indexed if d in children: node, children = children d else: children d = value, {} return def query self, value: int, max dist: int - list tuple int, int : if self.root is None: return results: list tuple int, int = stack = self.root while stack: node, children = stack.pop d = hamming value, node if d <= max dist: results.append d, node Triangle inequality prunes whole subtrees. lo, hi = d - max dist, d + max dist for edge, child in children.items : if lo <= edge <= hi: stack.append child return results def temporal overlap sig a: list int , sig b: list int , max dist: int = 6 - float: """Fraction of sig a frames that have a near-match in sig b.""" if not sig a: return 0.0 matched = 0 for ha in sig a: if any hamming ha, hb <= max dist for hb in sig b : matched += 1 return matched / len sig a The BK-tree exploits the triangle inequality: if the query is distance d from a node, any item within max dist of the query must be between d - max dist and d + max dist from that node, so every other subtree is pruned. In practice a tree over a few hundred thousand 64-bit hashes answers a max dist = 6 query by touching a low-single-digit percentage of nodes. The two-threshold design is what keeps precision high. A single frame within 6 bits is a candidate , not a verdict. The verdict comes from temporal overlap : we only declare two videos duplicates when, say, 70% of the shorter video's frames find a near-match in the longer one. A shared intro card matches on two or three frames and gets correctly rejected; a genuine re-upload matches on nearly all of them. Our origin runs LiteSpeed with PHP 8.4 over SQLite in WAL mode, and I am not going to bolt a vector database onto that for a problem SQLite handles fine. The trick for making Hamming-distance lookups indexable in a plain relational store is banded LSH . Split the 64-bit hash into four 16-bit bands. By the pigeonhole principle, if two hashes differ by at most 3 bits total, at least one of the four bands must be bit-identical. So we index each band, query by exact band equality to get a candidate set, and only then compute exact Hamming distance in application code. This turns an un-indexable "within k bits" query into four indexed equality lookups. One SQLite caveat worth stating plainly: its INTEGER type is a signed 64-bit value, and a pHash with the top bit set exceeds PHP INT MAX as an unsigned number. On a 64-bit PHP build the bit pattern round-trips correctly because PHP ints are also 64-bit signed — the same bits, just interpreted as negative. We compare bit patterns, never magnitudes, so this is safe as long as you never do arithmetic ordering on the raw hash.