{"slug": "vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw", "title": "Vector search in PostgreSQL: deduplicating speakers with pgvector and HNSW", "summary": "A media analytics platform using PostgreSQL's pgvector extension replaced an O(N²) self-join query with a CROSS JOIN LATERAL approach that leverages HNSW indexes to deduplicate thousands of speakers by voice embeddings, reducing a query that timed out at 5,000 speakers to one that scales efficiently. The fix asks the index for the k nearest neighbors per speaker instead of comparing all pairs, preserving the 0.71 similarity threshold for matching and 0.92 for merging.", "body_md": "# Vector search in PostgreSQL: deduplicating thousands of speakers with pgvector and HNSW\n\nMost pgvector articles are about RAG and LLM chatbots. This one is not. It is about a very concrete, non-LLM problem: identifying *who is speaking* on TV and radio, at scale, with nothing more exotic than PostgreSQL.\n\n## The problem\n\nI work on a media analytics platform that runs speech-to-text with speaker diarization on broadcast streams, 24/7. Each diarized segment produces a voice embedding: a 256-dimension vector that characterizes a voice. Two segments from the same person should produce vectors that are close in cosine similarity; two different people should not.\n\nThe goal is a `speakers`\n\ntable with one row per real-world person. Each speaker accumulates *fingerprints* (one embedding per segment), and carries a *centroid*: the average of its active fingerprint embeddings. When a new segment arrives, we match it against existing centroids; above a similarity threshold (0.71 for us) it is attached to the existing speaker, otherwise a new speaker is created.\n\nThe schema looks like this:\n\n```\nCREATE EXTENSION IF NOT EXISTS vector;\n\nCREATE TABLE speakers (\n    id UUID PRIMARY KEY,\n    name TEXT NOT NULL,\n    embedding VECTOR(256),   -- centroid of active fingerprints\n    status SMALLINT NOT NULL DEFAULT 0\n);\n\nCREATE TABLE speaker_fingerprints (\n    id UUID PRIMARY KEY,\n    speaker_id UUID NOT NULL REFERENCES speakers(id) ON DELETE CASCADE,\n    embedding VECTOR(256),\n    status SMALLINT NOT NULL DEFAULT 0\n);\n\nCREATE INDEX ON speakers USING hnsw (embedding vector_cosine_ops);\nCREATE INDEX ON speaker_fingerprints USING hnsw (embedding vector_cosine_ops);\n```\n\nThreshold-based matching is never perfect. Diarization is noisy, voices change with audio quality, and a borderline segment sometimes creates a brand new speaker for a person who already exists. Over months of continuous ingestion, you accumulate near-duplicate speakers. So we need a periodic deduplication job: find every pair of speakers whose centroids are almost identical, and merge them.\n\n## The naive version: an O(N²) self-join\n\nThe first implementation was the obvious one. Compare every speaker with every other speaker, take the best pair above the merge threshold:\n\n``` js\nSELECT a.id, b.id,\n       1 - (a.embedding <=> b.embedding) AS sim\n  FROM speakers a\n  JOIN speakers b ON a.id < b.id\n WHERE a.status >= 0 AND b.status >= 0\n   AND 1 - (a.embedding <=> b.embedding) >= 0.92\n ORDER BY sim DESC\n LIMIT 1;\n```\n\nThis works fine in the demo, with a hundred speakers. With thousands of speakers it is a disaster, and it is important to understand *why*: **the HNSW index is not used at all**. pgvector's indexes only accelerate one query shape: `ORDER BY embedding <=> $1 LIMIT k`\n\n. A distance expression inside a `WHERE`\n\nclause or a join predicate is evaluated the hard way: a sequential scan computing N×(N−1)/2 cosine distances. At 5,000 speakers that is more than 12 million 256-dimension distance computations for a single query. Ours started hitting the statement timeout, and to make it worse, the job called this query once *per merge*.\n\n## The fix: ask the index for k neighbours per speaker\n\nThe insight is to stop asking \"which pairs are above the threshold?\" (a question the index cannot answer) and instead ask, for each speaker, \"what are your k nearest neighbours?\", which is exactly the query shape HNSW is built for. In SQL this is a `CROSS JOIN LATERAL`\n\n:\n\n```\nWITH raw_pairs AS (\n    SELECT DISTINCT ON (LEAST(a.id, n.id), GREATEST(a.id, n.id))\n           LEAST(a.id, n.id)    AS sid_a,\n           GREATEST(a.id, n.id) AS sid_b,\n           1 - (a.embedding <=> n.embedding) AS similarity\n      FROM speakers a\n      CROSS JOIN LATERAL (\n          SELECT id, embedding\n            FROM speakers\n           WHERE status >= 0\n             AND embedding IS NOT NULL\n             AND id <> a.id\n           ORDER BY embedding <=> a.embedding   -- HNSW kicks in here\n           LIMIT 5                                -- k neighbours per speaker\n      ) n\n     WHERE a.status >= 0 AND a.embedding IS NOT NULL\n       AND 1 - (a.embedding <=> n.embedding) >= 0.92\n)\nSELECT * FROM raw_pairs ORDER BY similarity DESC;\n```\n\nThree details matter here:\n\n- The inner query is\n`ORDER BY … LIMIT k`\n\n, so each of the N outer rows costs one indexed k-NN lookup instead of a full scan. The overall cost goes from O(N²) to roughly O(N·log N). - k-NN returns both directions of every pair (A finds B, then B finds A).\n`DISTINCT ON (LEAST(id1, id2), GREATEST(id1, id2))`\n\ncollapses each unordered pair to a single row. - The threshold filter is still there, but it now filters a small candidate set (N×k rows) instead of driving the join.\n\nA small k (5 in our case) is enough. If a speaker happens to have more than k mergeable duplicates, the surplus is simply not visible in this pass, and that is fine, because merging runs in rounds.\n\n## Merging in rounds, not in one pass\n\nYou cannot just take the candidate list and merge every pair in it. Each merge moves the surviving speaker's centroid (it is recomputed as the average of the combined fingerprints), so a pair that was above the threshold when the list was built may no longer be a real duplicate two merges later. The opposite also happens: a merge can pull a centroid closer to a third speaker.\n\nSo the bulk merge is a loop:\n\n- Fetch all candidate pairs with the indexed query above, sorted by similarity descending.\n- Greedily walk the list and merge,\n*skipping any pair that involves a speaker already touched in this round*. Every merge in a round is therefore between two untouched speakers, computed from fresh centroids. - When the list is exhausted, start a new round: re-fetch candidates with the drifted centroids.\n- Stop when a round performs zero merges (with a hard cap on rounds as a safety net against pathological oscillation).\n\nEach merge keeps the speaker that has the most fingerprints and folds the smaller one into it, then the centroid is recomputed. This is also where the missed \"surplus\" duplicates from the k-NN limit get caught: after round one merged the closest pairs, the survivors find their remaining duplicates in round two.\n\n## Keeping centroids honest\n\nOne last piece makes the whole thing stable. A centroid computed from noisy fingerprints drifts, and a merge can import a few bad segments. After every recomputation, each fingerprint is re-evaluated against the new centroid: active fingerprints that fall below an outlier threshold are demoted, and, the interesting part, previously demoted fingerprints whose similarity is now *above* the threshold are re-promoted, because removing bad samples sharpens the centroid and can bring borderline segments back in. This demote/promote pass loops up to three times until the centroid converges. All of it is plain SQL: `AVG(embedding)`\n\nwith a `FILTER`\n\nclause, and two `UPDATE`\n\nstatements using the `<=>`\n\noperator.\n\n## Takeaways\n\n- pgvector's HNSW index accelerates exactly one query shape:\n`ORDER BY embedding <=> $1 LIMIT k`\n\n. If your distance expression lives in a`WHERE`\n\nclause or a join condition, you are doing a sequential scan, whatever indexes exist. `CROSS JOIN LATERAL`\n\nis the bridge: it turns \"compare everything with everything\" into \"one indexed k-NN lookup per row\".- HNSW is approximate. Combined with a small k, a single pass can miss pairs, design the surrounding process (rounds, periodic re-runs) so that misses are caught later instead of pretending the index is exhaustive.\n- When entities are mutable aggregates (centroids), never batch-apply decisions computed from stale state. Greedy non-overlapping merges per round keep every decision based on fresh data, at the cost of a few extra query rounds, which are cheap now that they are indexed.\n\nNo vector database, no extra infrastructure: PostgreSQL, one extension, and the right query shape.", "url": "https://wpnews.pro/news/vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw", "canonical_source": "https://rvier.fr/posts/deduplicating-speakers-with-pgvector-and-hnsw-EN", "published_at": "2026-08-24 22:04:33+00:00", "updated_at": "2026-08-24 22:12:58.676638+00:00", "lang": "en", "topics": ["machine-learning"], "entities": ["PostgreSQL", "pgvector", "HNSW"], "alternates": {"html": "https://wpnews.pro/news/vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw", "markdown": "https://wpnews.pro/news/vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw.md", "text": "https://wpnews.pro/news/vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw.txt", "jsonld": "https://wpnews.pro/news/vector-search-in-postgresql-deduplicating-speakers-with-pgvector-and-hnsw.jsonld"}}