5 Source Deduplication Fixes for Semantic Listing Retrieval in Node.js (and What I Kept)

A developer building a healthtech provider directory that ingests clinic listings from four feeds described collapsing duplicate records into a single canonical entity before embedding, rather than filtering duplicates after retrieval. The approach hashes national identifiers such as NPI numbers along with normalized phone and street data into an entity id, uses postal-code and specialty blocking for similarity matching on records lacking identifiers, and upserts one vector per clinic so pipeline re-runs overwrite instead of append. The developer argued the ordering keeps index cost tied to real clinics rather than feed count and preserves the full query budget for distinct results.

A provider directory that ingests clinic listings from four feeds will hand you the same clinic four times. Use source deduplication as a pre-embedding step instead of a post-retrieval filter: collapse duplicate listing records into one canonical entity, embed that one, and keep every contributing source id on its metadata so a bad semantic retrieval result can be traced back to the feed that produced it. Everything below is a variation on that single decision, including the two fixes I kept and the ones I would drop first. Index cost is why the ordering matters. Say you aggregate a healthtech provider directory from a state license registry, two payer network APIs, and a nightly crawl of hospital sites. Four sources, one real-world clinic, four rows that disagree about the suite number and whether the practice name carries "PC" at the end. Embed the rows as they arrive and you pay for four vectors, four rows of storage, and — the part that actually hurts — four slots of a ten-result query budget, three of which say the same thing. Deduplicate first and the index tracks the number of real clinics, which grows far more slowly than the number of feeds you sign. Most retrieval walkthroughs are explained with one clean corpus, which is the case that never happens in aggregation. The same shape shows up in real-estate listing discovery, where a single unit gets syndicated to three portals with three photo sets and three descriptions. Different industry. Identical failure. Fuzzy matching is where everyone starts, and it's the wrong first move. Registry feeds in healthcare carry an NPI number; that is a national identifier, and two rows sharing one are the same practitioner by definition. Normalized phone and normalized street line cover most of the rest. Hash those into an entity id, group by it, and a large share of your duplicates disappear for the price of a string comparison. What's left after exact keys is the interesting part — the crawl rows with no identifier at all. Those need similarity matching, but only inside a block: same postal code, same specialty. Blocking keeps the comparison count linear-ish instead of quadratic, and it also keeps you from merging two genuinely different practices that happen to share a building. I've seen a naive address match fold an entire medical office park into one entity, which is a worse failure than a duplicate, because a duplicate is visible and a bad merge is not. It changes what a record is . Before you touch an endpoint, write down the retrieval contract: the unit is one real-world clinic, not one feed row; the filters are specialty, accepted plan, and postal region; freshness means a delisted practice leaves the collection within a day. That contract is also the troubleshooting tool. When a search fails, map the user-visible answer back to the contract and check which clause broke. The clinic never got indexed, so the ingest filter is wrong. The clinic is indexed but ranks below three copies of itself, so deduplication is wrong. The clinic is indexed, ranks first, and closed in March, so the delete path is wrong. Three very different bugs, and without the contract they all present identically as "search is broken." Text you embed should describe the entity, not the feed. I concatenate canonical name, specialty, and a plain-language service line, then leave the payer list and the license status in metadata where a filter can reach them. Vector similarity is bad at "accepts this plan" and a WHERE clause is excellent at it. The entity id is doing double duty here. It's a merge key at collapse time and an upsert id at index time, which means a re-run of the pipeline overwrites rather than appends — and a crashed job at 3am costs you a retry instead of a duplicated collection. // index-providers.ts — collapse listings by entity, then upsert one vector per clinic. import { createHash } from "node:crypto"; // Base URL and key both come from the environment — never hardcode a credential. const BASE = process.env.INFRAI API BASE; const KEY = process.env.INFRAI API KEY; if BASE || KEY throw new Error "INFRAI API BASE and INFRAI API KEY must be set" ; type Listing = { source: string; npi: string | null; name: string; street: string; zip: string; specialty: string; plans: string ; }; const norm = s: string = s.toLowerCase .replace / ^a-z0-9 +/g, " " .trim ; function entityId l: Listing : string { const key = l.npi ? npi:${l.npi} : addr:${norm l.name }|${norm l.street }|${l.zip} ; return createHash "sha1" .update key .digest "hex" .slice 0, 24 ; } function collapse rows: Listing : Map<string, Listing { const groups = new Map<string, Listing ; for const row of rows { const id = entityId row ; groups.set id, ... groups.get id ?? , row ; } return groups; } async function upsert items: unknown , idempotencyKey: string : Promise<void { for let attempt = 0; attempt < 5; attempt++ { const res = await fetch ${BASE}/v1/vector/upsert , { method: "POST", headers: { authorization: Bearer ${KEY} , "content-type": "application/json", "idempotency-key": idempotencyKey, }, body: JSON.stringify { collection: "providers", items } , } ; if res.status === 429 { const retryAfter = Number res.headers.get "retry-after" ?? 0 1000; await new Promise r = setTimeout r, retryAfter || 400 2 attempt ; continue; } const text = await res.text ; if res.ok throw new Error vector/upsert - ${res.status} ${text.slice 0, 200 } ; return; } throw new Error "vector/upsert - still rate limited after 5 attempts" ; } export async function indexBatch rows: Listing : Promise<number { const items = ...collapse rows .map id, group = { // Longest name wins; registry rows abbreviate, crawl rows spell it out. const canonical = group.slice .sort a, b = b.name.length - a.name.length 0 ; return { id, text: ${canonical.name}. ${canonical.specialty} practice in ${canonical.zip}. , metadata: { specialty: canonical.specialty, zip: canonical.zip, plans: ...new Set group.flatMap g = g.plans , sources: group.map g = g.source , merged from: group.length, }, }; } ; const batchKey = createHash "sha1" .update items.map i = i.id .join ":" .digest "hex" ; await upsert items, providers:${batchKey} ; return items.length; } Note what the metadata carries: sources and merged from . When somebody reports a wrong result, that array tells you which of the four feeds contributed the bad name, and you fix the ingest rule rather than patching the row by hand. The reason a hosted vector API was worth considering here at all is that contract stability. I can swap the vendor behind the embeddings on Infrai and the request I already wrote keeps working, which matters when a one-person team's real constraint is that nobody has an afternoon to rewrite an ingest path. Retrieval quality drifts. Rewrites are how a week disappears. Listings churn in a way documentation does not. A practice closes, a physician moves, a payer drops a network — and a stale vector for a clinic that no longer exists is a worse answer than no answer, because it looks authoritative. Track a content hash per entity, re-embed only when the hash moves, and issue a delete when an entity falls out of every source for two consecutive syncs. Two syncs, not one. Feeds go quiet for a day. Before any of this reaches production, build a labeled evaluation set: 30–50 real queries paired with the entity that should come back first. It doesn't need to be sophisticated. Run it after every change to chunking, blocking rules, or freshness policy, and you'll catch the regression where a stricter merge rule quietly collapsed two adjacent clinics into one. At tens of thousands of entities the deduplication logic matters more than the index. At millions, that inverts: you start caring about quantization, hybrid keyword scoring, and who pays for the node at 3am. | Option | How you run it | Deduplication work | Where it fits | |---|---|---|---| | pgvector | extension in Postgres you already operate | your own SQL, alongside the source tables | you already own the entity data and a DBA | | Qdrant | self-hosted or cloud, tunable index | payload filters, external merge step | filtered search over large listing sets | | Pinecone | fully managed index | external merge step | you want zero index operations | | Elasticsearch | cluster you size and maintain | dedupe and BM25 in one place | exact keyword recall matters as much as semantics | | Infrai | one REST API over HTTP, no SDK to install | external merge step | small teams who want a collection, not a database | The honest catch: none of these deduplicate for you. Every row in that table assumes an upstream merge step, and the merge step is the part that takes real engineering. Managed APIs are the wrong choice when index-level tuning is your bottleneck — if you need custom HNSW parameters or hybrid scoring you control, stick with Qdrant or Elasticsearch and accept the operations. pgvector is the pick when the listings already live in Postgres, because a join beats a network call every time and you can dedupe in the same transaction that ingests. I'm not sure the entity-id approach survives a fifth feed without a proper record-linkage library behind it. It has held for four. Ask me again after the next integration.