5 Source Deduplication Fixes for Semantic Listing Retrieval in Node.js (and What I Kept)

A provider directory that ingests clinic listings from four feeds will hand you the same clinic four times. Use source deduplication as a pre-embedding step instead of a post-retrieval filter: collapse duplicate listing records into one canonical entity, embed that one, and keep every contributing source id on its metadata so a bad semantic retrieval result can be traced back to the feed that produced it. Everything below is a variation on that single decision, including the two fixes I kept and the ones I would drop first.

Index cost is why the ordering matters.

Say you aggregate a healthtech provider directory from a state license registry, two payer network APIs, and a nightly crawl of hospital sites. Four sources, one real-world clinic, four rows that disagree about the suite number and whether the practice name carries "PC" at the end. Embed the rows as they arrive and you pay for four vectors, four rows of storage, and — the part that actually hurts — four slots of a ten-result query budget, three of which say the same thing. Deduplicate first and the index tracks the number of real clinics, which grows far more slowly than the number of feeds you sign.

Most retrieval walkthroughs are explained with one clean corpus, which is the case that never happens in aggregation. The same shape shows up in real-estate listing discovery, where a single unit gets syndicated to three portals with three photo sets and three descriptions. Different industry. Identical failure.

Fuzzy matching is where everyone starts, and it's the wrong first move. Registry feeds in healthcare carry an NPI number; that is a national identifier, and two rows sharing one are the same practitioner by definition. Normalized phone and normalized street line cover most of the rest. Hash those into an entity id, group by it, and a large share of your duplicates disappear for the price of a string comparison.

What's left after exact keys is the interesting part — the crawl rows with no identifier at all. Those need similarity matching, but only inside a block: same postal code, same specialty. Blocking keeps the comparison count linear-ish instead of quadratic, and it also keeps you from merging two genuinely different practices that happen to share a building. I've seen a naive address match fold an entire medical office park into one entity, which is a worse failure than a duplicate, because a duplicate is visible and a bad merge is not.

It changes what a record is. Before you touch an endpoint, write down the retrieval contract: the unit is one real-world clinic, not one feed row; the filters are specialty, accepted plan, and postal region; freshness means a delisted practice leaves the collection within a day.

That contract is also the troubleshooting tool. When a search fails, map the user-visible answer back to the contract and check which clause broke. The clinic never got indexed, so the ingest filter is wrong. The clinic is indexed but ranks below three copies of itself, so deduplication is wrong. The clinic is indexed, ranks first, and closed in March, so the delete path is wrong. Three very different bugs, and without the contract they all present identically as "search is broken."

Text you embed should describe the entity, not the feed. I concatenate canonical name, specialty, and a plain-language service line, then leave the payer list and the license status in metadata where a filter can reach them. Vector similarity is bad at "accepts this plan" and a WHERE clause is excellent at it.

The entity id is doing double duty here. It's a merge key at collapse time and an upsert id at index time, which means a re-run of the pipeline overwrites rather than appends — and a crashed job at 3am costs you a retry instead of a duplicated collection.

// index-providers.ts — collapse listings by entity, then upsert one vector per clinic.
import { createHash } from "node:crypto";

// Base URL and key both come from the environment — never hardcode a credential.
const BASE = process.env.INFRAI_API_BASE;
const KEY = process.env.INFRAI_API_KEY;
if (!BASE || !KEY) throw new Error("INFRAI_API_BASE and INFRAI_API_KEY must be set");

type Listing = {
  source: string; npi: string | null; name: string;
  street: string; zip: string; specialty: string; plans: string[];
};

const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9]+/g, " ").trim();

function entityId(l: Listing): string {
  const key = l.npi ? `npi:${l.npi}` : `addr:${norm(l.name)}|${norm(l.street)}|${l.zip}`;
  return createHash("sha1").update(key).digest("hex").slice(0, 24);
}

function collapse(rows: Listing[]): Map<string, Listing[]> {
  const groups = new Map<string, Listing[]>();
  for (const row of rows) {
    const id = entityId(row);
    groups.set(id, [...(groups.get(id) ?? []), row]);
  }
  return groups;
}

async function upsert(items: unknown[], idempotencyKey: string): Promise<void> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${BASE}/v1/vector/upsert`, {
      method: "POST",
      headers: {
        authorization: `Bearer ${KEY}`,
        "content-type": "application/json",
        "idempotency-key": idempotencyKey,
      },
      body: JSON.stringify({ collection: "providers", items }),
    });
    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after") ?? 0) * 1000;
      await new Promise((r) => setTimeout(r, retryAfter || 400 * 2 ** attempt));
      continue;
    }
    const text = await res.text();
    if (!res.ok) throw new Error(`vector/upsert -> ${res.status} ${text.slice(0, 200)}`);
    return;
  }
  throw new Error("vector/upsert -> still rate limited after 5 attempts");
}

export async function indexBatch(rows: Listing[]): Promise<number> {
  const items = [...collapse(rows)].map(([id, group]) => {
    // Longest name wins; registry rows abbreviate, crawl rows spell it out.
    const canonical = group.slice().sort((a, b) => b.name.length - a.name.length)[0];
    return {
      id,
      text: `${canonical.name}. ${canonical.specialty} practice in ${canonical.zip}.`,
      metadata: {
        specialty: canonical.specialty,
        zip: canonical.zip,
        plans: [...new Set(group.flatMap((g) => g.plans))],
        sources: group.map((g) => g.source),
        merged_from: group.length,
      },
    };
  });

  const batchKey = createHash("sha1").update(items.map((i) => i.id).join(":")).digest("hex");
  await upsert(items, `providers:${batchKey}`);
  return items.length;
}

Note what the metadata carries: sources and merged_from. When somebody reports a wrong result, that array tells you which of the four feeds contributed the bad name, and you fix the ingest rule rather than patching the row by hand.

The reason a hosted vector API was worth considering here at all is that contract stability. I can swap the vendor behind the embeddings on Infrai and the request I already wrote keeps working, which matters when a one-person team's real constraint is that nobody has an afternoon to rewrite an ingest path. Retrieval quality drifts. Rewrites are how a week disappears.

Listings churn in a way documentation does not. A practice closes, a physician moves, a payer drops a network — and a stale vector for a clinic that no longer exists is a worse answer than no answer, because it looks authoritative. Track a content hash per entity, re-embed only when the hash moves, and issue a delete when an entity falls out of every source for two consecutive syncs.

Two syncs, not one. Feeds go quiet for a day.

Before any of this reaches production, build a labeled evaluation set: 30–50 real queries paired with the entity that should come back first. It doesn't need to be sophisticated. Run it after every change to chunking, blocking rules, or freshness policy, and you'll catch the regression where a stricter merge rule quietly collapsed two adjacent clinics into one.

At tens of thousands of entities the deduplication logic matters more than the index. At millions, that inverts: you start caring about quantization, hybrid keyword scoring, and who pays for the node at 3am.

The honest catch: none of these deduplicate for you. Every row in that table assumes an upstream merge step, and the merge step is the part that takes real engineering. Managed APIs are the wrong choice when index-level tuning is your bottleneck — if you need custom HNSW parameters or hybrid scoring you control, stick with Qdrant or Elasticsearch and accept the operations. pgvector is the pick when the listings already live in Postgres, because a join beats a network call every time and you can dedupe in the same transaction that ingests.

I'm not sure the entity-id approach survives a fifth feed without a proper record-linkage library behind it. It has held for four. Ask me again after the next integration.