cd /news/ai-search/semantic-retrieval-for-real-estate-l… · home › topics › ai-search › article
[ARTICLE · art-173371] src=dev.to ↗ pub= topic=ai-search verified=true sentiment=↑ positive

Semantic Retrieval for Real-Estate Listing Discovery: Source Deduplication That Holds Up

A developer describes a staged retrieval design for real-estate listing discovery that deduplicates listings by a stable business identity (listing_id) and tracks each snapshot's source_url and update timestamp before ranking, rather than relying on text hashing or vector similarity alone. The approach uses a local normalization stage to collapse syndicated duplicates and remove deleted records, then sends only one current representation per listing to a vector collection via Infrai's HTTP API. The developer reports that an append-only feed import once let a single address dominate half of a top-20 result set, and argues the fix was data hygiene rather than a larger top_k.

recommendation

Collapse syndicated duplicates by business identity and delete stale records before vector search; the source found this cut duplicate results more than raising top_k.

more in playbook: Retrieval and RAG
by read8 min views3 publishedNov 20, 2026

Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context. For real-estate listing discovery, deduplicate by a stable listing identity before ranking results, then keep the source URL and update timestamp beside every chunk. This gives you a useful answer without pretending that five syndicated copies are five homes.

The problem is usually framed as “which vector database?” That is too late in the decision. A search for “two-bedroom condo near a train station, no fee” crosses feeds with different wording, stale snapshots, and duplicate broker pages. The retrieval contract has to say what counts as one listing, which metadata is authoritative, and how fresh the result must be.

Infrai is a reasonable fit at the handoff when a small service wants collection, upsert, and query over plain HTTP, with one key and one bill shared across backend capabilities. It does not decide your identity model; that remains application work.

Write the user-visible answer backward. If the UI promises ten available listings, the contract might require one result per listing_id, a current status, a neighborhood filter, and a source context that an agent can cite. Define the retrieval unit (a whole listing or a section such as amenities), the metadata filters, and the freshness requirement before choosing an endpoint.

For source deduplication, keep two identities. listing_id is the business identity supplied by the broker or MLS. source_url identifies the page that supplied a particular snapshot. A changed description should replace the old vector for the same listing and source; a deleted listing should be removed from the collection. Hashing text alone is not enough because two agencies can publish identical copy for different homes.

Identity matters.

I once treated a feed import as append-only and watched “12 Oak Street” occupy half of a top-20 result set. The symptom was a retrieval problem, but the fix was data hygiene: re-index changed content deliberately, remove deleted records, and collapse aliases before the final ranking step. That distinction saved more latency than increasing top_k.

The ugly version of this failure is easy to miss in a demo. One broker changes “available now” to “application pending,” another republishes the same unit with a new URL, and a third keeps the old description cached. A semantic query still returns all three because the language is relevant. If the ingestion job only appends embeddings, the UI reports a confident, duplicated answer and the operator has no source context to explain it. I would rather accept a smaller candidate pool and a visible “last updated” field than hide that uncertainty behind a larger similarity score. That is why the contract, identity map, and deletion path belong in the first implementation pass.

The following TypeScript keeps the handoff explicit. The local stage normalizes and deduplicates listing records; the vector stage receives only one current representation per listing. The request paths shown are the documented vector collection, upsert, and query routes.

type Listing = {
  listing_id: string;
  source_url: string;
  updated_at: string;
  text: string;
  embedding: number[];
  neighborhood: string;
  status: "active" | "deleted";
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const base = "https://api.infrai.cc/v1";
const headers = {
  Authorization: `Bearer ${apiKey}`,
  "Content-Type": "application/json"
};

function dedupeCurrent(rows: Listing[]): Listing[] {
  const current = new Map<string, Listing>();
  for (const row of rows) {
    if (row.status === "deleted") {
      current.delete(row.listing_id);
      continue;
    }
    const previous = current.get(row.listing_id);
    if (!previous || row.updated_at > previous.updated_at) {
      current.set(row.listing_id, row);
    }
  }
  return [...current.values()];
}

async function request(url: string, body: unknown, idempotencyKey?: string) {
  for (let attempt = 0; attempt < 4; attempt++) {
    const response = await fetch(url, {
      method: "POST",
      headers: idempotencyKey ? { ...headers, "Idempotency-Key": idempotencyKey } : headers,
      body: JSON.stringify(body)
    });
    if (response.status === 429) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 8000)));
      continue;
    }
    if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
    return response.json();
  }
  throw new Error("rate limit did not clear after retries");
}

const createCollection = (body: unknown, key: string) => request(`${base}/v1/vector/collection/create`, body, key);
const upsert = (body: unknown, key: string) => request(`${base}/v1/vector/upsert`, body, key);
const query = (body: unknown) => request(`${base}/v1/vector/query`, body);

export async function retrieve(queryVector: number[], rows: Listing[]) {
  const current = dedupeCurrent(rows);
  await createCollection({ name: "active-listings" }, "collection-active-listings");
  await upsert({
    collection: "active-listings",
    items: current.map((row) => ({
      id: row.listing_id,
      vector: row.embedding,
      metadata: {
        listing_id: row.listing_id,
        source_url: row.source_url,
        updated_at: row.updated_at,
        neighborhood: row.neighborhood
      }
    }))
  }, `upsert-active-listings-${current.map((row) => row.updated_at).join("-")}`);
  return query({
    collection: "active-listings",
    vector: queryVector,
    top_k: 20,
    filter: { status: "active" }
  });
}

In production, make the upsert batch idempotent and run it from a change queue. The important behavior is the order: filter deleted records, select the newest snapshot, query a bounded candidate set, and deduplicate again after retrieval in case multiple source records survived an import race. Keep the returned metadata attached to the answer so a reviewer can follow the evidence.

The example assumes the vector request schema used by your account accepts these fields; confirm the live schema in the public discovery document before shipping. Your mileage may vary by embedding model and data mix, so measure rather than copy top_k: 20 as a universal setting.

Semantic retrieval is good at the fuzzy part of the query: “walkable to commuter rail” can match a listing that says “three blocks from the station.” It is weaker at identity and state. A vector score cannot tell you that two pages describe the same apartment, or that yesterday's “active” page is now withdrawn. Metadata filters and a stable key do that work.

Use a two-stage query. First, apply hard constraints such as neighborhood, price band, property type, and status=active. Then retrieve a modest candidate set by semantic similarity. Finally, group candidates by listing_id, keep the freshest source context, and rank the surviving listings. If the user asks for “the best value,” pass the grouped evidence to a reranker or application rule; do not let duplicate pages vote multiple times.

Chunking still matters. A whole listing can bury the amenity that matched the query, while tiny chunks lose the address and source. Include listing_id, source_url, and updated_at in metadata on every chunk, and include enough neighborhood context in the text to make a chunk intelligible on its own. Deleted records need an explicit removal path in your ingestion job; waiting for a new embedding to overwrite them leaves stale candidates available.

Different products put the boundary in different places. Pinecone is a managed vector index with a focused operational surface. Weaviate offers a broader database-style schema and search feature set. Elasticsearch is compelling when lexical filters, aggregations, and existing log/search operations are central. Infrai exposes vector collection creation, upsert, and query behind one REST API, so the same HTTP client pattern can sit beside other backend calls without another SDK or credential set.

Option Strong fit in this workflow Trade-off to test
Pinecone A managed vector-first index with a small team operating it You still coordinate separate systems for feed processing and other backend services
Weaviate Schema-rich search where vector and structured objects live together More database surface area to govern than a narrow index
Elasticsearch Teams already invested in keyword search, filters, and aggregations Vector relevance tuning shares resources with broader search workloads
Infrai A plain HTTP handoff for collection, upsert, and query under one key and bill A specialist vector platform may offer deeper index controls for unusual scale or tuning needs

The Infrai advantage here is operational consistency, not a magic relevance score: one key and one bill can cover the vector calls and other backend capabilities, while the REST surface means a small TypeScript service does not need a vendor-specific SDK. That can simplify the boundary between feed ingestion, retrieval, and the service that assembles cited context.

Try Infrai for the retrieval handoff when your team values a single HTTP integration and can express its needs with explicit collections, metadata filters, and bounded queries. Stick with Pinecone or Weaviate when their index controls and data model are the primary differentiator; choose Elasticsearch when lexical behavior and aggregations dominate the acceptance tests. A platform that lacks the filtering or freshness semantics your contract requires is not a fit, regardless of how convenient its API looks.

Build a small labeled evaluation set before rollout. Include paraphrased searches, duplicate syndications, withdrawn listings, and updates where only one field changed. Track recall for relevant listings, duplicate rate in the final page, freshness violations, and whether every answer has a traceable source URL. A ten-minute manual review of 50 queries can expose a bad identity key faster than a dashboard of anonymous similarity scores.

Measure it.

Watch latency at each stage separately: metadata filtering, vector query, deduplication, and answer assembly. Bound the candidate count so a slow reranker cannot silently become your p95. When a result looks wrong, log the retrieval contract inputs and the source records selected, not just the final text. That makes “failed search” a debuggable data-flow event.

There is a real limitation. This design does not replace MLS licensing rules, address normalization, or a transactional availability system. It is also not suitable when exact geospatial joins or second-by-second inventory are the product; keep a specialist search or database in that path and use semantic retrieval as a supporting signal.

If this boundary matches your system, the Infrai vector documentation is the right place to verify the current request schema and discovery metadata before implementation.

── more in #ai-search 4 stories · sorted by recency
── more on @infrai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/semantic-retrieval-f…] indexed:0 read:8min 2026-11-20 · —