Real-Estate Discovery with Meaning-Based Retrieval and Canonical Source Records

A developer built a real-estate discovery knowledge bot that separates semantic retrieval from duplicate handling, using a deterministic source-canonicalization pass as the authority for merging listings. The design keys records on a stable source identity (feed name plus publisher listing ID) with a content identity as a monitored fallback, appends observations rather than overwriting them, and applies freshness filters only after canonicalization. The developer's rule: "identity before similarity.

Short answer: keep semantic retrieval responsible for relevance, then make a deterministic source-canonicalization pass the authority for duplicate listings. In a property-management knowledge bot, freshness wins over a slightly higher similarity score: an old lease policy attached to a current unit is a correctness bug, not a ranking quirk. I use this as the decision rule: retrieve broadly enough to avoid missing a listing, canonicalize by stable source identity, and only then apply freshness and business filters. The RAG paper describes retrieval-augmented generation as a generator backed by an external memory Lewis et al., 2020 . That memory is useful only when two copies of the same apartment do not masquerade as two independent facts. A listing record needs two identities. The first is a source identity, such as a feed name plus the publisher's listing ID. The second is a content identity, calculated from normalized address, unit, and salient attributes. Source identity is authoritative; content identity is a carefully monitored fallback because two neighboring units can look nearly identical. The ingestion contract should preserve source id , source listing id , observed at , published at , status , and the raw source URL. Do not overwrite an observation in place. Append the observation, point the canonical record at the newest accepted version, and retain the previous version for audit and rollback. This boundary matters during a feed replay: a partner can resend yesterday's export after today's correction, and if that replay replaces the canonical row merely because it arrived later, a stale rent or old availability flag becomes the answer. Arrival time is an operational fact; it is not listing freshness. Keeping both observations also gives an operator enough context to explain a correction without reconstructing an entire feed run from logs. I once diagnosed a “duplicate” alert that was actually a unit-number parsing error: 4B and 4-B landed in different keys, while two genuinely different apartments shared a geocoder's rounded coordinate. The fix was not a more aggressive embedding threshold. It was explicit normalization plus a manual-review state for ambiguous matches. Three words: identity before similarity. Treat retrieval as a staged pipeline. Sparse filters remove impossible records first: market, property type, bedroom count, and availability policy. Dense retrieval then handles meaning, so “quiet two-bedroom near transit” can find descriptions that do not share those exact words. The final stage joins every hit to its canonical record and applies freshness rules. A useful query result carries evidence, not just a score: canonical ID, matched chunk IDs, source IDs, source timestamps, and the reason a duplicate was merged or rejected. That payload lets a support engineer answer “why did this listing appear?” without replaying the entire index build. Here is the critical path in Python. The interfaces are deliberately generic so the policy can be tested independently of a vector database. python from dataclasses import dataclass from datetime import datetime, timezone import re import unicodedata @dataclass frozen=True class Hit: canonical id: str source id: str source listing id: str score: float observed at: datetime def normalize text value: str - str: value = unicodedata.normalize "NFKC", value .casefold value = re.sub r" ^a-z0-9 +", " ", value return " ".join value.split def choose current hits: list Hit , now: datetime - list Hit : by canonical: dict str, Hit = {} for hit in hits: age days = now - hit.observed at .total seconds / 86400 if age days 14: continue current = by canonical.get hit.canonical id if current is None or hit.observed at, hit.score current.observed at, current.score : by canonical hit.canonical id = hit return sorted by canonical.values , key=lambda item: item.score, reverse=True def retrieve listing query: str, sparse search, semantic search, now: datetime | None = None - list Hit : now = now or datetime.now timezone.utc candidates = sparse search query, limit=80 candidates += semantic search query, limit=80 unique pairs = { item.source id, item.source listing id : item for item in candidates} return choose current list unique pairs.values , now :20 The 14-day window above is a policy example, not a universal truth. Set it from the property's leasing cadence and publish a metric for excluded records. A hard-coded window with no metric is how silent data loss starts. Use a precedence ladder. An exact source id, source listing id match is a merge. If that key is absent, compare normalized address, unit label, and a bounded set of attributes such as beds and baths. If those agree, issue a candidate link rather than an automatic merge when the source has a history of reused IDs. Keep the original fields alongside normalized fields; agents and auditors need both. Do not hash the entire description. Marketing copy changes weekly, and a typo should not create a new apartment. Conversely, do not rely on address alone: a building can have hundreds of units and a property manager may syndicate the same unit under multiple feed IDs. Images deserve special handling. Perceptual similarity can support review, but it should not be the primary identity key. Staged construction photos and a renovated unit can share a photo set while representing different availability windows. | Option | Strength | Failure mode | Appropriate use | |---|---|---|---| | Source ID pair | Deterministic and auditable | Breaks when a publisher rekeys records | Primary merge key | | Normalized address + unit | Works across syndication feeds | Sensitive to missing or malformed unit labels | Candidate matching | | Embedding similarity | Tolerates paraphrase and noisy text | Can collapse neighboring units | Recall aid, never authority | | Human review queue | Handles ambiguous edge cases | Adds latency and staffing cost | Low-confidence matches | The rejected option is “deduplicate by top embedding score.” It looks elegant in a demo and fails under near-identical floor plans. Keep it for recall diagnostics or review suggestions. It is not suitable when a tenant could be shown the wrong rent, availability date, or application link. Record three clocks: when the publisher says the listing changed, when your system observed it, and when the index made it searchable. A freshness badge should be derived from those clocks and the source's service-level expectation. If published at is missing, say “observed recently” rather than inventing a publication date. Deletion needs a first-class event. A feed omission is ambiguous: it might mean leased, temporarily paused, or a failed export. Require an explicit inactive status or a configurable absence threshold before removing a canonical record from answers. During that interval, suppress it from availability queries but keep it for audit. For a property-management bot, chunk boundaries should follow facts that expire together. Put rent, availability, pet policy, and application URL in a compact listing chunk; put building rules and maintenance procedures in separate chunks. When rent changes, re-embed the listing chunk only. This reduces stale mixtures where a current address is retrieved beside an old price. Your mileage may vary. Some teams can tolerate a few hours of lag; others need minute-level updates during leasing season. Measure answer age at the point of generation, not only during ingestion. Log deduplication decisions with reason codes such as exact source key , normalized candidate , ambiguous review , and stale excluded . Track duplicate rate by source, canonical-record churn, freshness lag percentiles, and the share of answers citing records past policy. These are leading indicators; a rising duplicate rate usually appears before users complain. Build a replayable fixture set from redacted listings. Include unit labels with punctuation variants, reused publisher IDs, address abbreviations, missing timestamps, and two adjacent units with identical descriptions. A change to normalization should run against that fixture before deployment. Keep a “why this result” trace available to operators and a compact citation in the bot response. When a user asks about a two-bedroom, the bot should expose the canonical listing and its observed time, then state when information is uncertain. That is more useful than pretending a cosine score is a confidence percentage. The trade-off is real: strict canonicalization can lower recall, while permissive merging can damage trust. Choose strictness based on the cost of a false merge. For leasing decisions, stick with a review queue and conservative filters; for exploratory neighborhood research, a looser candidate set is acceptable. A final check I use before shipping: can an engineer replay one answer, identify every source record behind it, and explain why older copies were excluded? If the answer is no, the retrieval system is not ready for production, regardless of its benchmark score.