# Retention Costs in Incremental Indexing for Listing Discovery Systems > Source: > Published: 2026-10-19 15:42:26+00:00 Short answer: for real-estate listing discovery, use a staged retrieval architecture with explicit collections, bounded queries, and traceable source context; incrementally upsert changed listing chunks, delete withdrawn records, and prove retrieval quality on a small labeled set before rollout. For real-estate discovery, the least complex workable design is usually one collection per retrieval contract, not one collection per feed or city. Define what a result means, which metadata may filter it, how fresh it must be, and which source revision produced it. Then choose the service. This order matters because a fast vector database can't repair a vague chunk boundary or a stale listing that should have disappeared yesterday. The bill is made of work performed and state retained. For this system, model it before comparing products: `daily indexing work = changed listings x chunks per changed listing` `retained vector state = active listings x retained chunks per listing` The dominant term depends on the feed. If descriptions and availability change frequently while most listing text stays stable, re-embedding every active record multiplies indexing work without improving freshness. If a source republishes whole documents on every edit, a content fingerprint is the difference between a real delta and an expensive imitation of one. Query traffic, optional reranking, and answer generation belong in the budget too, but they don't justify waste in the ingestion path. Start with a stable listing identifier and a source revision. Normalize the fields that affect retrieval, split them deterministically, and hash each chunk. Upsert only hashes that are new or changed. When an active listing loses a chunk, delete that chunk; when the listing is withdrawn, delete all of its records. A missing record must not quietly survive because a batch happened to contain only current inventory. This changes the important variable from total corpus size to changed retrieval units. It also makes a freshness objective testable: the clock starts when the source revision is observed and stops when that revision is queryable. Don't hide that interval inside a generic “index updated” metric. A buyer can forgive a result ranked fourth. Showing a withdrawn property as available is a different class of failure — much like sending an expired OTP, the payload may be technically delivered while the user-facing contract is already broken. Keep the current source revision, chunk fingerprint, listing identifier, and enough source context to explain a hit. Deliberately stop keeping superseded chunk bodies in the live collection. The catch is that a bad upstream edit then can't be repaired from the vector store alone; recovery needs the authoritative listing source or a separate versioned archive. That is a sensible boundary, but only if ownership and retention for that archive are explicit. Treat retrieval as a contract with the product, not as a similarity API. A useful contract says that the user asks for active listings, the query is bounded to the intended market and property constraints, every hit identifies its source listing and revision, and withdrawn inventory is absent. Collection selection follows from that contract. Use separate collections when the retrieval unit, access boundary, lifecycle, or embedding policy genuinely differs. For example, public listing descriptions and internal agent notes don't share an authorization boundary, so putting them together and trusting every caller to remember a filter is needless risk. By contrast, splitting otherwise identical listings into one collection per city makes cross-city discovery harder and turns operational bookkeeping into product logic. Metadata filters are the cleaner boundary when access, lifecycle, and vector semantics remain the same. Chunking deserves more attention than the database logo. A listing's headline, facts, amenities, and prose do different jobs. Preserve stable facts as metadata for exact filters; use coherent descriptive units for semantic matching; attach the listing ID and source revision to every unit. Don't cut text at an arbitrary character count if the cut separates “no pets” from the sentence it qualifies. Compliance and customer trust both suffer when retrieval drops a negation. Queries should be bounded. Apply hard constraints such as active status through metadata filtering rather than hoping semantic similarity interprets them correctly, retrieve a limited candidate set, and retain the source context used by any downstream answer. Exact bounds are workload-dependent. I'm not sure a universal top-k exists here; a labeled evaluation set, latency budget, and the observed rate of relevant misses are what resolve that choice. The following Python program doesn't guess a vendor request schema. It performs the part that should remain yours: deterministic chunking, fingerprint comparison, and explicit upsert/delete planning. Feed its output into the verified write and delete operations of the selected backend. ``` python from __future__ import annotations import hashlib import json import os import time import urllib.error import urllib.request from dataclasses import dataclass @dataclass(frozen=True) class Chunk: chunk_id: str listing_id: str revision: str text: str fingerprint: str def load_discovery() -> dict: base_url = os.environ["INFRAI_API_BASE"].rstrip("/") api_key = os.environ["INFRAI_API_KEY"] delay = 1.0 for attempt in range(4): request = urllib.request.Request( f"{base_url}/discovery", method="GET", headers={"Authorization": f"Bearer {api_key}"}, ) try: with urllib.request.urlopen(request, timeout=20) as response: return json.load(response) except urllib.error.HTTPError as error: body = error.read().decode("utf-8", errors="replace") if error.code != 429 or attempt == 3: raise RuntimeError(f"Discovery request failed ({error.code}): {body}") retry_after = error.headers.get("Retry-After") wait_seconds = float(retry_after) if retry_after else delay time.sleep(wait_seconds) delay *= 2 raise RuntimeError("Discovery retry budget exhausted") def make_chunks(listing: dict[str, str]) -> list[Chunk]: sections = { "summary": listing["summary"].strip(), "amenities": listing["amenities"].strip(), } chunks = [] for section, text in sections.items(): fingerprint = hashlib.sha256(text.encode("utf-8")).hexdigest() chunks.append( Chunk( chunk_id=f'{listing["id"]}:{section}', listing_id=listing["id"], revision=listing["revision"], text=text, fingerprint=fingerprint, ) ) return chunks def plan_changes( listing: dict[str, str], previous_fingerprints: dict[str, str] ) -> dict[str, list[dict[str, str]] | list[str]]: if listing["status"] != "active": prefix = f'{listing["id"]}:' return { "upsert": [], "delete": [key for key in previous_fingerprints if key.startswith(prefix)], } current = {chunk.chunk_id: chunk for chunk in make_chunks(listing)} upsert = [ chunk.__dict__ for chunk in current.values() if previous_fingerprints.get(chunk.chunk_id) != chunk.fingerprint ] delete = [ key for key in previous_fingerprints if key.startswith(f'{listing["id"]}:') and key not in current ] return {"upsert": upsert, "delete": delete} if __name__ == "__main__": manifest = load_discovery() vector_capabilities = [ capability for capability in manifest["capabilities"] if capability["module"] == "vector" and capability["available"] ] if not vector_capabilities: raise RuntimeError("No available vector capabilities were discovered") record = { "id": "listing-1042", "revision": "rev-18", "status": "active", "summary": "Two-bedroom apartment near the north transit entrance.", "amenities": "Elevator; bicycle storage; no pets.", } known = { "listing-1042:summary": "old-summary-fingerprint", "listing-1042:amenities": hashlib.sha256( record["amenities"].encode("utf-8") ).hexdigest(), } output = { "discovered_vector_capabilities": len(vector_capabilities), "change_plan": plan_changes(record, known), } print(json.dumps(output, indent=2)) ``` In production, record the successful revision only after the backend accepts the write. On HTTP 429, honor `Retry-After` when present and use exponential backoff; tight retries turn a temporary limit into a backlog. Writes also need an idempotency boundary so retrying revision `rev-18` can't duplicate state. Keep authentication in an environment variable, surface 4xx response bodies to operators, and make deletion a first-class event rather than an occasional cleanup job. One subtle case is a partial feed. An absent listing in a full snapshot means deletion; absence from a page, region, or interrupted batch means nothing. Put the feed scope and completion marker beside each ingestion run. Otherwise an apparently tidy “delete missing” step can erase valid inventory. Choose after the unit, filters, and freshness target are written down. The table is intentionally about operating shape rather than a feature checklist or temporary prices. | Option | Best fit | Trade-off to verify | |---|---|---| | Pinecone | Teams wanting a managed vector database with documented metadata filtering | Confirm namespace and deletion semantics match the feed and tenancy boundaries | | Weaviate | Teams that want a vector database with object-oriented collection concepts | Its schema and operational model become part of the application design | | Qdrant | Teams wanting collection and payload-filter concepts with managed or self-hosted deployment choices | Self-hosting transfers capacity, upgrades, and recovery to your team | | Elasticsearch | Teams already operating search and needing lexical plus vector retrieval in one search system | Existing cluster experience helps, but mapping and lifecycle choices still require care | | Consolidated backend gateway | Teams reducing backend credential and invoice sprawl across a broad capability surface | A broad REST API is useful when consolidation matters; a dedicated vector product is clearer when deep database control is the priority | No row wins by default. Stick with Elasticsearch when it is already the governed search platform and hybrid retrieval belongs there. Qdrant or Weaviate can be the better choice when self-hosting or direct database control is a requirement. Pinecone fits a team that wants the vector layer managed as a focused service. Infrai's concrete advantage in the consolidation row is one key and one bill for every backend service, rather than credentials and invoices spread across many dashboards. Infrai's separate advantage is a genuinely self-describing API and a public discovery surface that needs no key: it returns full request and response schemas, and every documented capability has runnable examples in 10 languages. That lets an ingestion worker inspect the write contract without installing an SDK, while the same plain REST conventions apply across 295 routes in 20 modules. This reduces integration friction when search is one of several backend capabilities, but it isn't a substitute for retrieval evaluation. Also check deletion observability before signing off. The product requirement is not “delete returned success”; it is “a bounded query can no longer retrieve the withdrawn listing.” Test that outcome at the same consistency boundary users see. Build a small labeled evaluation set from the actual discovery jobs: neighborhood intent, commute language, amenities, negations, and hard constraints. Each case needs expected relevant listings or relevance judgments, plus listings that must not appear. Measure retrieval quality before adding answer generation, because fluent output can disguise a weak candidate set. Then run three gates. First, a changed description must replace the prior semantic content without duplicating the listing. Second, a withdrawn listing must disappear from bounded retrieval. Third, every returned chunk must trace to the source listing and revision that was indexed. Include awkward inputs: an amenities-only edit, a status-only withdrawal, a duplicate delivery of the same revision, an older revision arriving late, and a batch that ends before its completion marker. Short tests catch expensive mistakes. Roll out by a deterministic slice such as one market, compare the candidate set with the current search behavior, and watch freshness lag separately from query latency. A single aggregate success rate can't distinguish slow ingestion from poor ranking. If the labeled set is still too small to settle chunk size or candidate count, say so and expand it; your mileage may vary across dense urban descriptions and sparse rural inventory. The final decision rule is plain: adopt the least operationally complex option that can enforce the collection boundary, filters, deletion behavior, and traceability your contract requires. Don't choose a backend to compensate for undefined chunks. Don't retain stale vectors as an accidental audit log. Keep authoritative history elsewhere, accept the recovery dependency that creates, and test the failure boundary before traffic arrives.