No Users? No Problem: How to Train Your RAG System Before Your First Query

wpnews.pro

Every engineering team building semantic search or Retrieval-Augmented Generation (RAG) hits the same cold-start dilemma: how do you optimize retrieval when you do not have any user queries?

In classical information retrieval, tuning search is an empirical discipline. You log search terms, analyze clicks, track dwell time, and train cross-encoders on historical impressions to rank relevant documents higher.

When launching a brand-new natural language search system, none of that data exists. You have zero search logs and zero clicks, yet you must select embedding models, tune chunking, and calibrate rerankers before day one.

Teams usually pick between two flawed shortcuts:

The Off-The-Shelf Launch: Deploying a generic embedding model with default vector search. On launch day, the system fails on domain-specific queries because generic vector similarity confuses superficial keywords with user intent.
The LLM Judge Echo Chamber: Prompting an LLM to generate synthetic queries for each document in isolation, then using another LLM to grade relevance. This creates an echo chamber where models reward their own latent biases, generating artificially verbose queries that bear zero resemblance to how real humans search.

Recently, the search engineering team at Airbnb published a breakthrough framework accepted at CIKM 2026 that solves this problem: CASTLE (Contrastive And Seed-guided Training for natural Language sEarch).

By deriving contrastive listing pairs from historical booking sessions, Airbnb created a framework that synthesizes realistic search queries and generates ground-truth relevance labels by construction, slashing query distribution divergence by 9.2x before serving a single live user query.

Here is how CASTLE works, the universal pattern behind synthetic search bootstrapping, and how your team can train production-ready RAG systems before your first user types a word.

1. The Core Paradigm: The Synthetic Query Bootstrap Pattern #

The foundational insight behind Airbnb’s work is an inversion of standard retrieval evaluation:

Do not wait for users to search your documents. Use your documents and historical actions to synthesize your users’ queries.

In a standard search pipeline, user queries arrive first and the engine retrieves documents.

In the Synthetic Query Bootstrap Pattern, you reverse the flow:

The Grounding Anchor: Anchor on existing documents and implicit actions (downloads, purchases, bookings).
Contrastive Conditioning: Force an LLM to write a query distinguishing a preferred document from a viewed alternative.
Labels by Construction: Because the query targets Document A and excludes Document B, the relevance label (1 vs 0) is guaranteed without a post-hoc LLM judge.

Grounding query generation in contrastive pairs creates realistic query text and clean, non-circular training labels simultaneously.

2. The CASTLE Architecture: Training Search Before Launch #

The CASTLE system bridges raw booking telemetry with dense vector retrieval across four coordinated stages:

Stage 1: Contrastive Grounding from De-Identified Sessions

Prior synthetic query frameworks generate queries from a single document, producing superficial questions that repeat the exact keywords in the text.

Airbnb solved this by mining contrastive listing pairs from de-identified booking sessions:

The Positive Listing (L+): The property that the guest reserved.
The Hard Negative Listing (L-): A property that the guest inspected in the exact same search session, but chose not to book.

Because the guest considered both listings, the delta between L+ and L- represents genuine user preference. Airbnb extracts structured attributes alongside review summaries, with personal information redacted.

Stage 2: The 3-Way Prompt Engine

Once a contrastive pair is assembled, an unconstrained LLM will naturally drift into repetitive phrasing, generating queries of identical length and syntax (known as query collapse).

To ensure generated queries mirror real human behavior, CASTLE routes pair features through a 3-way prompt engine:

Template-Based Prompts: Enforces structural diversity across fundamental search intents (e.g. family vacations, pet-friendly travel, remote work retreats).
Few-Shot Seed Prompts: Uses approximately 500 hand-curated search queries from user research as linguistic exemplars, grounding synthetic generation in authentic human phrasing.
Attribute-Grounded Prompts: Forces the generator to focus on the explicit delta between the two listings (such as a fenced yard, private dock, or EV charger).

The generator receives a clear instruction: “Generate a concise search query that is satisfied by Listing L+ but is NOT satisfied by Listing L-.”

Stage 3: Labeling by Construction and the Virtual Judge

The resulting output is a high-quality training triplet: (Query, Positive Listing, Hard Negative Listing).

Because the query was explicitly generated to exploit the delta between L+ and L-, the relevance label is correct by construction. The booked listing receives a relevance score of 1, and the viewed-not-booked listing receives a relevance score of 0. There is zero circularity, zero human annotation cost, and zero hallucinated relevance scoring.

For brand-new listings that have zero booking history, Airbnb supplements the pipeline with a Virtual Judge (VJ). The Virtual Judge uses an LLM evaluator to score relevance for long-tail inventory, ensuring catalog coverage without polluting the core contrastive set.

Stage 4: Ranking Model Fine-Tuning and Pre-Launch Gate

With synthetic triplets generated, Airbnb fine-tunes two complementary ranking tiers:

The Bi-Encoder Dense Retriever: Fine-tuned to maximize cosine similarity between the synthetic query and positive listings while repelling hard negatives, enabling high-recall vector search via ANN indexes.
The Cross-Encoder Reranker: Ingests concatenated query and listing metadata to compute deep interaction scores, precisely calibrating top results.

Before deploying to production, the models pass through an automated Quality Gate. Airbnb evaluates the Kullback-Leibler (KL) divergence between synthetic query distributions and real-world search patterns.

The results were decisive: CASTLE achieved a KL divergence of 1.01, representing a 9.2x improvement over prior baselines like Promptagator (9.33). Furthermore, query length variance matched real human guest search behavior almost perfectly.

3. The 3 Production Scars: What Fails in Practice #

Teams attempting to bootstrap search or RAG systems with synthetic data regularly encounter three severe traps:

Scar #1: The Synthetic Verbosity Trap

When prompted to write a query, LLMs instinctively write complete grammatical sentences. Real humans do not search that way: they type short, telegraphic fragments like “tahoe cabin hot tub dog friendly”. If you fine-tune your retrieval models on verbose synthetic sentences, your model learns to depend on syntactic filler words, collapsing on concise real-world search phrases.

The Production Guardrail: Strictly constrain query generation length. Inject seed exemplars showing 4 to 8 word keyword queries, and penalize synthetic queries that exceed conversational brevity thresholds.

Scar #2: The Self-Fulfilling Evaluation Loop

A fatal mistake in synthetic data generation is using the same LLM family to generate both the training queries and the evaluation benchmarks. If the same model family generates your synthetic queries and evaluates your retrieval accuracy, your benchmark measures how well your retriever mirrors internal model biases, not how well it satisfies human intent.

The Production Guardrail: Separate generation from evaluation. Use human subject matter expert audits on sampled triplets (Airbnb achieved 91% to 93% agreement across three independent raters), or evaluate synthetic retrieval against an external, orthogonal task benchmark.

Scar #3: The Easy Negative Delusion

If you construct training pairs by pairing a positive document with a randomly sampled document from your database, your negative examples will be trivially easy. Your model will achieve 99% offline accuracy, but fail in production because it never learned to distinguish between two highly similar documents in the same neighborhood.

The Production Guardrail: Mine hard negatives within the same local context cluster. Your negative document must share 80% of the positive document’s attributes while differing only on the specific discriminator identified in the query.

4. How to Prototype This Weekend (Synthetic Triplet Generator) #

You can prototype the core of the CASTLE contrastive query generator using Python and Pydantic:

from typing import Dict, Any
from pydantic import BaseModel, Field

class ContrastiveQueryOutput(BaseModel):
    query: str = Field(description="A concise 4-8 word search phrase")
    distinguishing_feature: str = Field(description="The key feature in Doc A missing from Doc B")

def generate_contrastive_query(client, doc_a: Dict[str, Any], doc_b: Dict[str, Any]) -> dict:
    prompt = f"""
    You are an expert at simulating natural human search behavior.
    Document A (Preferred): {doc_a['title']}. Features: {doc_a['attributes']}.
    Document B (Alternative): {doc_b['title']}. Features: {doc_b['attributes']}.

    Task: Write a realistic, concise 4-8 word search query that a human would type 
    if they specifically wanted Document A and would reject Document B.
    Do NOT write full sentences. Mirror natural search shorthand.
    """
    
    response: ContrastiveQueryOutput = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_model=ContrastiveQueryOutput,
    )
    
    return {
        "query": response.query,
        "positive_doc_id": doc_a["id"],
        "negative_doc_id": doc_b["id"],
        "positive_label": 1,
        "negative_label": 0
    }

This harness produces contrastive training triplets that are mathematically labeled by construction, ready for fine-tuning sentence transformers or calibrating your reranker threshold.

5. The Strategic Bottom Line #

For engineering leaders, Airbnb’s CASTLE framework delivers an essential takeaway:

Launching semantic search or RAG without pre-launch evaluation is flying blind: but you do not need historical search logs to build a rigorous benchmark.

Relying on out-of-the-box embedding models leaves your product vulnerable to launch-day search failures. Waiting months to collect user queries delays your development velocity and burns customer trust.

By implementing the Synthetic Query Bootstrap Pattern:

You generate golden evaluation benchmarks before writing frontend code.
You fine-tune embeddings and rerankers on realistic contrastive distributions, achieving sub-1.1 KL divergence against real user behavior.
You transition smoothly from synthetic pre-training to live click logs (via CASTLE-R) as real traffic begins to accumulate.

The best search and RAG systems are not built by teams that wait for user data. They are built by teams that synthesize it with architectural discipline.