The pitch usually goes: send the catalog and the user's history to a model, get back a ranked list, delete the recommender. It demos beautifully on 200 items and falls over the moment the catalog has 400,000 and the response has 80 milliseconds to come back.
The 2026 research doesn't support the replacement story either. RecoChain (arxiv 2604.25787) proposes unified generative retrieval and ranking, then evaluates it against classic top-k ranking metrics, because the measurement requirement doesn't go away. RRCM (arxiv 2605.07129) frames LLM recommendation as retrieval-and-reasoning, where the model decides when to pull more evidence and optimization still runs against the final ranking reward, not against the quality of the model's prose.
Which puts the LLM inside two layers of a four-layer stack, not on top of it.
The four layers
Data and features. User events, item metadata, collaborative signals. Both papers above assume this layer is solid before anything generative gets added. It is also where roughly 80% of the project effort actually goes, whether you build or buy.
Candidate retrieval. Cut millions of items to a few hundred. Hybrid by default: a collaborative path plus a metadata and embedding path, usually pgvector or a managed equivalent.
Ranking. A deterministic model that scores and orders the shortlist. This is the layer with offline and online evaluation attached, and it's the one you're actually shipping.
Feedback and experimentation. Clicks, conversions, saves, return visits, flowing back into layer 1.
The LLM spans layers 2 and 3. It does semantic feature extraction, and it decides what evidence to retrieve. It does not emit the final ordering.
Why the latency budget forces this
Work backwards from a 100ms p99 for the whole call and the architecture stops being a matter of taste:
request
├─ 5ms feature fetch (user vector, recent events, cached)
├─ 20ms retrieval, parallel
│ ├─ collaborative: ANN over user-item embeddings → 200 candidates
│ └─ metadata: pgvector + hard filters → 200 candidates
│ union, dedupe → ~300
├─ 35ms ranking: one batched scoring pass over ~300 rows
├─ 10ms business rules: diversity, in-stock, dedupe by brand
└─ 5ms assemble + log the impression
There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback.
That fallback matters more than the feature. A recommendation strip that degrades to a popularity-ranked list under load is fine. One that returns nothing because a model call hung is a broken page.
Log impressions, not just clicks
The single most common data-layer mistake: recording what users clicked without recording what they were shown.
impression: { request_id, user_id, item_ids[], positions[],
ranker_version, retrieval_source[], ts }
interaction:{ request_id, item_id, type, ts }
Without the impression row you cannot compute a click-through rate, cannot correct for position bias, and cannot run a counterfactual evaluation of a new ranker against logged traffic. You are left with online A/B tests as the only way to learn anything, which is slow and expensive. retrieval_source per item matters too: when quality drops you need to know whether the collaborative path or the embedding path produced the bad candidate.
Offline metrics worth wiring from the start: recall@k for retrieval (did the item the user eventually chose make the shortlist at all?) and NDCG@k for ranking. These answer different questions and a lot of teams conflate them. A ranking model cannot fix a candidate set that never contained the right item.
Recommendation versus matching
A recommendation engine ranks items for one user. A matching engine pairs two sides of a marketplace: buyer and seller, designer and client, shift and worker.
Same four layers. Two differences worth planning for. Matching has mutual constraints, so a candidate has to clear both sides' filters, which means the retrieval layer runs twice and intersects. And matching usually has supply exhaustion: recommending the same top-rated contractor to 400 buyers produces 399 disappointments, so the ranking layer needs a fairness or throttling term that pure item recommendation doesn't.
Build order, from zero
Teams that invert this ship a demo in week two and spend month four discovering they have no way to tell whether it works.
Build or buy
The real axis is who holds the behavioral data. If those signals are the product's differentiation, renting layers 1 and 2 means renting the moat. If recommendations are a convenience feature on someone else's core product, SaaS is the correct answer and the architecture above still tells you what you're renting.
Either way the 80% data-preparation share doesn't move. Buying changes who runs the layers, not whether they exist.
Checklist
One note on claims
The vendor marketing in this category is unusually bad. "3% to 45% conversion lift" with no baseline, no methodology, and no named company is not a benchmark. If you haven't measured a number on your own traffic, saying so is a trust signal rather than a weakness, and it's a reasonable thing to expect from a partner too.
If you're scoping one
The order matters more than the model choice: data pipeline, then retrieval, then a ranking layer you can evaluate, then generative components under control. That's the conversation worth having before anyone picks a vector database. brocoders.com