Recommendation systems have evolved from simple neighborhood search to complex deep learning or generative model-based architectures over time. This article will discuss the journey of recommendation systems from the earliest filtering-based approach towards the most recent autoregressive generation. Each method is broken into sections: idea and motivation, high-level overview, deep dive, and advantages. If you wish not to go into details, you are free to skip the Deep Dive sections.
Idea and Motivation #
Collaborative filtering, or CF, operates on the fundamental assumption that users who have agreed on the past will agree in the future also. This approach recommends items purely based on past user interactions with different items. CF has two paradigms
- Item-based - Here, the similarity between two items is measured based on the consumption patterns. Similar movies are those rated similarly by users. If User A, User B, and User C all tend to rate Movie A and Movie B identically (e.g., giving both 5 stars), the algorithm determines those two movies are similar.
- User-based - Identifies similar users whose interactions and preferences closely align with the target user. As shown in the figure below, to recommend a movie to Joe, the system first finds similar users based on common interests. Then, a good recommendation for Joe will be a movie that these similar users collectively agree upon; in this case, Movie #3
High-level approach #
The goal is to find similar users (in a user-oriented approach) or similar items(in an item-oriented approach). In a user-oriented approach, similar users are those who liked the same movies. In the item-oriented approach, similar movies are the movies that are rated similarly by users. This will create an item-item similarity matrix (a huge matrix which captures similarity scores between all item-item pairs) or a user-similarity matrix (user-user similarity scores between all pairs of users). From this, the top similar items for an item or top similar users for a user can be retrieved. This will then be used to recommend items to a user.
Deep dive #
Pairwise correlations between two users (user a and user b) or items (item i and item j) are calculated using metrics such as cosine similarity. Then similarity across all items and users will be calculated, and the k nearest neighbors will be stored.
To recommend an item to a user, the k nearest neighbors for the user and the items rated by them are fetched. Now, for each of these items, a weighted score will be calculated for the user, based on the rating given by each neighbor and their similarity score with the user.
Advantages #
- Requires no additional metadata or any complex models.
- Highly interpretable
Idea and Motivation #
Previous recommendation systems had several bottlenecks, like
- Similarities are purely based on past interactions. If user A and user B actually shared similar interests but haven’t rated the same movies, they will be considered different
- In the real world, the user-item interactions are very sparse because a user might be only interacting with 10% or 15% of the items. This little overlap can result in inaccuracies and overgeneralization.
- The recommendation is entirely based on the user-item matrix. To recommend an item to the user, the algorithm has to scan the entire user dataset to find similar users and find the target movies on the go.
High-level approach #
Instead of completely relying on hard-coded user-item ratings, here each item/movie is represented by certain factors. As shown in the figure below, each movie can be represented on different scales. For example, where does the movie sit on the scale: serious vs. escapist? Similarly, a movie can be represented using different measures such as comedy vs. drama, action vs. romance, etc. The users will now be placed in this space based on how much they have rated movies from each genre. In this example, we can say that the user “Gus” will mostly like Dumb and Dumber and might hate The Color Purple. One important point here is that the algorithm does not actually read movie genres or know what a “comedy movie” is. It mathematically discovers these factors or scales purely by finding patterns in how users rate items together.
Deep dive #
To find the latent factors for each item and user, the sparse user-item interaction matrix is approximated as the product of two low-rank dense matrices.
where P (U x K dimension) contains the user latent features and Q (I x K dimension) consists of latent vectors for items. To handle bias (some users tend to be more critical and give low ratings for all movies; some movies can be blockbusters and are mostly rated very high), bias terms are also introduced as
where \mu is the global average rating, b_u is the user bias, and b_i is the item bias
The parameters bu, bi, p, and q are learned by minimizing the squared error loss over the actual recorded rating data
The L2 regularization term lambda prevents overfitting here.
To recommend an item or movie to a user, a simple dot product with additional user and item bias terms is used.
The candidate items are ranked based on these scores for a given user u.
Advantages #
- The dimensionality reduction solves the extreme sparsity issue.
- The model was able to capture relationships between users who have never rated the same items
Idea and Motivation #
Earlier methods like Collaborative filtering and matrix factorization represent the user and item purely based on interaction history. Matrix factorization does use some derived latent factors, but these are again derived from the interaction or rating history. These methods fail to capture other details. The two-tower model is a deep learning-based approach which non-linear transformations to generate user and item representations. The two-tower model makes use of other information like user age, user demographics, item description, etc and creates a rich representation for each item and user.
High-level approach #
Instead of rating entirely on the rating patterns, a two-tower model will incorporate rich, meaningful context. This consists of two independent Deep Neural network models. The user model, or user tower, will ingest all user-specific features like user age, user demographics, device, previous interactions, etc. The item model, or item tower, ingests item description, image features, and other item-specific details. Both models process this data and create a vector representation. Even though the towers process completely different types of data, they translate everything onto the same shared dimension. To generate recommendations, the system simply calculates the similarity between a user and the items. A user doesn’t have to be scanned against all the available items to find similarity. Since the users and items are mapped to the same shared space, this approach uses neighborhood search to check only items in the immediate neighborhood of the users.
Deep dive #
- User tower: This will produce a user feature vector using user features like User ID, past interaction history, user demographics, device type, etc. These features are passed through a deep learning neural network to output the dense vector u
- Item tower: This will produce an item feature vector using item-specific features like item ID, text descriptions, image features, etc. These features are passed through another deep neural network model to get the dense vector v
- Both towers are trained using positive item-user interactions and sampled negatives. The objective is to maximize the similarity score (Dot product or Cosine similarity) between the positive samples and minimize the score for the negative or irrelevant items.
Item vector embeddings are generated offline and stored separately in a vector index. To recommend an item to a user, the updated user’s features (including latest interactions) are passed through the user tower (forward pass) to compute the embedding u. Now, to get the relevant recommendations for the user, a similarity score needs to be calculated between u and item vectors. But instead of computing the dot product across millions or billions of items, Approximate Nearest Neighbor Search is used (there are Graph based, cluster based or product quantization-based methods) to retrieve the top k candidates in milliseconds or even less time.
Advantages #
- Decoupling user and item embeddings enables offline computing and indexing of item vectors
- Integrate different types of features like Sparse IDs, dense embeddings(textual descriptions), image features, etc.
- Content-based user and item features help in recommendation for new users or unseen items even without any historical interaction data
Idea or Motivation #
While the two-tower approach did a great breakthrough in capturing a non-linear representation of users and items, there were several bottlenecks. One issue is that the two-tower model represents the entire user’s interaction in a single fixed-size embedding u, followed by a single dot product (u.v) for similarity search. This often misses the user intent across multiple scenarios and interactions. Another drawback was the use of random integer IDs to represent items. Item 1 and Item 2 might be similar, but their IDs don't share any semantic relationship to indicate the similarity. Generative retriever treats recommendation as an autoregressive sequence-to-sequence generation task. Instead of searching for similar items, a Transformer model directly generates the target item identifier. This recommendation system is referred to as Transformer Index for GEnerative Recommenders (TIGER), developed by Google.
High-level approach #
TIGER completely shifts away from the concept of database search for recommendation. Instead, it treats it as a generative task. Given the user's preferences, what will be the next item the user might purchase? TIGER represents an item using fixed-size Semantic IDs represented as tuples (Eg:, (7,14,21,…)). These IDs represent the item in terms of certain factors, like in the latent factor approach. But unlike latent factor models, here the IDs are generated using a complex encoder model. The IDs have semantic meaning, where the items with similar features share the same ID prefixes. The user’s chronological interaction history will be passed to another Transformer model (encoder-decoder) as a sentence of these Semantic IDs. It then uses the Transformer to predict the next ID, token by token, exactly like a Large Language Model predicting the next word in a sentence.
Deep dive #
The architecture consists of multiple stages as shown in the figure below.
Stage 1: Semantic ID generation
The item metadata (descriptions, titles, categories, style, etc.) is first passed through a pretrained language model (like Sentence T5) to generate a dense semantic text embedding vector v. This vector is then passed to a Residual Quantized Variational Auto-Encoder (RQ-VAE) model and converted into a continuous latent vector z. This vector is recursively passed across m sequential codebooks of size K. Each codebook can be thought of as a level, where it represents the item in terms of each type of factor. For example, in the figure below, an item is represented as (7,1,4). The value 7 from Level 1 can represent its category (eg: mobile device), the value 1 from Level 2 can represent its subcategory (eg: touch screen), and the value 4 from Level 3 can represent a finer attribute like size (eg: 5.5 inch).
The quantized centroid sum will be
where e_cj is the semantic code index for jth notebook. This z is passed through a decoder model to reconstruct the original dense vector x. This network is trained to minimize the mean squared error
Stage 2: Generative model
The user’s historical interaction sequence, as shown in Figure 3, will be the flattened semantic IDs of previously interacted items. This vector is then passed to a Transformer model for generation. The model is trained end-to-end via teacher-forced cross-entropy loss to predict the target item’s semantic ID tokens autoregressively.
Inference
To recommend an item to a user, the user’s interaction history is given to the generative Transformer model. To prevent the model from hallucinating invalid semantic ID tokens, the generation can be constrained using a prefix tree.
Advantages #
Hierarchical Semantic IDs that share prefix tokens across related items will enable rich collaborative and semantic transfer learning, naturally softening the cold-start bottleneck for content-similar items. Semantic token vocabularies are drastically smaller (m x K, often just a few thousand tokens) than classic sparse embedding tables that scale linearly with millions of raw item IDs.
Generative recommendation frameworks like Google’s TIGER rely on static tokenizers. An item is mapped into a single immutable Semantic ID tuple using context features passed through a complex model (quantized autoencoder). As mentioned, this also shares common prefix tokens (items in the same broad category share the same prefix) and assumes universal similarity across different items. But in reality, an item can mean different things to different users.
Consider the following figure
Consider the item watch, which was bought by three distinct users on three different occasions. The first user purchased it for formal attire. The second user purchased it as an investment or a luxury asset. The third user purchased it as a romantic/holiday gift.
A static single semantic ID cannot represent this item in these three different perspectives. A personalized, context-aware tokenizer, or Pctx, addresses this by introducing personalized and context-aware tokenization, which allows a single item to be mapped into a set of discrete semantic IDs.
High-level approach #
In TIGER, an item is represented by a single Semantic ID tuple based on its fixed features. The Pctx (Personalized ConTeXt aware tokenizer) framework addresses this by shifting from a rigid 1-1 mapping to a dynamic set of representations. Instead of a single Semantic ID, an item will be represented using a set of IDs representing the semantic meaning of the item along with how the item fits in multiple contexts.
By analyzing the unique sequence of a user’s past behaviors, the model learns to tokenize the item differently for each user, which then helps in recommending items based on the user's intent. If a particular item is bought by N users, then that item might have N different contexts. But representing each item based on all different users is practically not feasible or scalable. If an item like a watch is bought by 500000 users, generating 500000 unique Semantic IDs will crash the system. Instead, Pctx solves this by grouping similar user interactions. So for an item like a watch, there will be K different groups of people where K« N, as shown in Figure 5.
- User group 1: Users buying for formal attire.
- User group 2: Users buying as an investment.
- User group 3: Users buying as a gift.
These groups are called clusters. The center of each cluster (the centroid) is then converted into a Semantic ID using an encoding model (Similar to TIGER). Instead of 50,000 IDs, the watch now has just three or four context-aware IDs that represent its real-world use cases.
Deep dive #
The whole approach can be split into two phases:
Phase 1: Multi-facet Semantic ID generation
- Suppose User A’s history is [Suit, Briefcase, Formal shoes] —> Watch, and User B’s history is [Flowers, Gifts] —>Watch. These context vectors will be passed through an auxiliary model to generate a dense vector.
- where
- e_vi is the encoded user context representation for item v_i and its associated context [v1, v2, . . . , vi−1]
- [ v_1, v_2, …, v_i-1] is the user’s historical interaction sequence preceding the ith step (the historical context).
- f(·) denotes a neural sequence model
- Now, instead of generating a static semantic ID for the target item (the watch in the above example), the model will generate context-aware IDs.
- The challenge here is that there can be tens of thousands of users who might have bought a Watch. In that case, that item will have a huge set of semantic IDs, which will again lead to scalability issues.
- To solve this, Pctx aggregates all the historical interaction vectors for a specific item and groups them into a small set of facets, as shown in Figure 5 as Context Representations. The centroid for each cluster represents the item representation in a particular scenario. For example, in the above figure
- Centroid 1: Users have bought it for formal attire
- Centroid 2: Users have bought it for investment
- Centroid 3: Users have bought it as gifts
- These cluster centroids are then quantized using RQ VAE (discussed in the TIGER model) to generate multiple semantic IDs for an item
Phase 2: Generative model training
Historical interactions, which are converted into the Semantic IDs, are passed through a sequence-to-sequence Transformer model. The model is trained via standard teacher-forced cross-entropy loss to autoregressively predict the target item’s specific facet Semantic ID token by token.
Inference
The user's historical interaction sequence is given to the trained generative model to generate the possible candidates. Because item i may map to multiple valid SIDs (e.g., SID_1(i) and SID_2(i)), individual beam probabilities for all paths corresponding to the same physical item are summed
The items are aggregated by their probability scores, and the top k items will be recommended to the user.
Advantages #
- Resolves Intent Ambiguity: Decouples item identity from a single static representation, enabling the model to recommend the same catalog item across completely divergent user journey paths.
- Preserves Compact Footprint: Expands the semantic expressive power by generating multiple IDs per item without expanding the underlying discrete token vocabulary size.
Note #
This work is currently under review as a conference paper at ICLR 2026.
This article discussed the progression of recommendation systems from memory-based neighborhood filtering models to advanced personalised generative tokenization-based approaches. While the two-tower model remains one of the widely used methods due to their low latency and decoupled indexing, generative recommenders eliminate the trade-offs of uninformative random IDs and rigid static single-vector embeddings.
I hope you enjoyed the article. Please feel free to reach out for any questions or suggestions via LinkedIn
1. https://datajobs.com/data-science-repo/Recommender-Systems-[Netflix].pdf
2. https://tullie.ai/blog/two-tower-recommendation-models
- https://arxiv.org/pdf/2305.05065
- https://openreview.net/pdf?id=ahpO7S1Ppi