NanoSASRec

The simplest, most readable implementation of Self-Attentive Sequential Recommendation (SASRec), following the research line that made it the standard baseline and the improvements that eventually led to the Hierarchical Sequential Transduction Unit (HSTU).

The official curriculum is 10 training scripts backed by a small shared sasrec/

package.

Inspired by nanoGPT: every training version is runnable directly, every design decision cites a paper, and every version has a concrete reproduction target.

Most recommender systems (collaborative filtering, two-tower models, deep learning recommender models) learn what a user likes. They struggle with what a user needs right now. On Amazon, someone who just bought a DSLR camera doesn't need more camera recommendations. They need a lens, then a memory card, then a camera bag. With movies, a user who just watched all three Lord of the Rings films back-to-back is on a fantasy binge, not looking for the romantic comedies their five-year profile suggests. The sequence encodes intent progression that a static user profile can't capture.

This matters because real user behavior is streaky. People shop in projects, watch in moods, and listen in phases. Sequential models pick up on these short-term patterns while still remembering long-term preferences, which is why they recently dominate production recommendation at companies like Meta, YouTube, Kuaishou, and Meituan.

Sequential recommendation models the order of a user's actions. RNNs (GRU4Rec, 2016) did it first, but Transformers turned out to be better at it. SASRec (Kang & McAuley, 2018) was the first to apply a causal Transformer decoder to item sequences.

This project builds up SASRec from its original framing all the way to its HSTU variant, one component at a time. Each step fixes one problem, shows the metric improvement, and cites the paper that figured it out. To use LLMs as an analogy: if SASRec is the original Transformer, HSTU is GPT-2. Same model lineage, but with the refinements that unlocked scaling laws. By v10 you'll understand causal self-attention, loss function design for large catalogs, scaling laws for recommendation, and the specific architectural decisions that separate a research baseline from a production system.

uv sync
uv run python data.py --dataset ml-1m
uv run python train_v1.py --epochs 200 --eval-every 20  # --device cuda

Train v1 on MovieLens 1M in ~7 minutes on a MacBook. From there, use the Versions and Datasets tables to see how the curriculum progresses, then follow CURRICULUM.md for the v1-v10 commands and expected results. Reproduce the Zhai et al. (2024) HSTU MovieLens 20M benchmark in ~4 hours on a single A100.

See CURRICULUM.md for a walkthrough of the 10 official versions with training outputs, benchmark numbers, and detailed analysis of what works, what doesn't, and why.

Recent history:  Toy Story (1995), Bug's Life (1998), Antz (1998)
Ground truth:    Pocahontas (1995)
Model top-3:
  1. Lion King, The (1994)
  2. Little Mermaid, The (1989)
  3. Jungle Book, The (1967)

Each version is runnable independently. Read v1-v10 in order for the full progression, or jump to any version.

train_v11.py

is an appendix experiment for Dense All Action loss. It is useful context, but not part of the official curriculum path.

Below is the full-catalog evaluation on ML-20M (dim=256, 8 heads, 50 epochs) from v5-v10. Every model ranks the ground truth against all 18,345 items, not a sampled subset.

• relative bias | 0.3457 | 0.2085 | | v8 (8 blocks) | HSTU architecture | 0.3339 | 0.2006 | | v9 (8 blocks) | + φ1 pre-activation | 0.3425 | 0.2071 | | v10 (8 blocks) | + temp + L2 + collision | 0.3355 | 0.2016 |

Compared to Zhai et al. (2024) published results:

See CURRICULUM.md for the full analysis from v1 to v10.

See PAPERS.md for the full research lineage from GRU4Rec through HSTU and which paper justifies which code decision.

Python 3.12+ and PyTorch 2.7+. No frameworks, no config systems, no custom CUDA kernels.

dependencies = [
    "torch>=2.7",
    "numpy>=1.24",
]

If you plan to edit the repo, install the dev tools and run Ruff before committing changes:

uv sync --group dev
uv run ruff format .
uv run ruff check .

For longer ML-20M runs, remote.py

can launch any train_v*.py

script on Modal. Create a Modal account and authenticate locally before using it.

uv sync --group cloud
uv run modal run --detach remote.py --cmd "python -u train_v9.py --dataset ml-20m"

remote.py

automatically preprocesses the requested dataset, mounts the local training scripts and sasrec/

package, adds --device cuda

, and persists downloaded data in a Modal Volume.

If you plan to fan out several ML-20M jobs with sweep.sh

or manually, run one Modal job first and let it finish preprocessing. That warms the shared data volume and avoids multiple first-time jobs racing to download and extract the same dataset. If there are issues with dataset processing you may need to wipe the Modal Volume.

You can also use a regular GPU VM from providers like Lambda, Lightning AI, or any cloud where uv sync

can run.

This project is not sponsored.

In-batch negatives: use positive items from other users in the batch as harder negatives. Likely the missing piece for v10's temperature + L2 norm to work.Production optimization: KV-cache for incremental inference, FAISS for approximate nearest neighbor retrieval, mixed precision training.** Stochastic Length training**: randomly sample contiguous subsequences during training instead of always using the most recent items. SeePAPERS.md.