NanoSASRec

NanoSASRec, an open-source project inspired by nanoGPT, provides a minimal and readable implementation of the Self-Attentive Sequential Recommendation (SASRec) model, building up to the Hierarchical Sequential Transduction Unit (HSTU). The project includes 10 training scripts that progressively improve the model, citing papers for each design decision and targeting concrete reproduction benchmarks. It demonstrates that sequential models outperform static user profiles by capturing short-term intent patterns, a technique now dominant in production recommendation systems at companies like Meta, YouTube, and Kuaishou.

The simplest, most readable implementation of Self-Attentive Sequential Recommendation SASRec , following the research line that made it the standard baseline and the improvements that eventually led to the Hierarchical Sequential Transduction Unit HSTU . The official curriculum is 10 training scripts backed by a small shared sasrec/ package. Inspired by nanoGPT https://github.com/karpathy/nanoGPT : every training version is runnable directly, every design decision cites a paper, and every version has a concrete reproduction target. Most recommender systems collaborative filtering, two-tower models, deep learning recommender models learn what a user likes . They struggle with what a user needs right now . On Amazon, someone who just bought a DSLR camera doesn't need more camera recommendations. They need a lens, then a memory card, then a camera bag. With movies, a user who just watched all three Lord of the Rings films back-to-back is on a fantasy binge, not looking for the romantic comedies their five-year profile suggests. The sequence encodes intent progression that a static user profile can't capture. This matters because real user behavior is streaky . People shop in projects, watch in moods, and listen in phases. Sequential models pick up on these short-term patterns while still remembering long-term preferences, which is why they recently dominate production recommendation at companies like Meta, YouTube, Kuaishou, and Meituan. Sequential recommendation models the order of a user's actions. RNNs GRU4Rec, 2016 did it first, but Transformers turned out to be better at it. SASRec Kang & McAuley, 2018 was the first to apply a causal Transformer decoder to item sequences. This project builds up SASRec from its original framing all the way to its HSTU variant, one component at a time. Each step fixes one problem, shows the metric improvement, and cites the paper that figured it out. To use LLMs as an analogy: if SASRec is the original Transformer, HSTU is GPT-2. Same model lineage, but with the refinements that unlocked scaling laws. By v10 you'll understand causal self-attention, loss function design for large catalogs, scaling laws for recommendation, and the specific architectural decisions that separate a research baseline from a production system. uv sync uv run python data.py --dataset ml-1m uv run python train v1.py --epochs 200 --eval-every 20 --device cuda Train v1 on MovieLens 1M in ~7 minutes on a MacBook. From there, use the Versions versions and Datasets datasets tables to see how the curriculum progresses, then follow CURRICULUM.md /eifuentes/nanoSASRec/blob/main/CURRICULUM.md for the v1-v10 commands and expected results. Reproduce the Zhai et al. 2024 HSTU MovieLens 20M benchmark in ~4 hours on a single A100. See CURRICULUM.md /eifuentes/nanoSASRec/blob/main/CURRICULUM.md for a walkthrough of the 10 official versions with training outputs, benchmark numbers, and detailed analysis of what works, what doesn't, and why. Recent history: Toy Story 1995 , Bug's Life 1998 , Antz 1998 Ground truth: Pocahontas 1995 Model top-3: 1. Lion King, The 1994 2. Little Mermaid, The 1989 3. Jungle Book, The 1967 | Version | What changes | Dataset | Key finding | |---|---|---|---| train v1.py | Vanilla SASRec + BCE loss | ML-1M | BCE overconfidence problem | train v2.py | Sampled softmax loss | ML-1M | Competition between candidates fixes ranking | train v3.py | gBCE loss + calibration | ML-1M | Calibrated probabilities, but SSM ranks better | train v4.py | Init, reversed positions, dropout | ML-1M | Implementation details compound to +14% NDCG | train v5.sh | Depth sweep 2/4/8/16 blocks | ML-20M | Depth pays off, diminishing returns after 8 | train v6.py | ReZero residual scaling | ML-20M | 8 blocks + ReZero 16 blocks without | train v7.py | Relative position + temporal bias | ML-20M | Best model in repo, biggest single improvement | train v8.py | SiLU + gating HSTU | ML-20M | Architecture swap is not a clear win at this scale | train v9.py | φ1 SwishLayerNorm pre-activation | ML-20M | Beats the Zhai et al. 2024 HSTU result by +4.6% | train v10.py | Temperature, L2 norm, collision masking | ML-20M | Production techniques need in-batch negatives to help | Each version is runnable independently. Read v1-v10 in order for the full progression, or jump to any version. train v11.py is an appendix experiment for Dense All Action loss. It is useful context, but not part of the official curriculum path. | Dataset | Role | Users | Items | Avg len | When used | |---|---|---|---|---|---| | ML-1M | Curriculum baseline | 6,040 | 3,416 | 165.5 | v1-v4 | | ML-20M | Benchmark target | 138,493 | 18,345 | 144.3 | v5-v10 | Below is the full-catalog evaluation on ML-20M dim=256, 8 heads, 50 epochs from v5-v10. Every model ranks the ground truth against all 18,345 items, not a sampled subset. | Version | What changed | HR@10 | NDCG@10 | |---|---|---|---| | v5 2 blocks | SASRec baseline | 0.3009 | 0.1760 | | v5 4 blocks | + depth | 0.3149 | 0.1869 | | v5 8 blocks | + more depth | 0.3237 | 0.1941 | | v5 16 blocks | + even more depth | 0.3350 | 0.2026 | | v6 8 blocks | + ReZero | 0.3365 | 0.2020 | v7 8 blocks | + relative bias | 0.3457 | 0.2085 | | v8 8 blocks | HSTU architecture | 0.3339 | 0.2006 | | v9 8 blocks | + φ1 pre-activation | 0.3425 | 0.2071 | | v10 8 blocks | + temp + L2 + collision | 0.3355 | 0.2016 | Compared to Zhai et al. 2024 published results: | Method | HR@10 | NDCG@10 | |---|---|---| | SASRec | 0.2889 | 0.1621 | | HSTU | 0.3273 | 0.1895 | | HSTU-large | 0.3556 | 0.2098 | See CURRICULUM.md /eifuentes/nanoSASRec/blob/main/CURRICULUM.md for the full analysis from v1 to v10. See PAPERS.md /eifuentes/nanoSASRec/blob/main/PAPERS.md for the full research lineage from GRU4Rec through HSTU and which paper justifies which code decision. Python 3.12+ and PyTorch 2.7+. No frameworks, no config systems, no custom CUDA kernels. dependencies = "torch =2.7", "numpy =1.24", If you plan to edit the repo, install the dev tools and run Ruff before committing changes: uv sync --group dev uv run ruff format . uv run ruff check . For longer ML-20M runs, remote.py can launch any train v .py script on Modal https://modal.com/ . Create a Modal account and authenticate locally before using it. uv sync --group cloud uv run modal run --detach remote.py --cmd "python -u train v9.py --dataset ml-20m" remote.py automatically preprocesses the requested dataset, mounts the local training scripts and sasrec/ package, adds --device cuda , and persists downloaded data in a Modal Volume . If you plan to fan out several ML-20M jobs with sweep.sh or manually, run one Modal job first and let it finish preprocessing. That warms the shared data volume and avoids multiple first-time jobs racing to download and extract the same dataset. If there are issues with dataset processing you may need to wipe the Modal Volume . You can also use a regular GPU VM from providers like Lambda https://lambda.ai/instances , Lightning AI https://lightning.ai/ , or any cloud where uv sync can run. This project is not sponsored. In-batch negatives : use positive items from other users in the batch as harder negatives. Likely the missing piece for v10's temperature + L2 norm to work. Production optimization : KV-cache for incremental inference, FAISS for approximate nearest neighbor retrieval, mixed precision training. Stochastic Length training : randomly sample contiguous subsequences during training instead of always using the most recent items. See PAPERS.md /eifuentes/nanoSASRec/blob/main/PAPERS.md .