{"slug": "nanosasrec", "title": "NanoSASRec", "summary": "NanoSASRec, an open-source project inspired by nanoGPT, provides a minimal and readable implementation of the Self-Attentive Sequential Recommendation (SASRec) model, building up to the Hierarchical Sequential Transduction Unit (HSTU). The project includes 10 training scripts that progressively improve the model, citing papers for each design decision and targeting concrete reproduction benchmarks. It demonstrates that sequential models outperform static user profiles by capturing short-term intent patterns, a technique now dominant in production recommendation systems at companies like Meta, YouTube, and Kuaishou.", "body_md": "The simplest, most readable implementation of Self-Attentive Sequential Recommendation (SASRec), following the research line that made it the standard baseline and the improvements that eventually led to the Hierarchical Sequential Transduction Unit (HSTU).\n\nThe official curriculum is 10 training scripts backed by a small shared `sasrec/`\n\npackage.\n\nInspired by [nanoGPT](https://github.com/karpathy/nanoGPT): every training version is runnable directly, every design decision cites a paper, and every version has a concrete reproduction target.\n\nMost recommender systems (collaborative filtering, two-tower models, deep learning recommender models) learn what a user *likes*. They struggle with what a user *needs right now*. On Amazon, someone who just bought a DSLR camera doesn't need more camera recommendations. They need a lens, then a memory card, then a camera bag. With movies, a user who just watched all three Lord of the Rings films back-to-back is on a fantasy binge, not looking for the romantic comedies their five-year profile suggests. The sequence encodes intent progression that a static user profile can't capture.\n\nThis matters because real user behavior is *streaky*. People shop in projects, watch in moods, and listen in phases. Sequential models pick up on these short-term patterns while still remembering long-term preferences, which is why they recently dominate production recommendation at companies like Meta, YouTube, Kuaishou, and Meituan.\n\nSequential recommendation models the *order* of a user's actions. RNNs (GRU4Rec, 2016) did it first, but Transformers turned out to be better at it. **SASRec** (Kang & McAuley, 2018) was the first to apply a causal Transformer decoder to item sequences.\n\nThis project builds up SASRec from its original framing all the way to its HSTU variant, one component at a time. Each step fixes one problem, shows the metric improvement, and cites the paper that figured it out. To use LLMs as an analogy: if SASRec is the original Transformer, HSTU is GPT-2. Same model lineage, but with the refinements that unlocked scaling laws. By v10 you'll understand causal self-attention, loss function design for large catalogs, scaling laws for recommendation, and the specific architectural decisions that separate a research baseline from a production system.\n\n```\nuv sync\nuv run python data.py --dataset ml-1m\nuv run python train_v1.py --epochs 200 --eval-every 20  # --device cuda\n```\n\nTrain v1 on MovieLens 1M in ~7 minutes on a MacBook. From there, use the [Versions](#versions) and [Datasets](#datasets) tables to see how the curriculum progresses, then follow [CURRICULUM.md](/eifuentes/nanoSASRec/blob/main/CURRICULUM.md) for the v1-v10 commands and expected results. Reproduce the Zhai et al. (2024) HSTU MovieLens 20M benchmark in ~4 hours on a single A100.\n\nSee [CURRICULUM.md](/eifuentes/nanoSASRec/blob/main/CURRICULUM.md) for a walkthrough of the 10 official versions with training outputs, benchmark numbers, and detailed analysis of what works, what doesn't, and why.\n\n```\nRecent history:  Toy Story (1995), Bug's Life (1998), Antz (1998)\nGround truth:    Pocahontas (1995)\nModel top-3:\n  1. Lion King, The (1994)\n  2. Little Mermaid, The (1989)\n  3. Jungle Book, The (1967)\n```\n\n| Version | What changes | Dataset | Key finding |\n|---|---|---|---|\n`train_v1.py` |\nVanilla SASRec + BCE loss | ML-1M | BCE overconfidence problem |\n`train_v2.py` |\nSampled softmax loss | ML-1M | Competition between candidates fixes ranking |\n`train_v3.py` |\ngBCE loss + calibration | ML-1M | Calibrated probabilities, but SSM ranks better |\n`train_v4.py` |\nInit, reversed positions, dropout | ML-1M | Implementation details compound to +14% NDCG |\n`train_v5.sh` |\nDepth sweep (2/4/8/16 blocks) | ML-20M | Depth pays off, diminishing returns after 8 |\n`train_v6.py` |\nReZero residual scaling | ML-20M | 8 blocks + ReZero > 16 blocks without |\n`train_v7.py` |\nRelative position + temporal bias | ML-20M | Best model in repo, biggest single improvement |\n`train_v8.py` |\nSiLU + gating (HSTU) | ML-20M | Architecture swap is not a clear win at this scale |\n`train_v9.py` |\nφ1 SwishLayerNorm pre-activation | ML-20M | Beats the Zhai et al. (2024) HSTU result by +4.6% |\n`train_v10.py` |\nTemperature, L2 norm, collision masking | ML-20M | Production techniques need in-batch negatives to help |\n\nEach version is runnable independently. Read v1-v10 in order for the full progression, or jump to any version.\n\n`train_v11.py`\n\nis an appendix experiment for Dense All Action loss. It is useful context, but not part of the official curriculum path.\n\n| Dataset | Role | Users | Items | Avg len | When used |\n|---|---|---|---|---|---|\n| ML-1M | Curriculum baseline | 6,040 | 3,416 | 165.5 | v1-v4 |\n| ML-20M | Benchmark target | 138,493 | 18,345 | 144.3 | v5-v10 |\n\nBelow is the full-catalog evaluation on ML-20M (dim=256, 8 heads, 50 epochs) from v5-v10. Every model ranks the ground truth against all 18,345 items, not a sampled subset.\n\n| Version | What changed | HR@10 | NDCG@10 |\n|---|---|---|---|\n| v5 (2 blocks) | SASRec baseline | 0.3009 | 0.1760 |\n| v5 (4 blocks) | + depth | 0.3149 | 0.1869 |\n| v5 (8 blocks) | + more depth | 0.3237 | 0.1941 |\n| v5 (16 blocks) | + even more depth | 0.3350 | 0.2026 |\n| v6 (8 blocks) | + ReZero | 0.3365 | 0.2020 |\nv7 (8 blocks) |\n+ relative bias |\n0.3457 |\n0.2085 |\n| v8 (8 blocks) | HSTU architecture | 0.3339 | 0.2006 |\n| v9 (8 blocks) | + φ1 pre-activation | 0.3425 | 0.2071 |\n| v10 (8 blocks) | + temp + L2 + collision | 0.3355 | 0.2016 |\n\nCompared to Zhai et al. (2024) published results:\n\n| Method | HR@10 | NDCG@10 |\n|---|---|---|\n| SASRec | 0.2889 | 0.1621 |\n| HSTU | 0.3273 | 0.1895 |\n| HSTU-large | 0.3556 | 0.2098 |\n\nSee [CURRICULUM.md](/eifuentes/nanoSASRec/blob/main/CURRICULUM.md) for the full analysis from v1 to v10.\n\nSee [PAPERS.md](/eifuentes/nanoSASRec/blob/main/PAPERS.md) for the full research lineage from GRU4Rec through HSTU and which paper justifies which code decision.\n\nPython 3.12+ and PyTorch 2.7+. No frameworks, no config systems, no custom CUDA kernels.\n\n```\ndependencies = [\n    \"torch>=2.7\",\n    \"numpy>=1.24\",\n]\n```\n\nIf you plan to edit the repo, install the dev tools and run Ruff before committing changes:\n\n```\nuv sync --group dev\nuv run ruff format .\nuv run ruff check .\n```\n\nFor longer ML-20M runs, `remote.py`\n\ncan launch any `train_v*.py`\n\nscript on [Modal](https://modal.com/). Create a Modal account and authenticate locally before using it.\n\n```\nuv sync --group cloud\nuv run modal run --detach remote.py --cmd \"python -u train_v9.py --dataset ml-20m\"\n```\n\n`remote.py`\n\nautomatically preprocesses the requested dataset, mounts the local training scripts and `sasrec/`\n\npackage, adds `--device cuda`\n\n, and persists downloaded data in a *Modal Volume*.\n\nIf you plan to fan out several ML-20M jobs with `sweep.sh`\n\nor manually, run one Modal job first and let it finish preprocessing. That warms the shared data volume and avoids multiple first-time jobs racing to download and extract the same dataset. If there are issues with dataset processing you may need to wipe the *Modal Volume*.\n\nYou can also use a regular GPU VM from providers like [Lambda](https://lambda.ai/instances), [Lightning AI](https://lightning.ai/), or any cloud where `uv sync`\n\ncan run.\n\nThis project is not sponsored.\n\n**In-batch negatives**: use positive items from other users in the batch as harder negatives. Likely the missing piece for v10's temperature + L2 norm to work.**Production optimization**: KV-cache for incremental inference, FAISS for approximate nearest neighbor retrieval, mixed precision training.** Stochastic Length training**: randomly sample contiguous subsequences during training instead of always using the most recent items. See[PAPERS.md](/eifuentes/nanoSASRec/blob/main/PAPERS.md).", "url": "https://wpnews.pro/news/nanosasrec", "canonical_source": "https://github.com/eifuentes/nanoSASRec", "published_at": "2026-07-14 18:32:37+00:00", "updated_at": "2026-07-14 18:48:05.486334+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "large-language-models", "ai-research", "ai-products"], "entities": ["NanoSASRec", "SASRec", "HSTU", "Meta", "YouTube", "Kuaishou", "Meituan", "MovieLens"], "alternates": {"html": "https://wpnews.pro/news/nanosasrec", "markdown": "https://wpnews.pro/news/nanosasrec.md", "text": "https://wpnews.pro/news/nanosasrec.txt", "jsonld": "https://wpnews.pro/news/nanosasrec.jsonld"}}