LLMs From Scratch Reaches 100,000 GitHub Stars

Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people.

Thanks also to everyone who shared ideas and opened PRs with improvements!

Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories).

I am also currently working on a larger applied custom “small” LLM project. It has been keeping me super busy this month, but I will share more on that soon in an upcoming Substack mega-article! It’s my longest one yet!

If you are new to it, some of the highlights in the repo include #

Of course, the complete code path from

tokenizationandattentiontopretraining,classification, andinstruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.) - From-scratch implementations of

Llama,Qwen,Gemma, andOlmo(smaller variants that run locally and can be plugged into the training scripts). - From-scratch implementations of attention alternatives and other architecture components, such as

GQA,MLA,sliding-window attention,Gated DeltaNet,DeepSeek Sparse Attention,cross-layer KV sharing, andmixture-of-experts - Materials on

KV caching,training performance,memory-efficient weight ,DPO,evaluation, andLoRA

So, if you don’t have any weekend plans yet, happy tinkering!

Source: website version of my Substack note.

Read Next #

Kimi K3 Architecture Notes Short architecture note on Kimi K3, including LatentMoE, Kimi Delta Attention, Attention Residuals, NoPE, multimodality, and inference-efficiency choices.

A Few Notable Open-Weight Models This Week Short note on the architectures of six new open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3.

Correction for Listing 6.5 in Build a Reasoning Model From Scratch Short correction note for the random seed in Listing 6.5 on page 198 of Build a Reasoning Model From Scratch.