A self-study course building a tiny LLM stack from scalar autodiff up through a working chat assistant, in two parts. Modeled after From NAND to Tetris: the codebase grows layer-by-layer, and every block of the stack is something you write yourself.
The hard constraint: everything runs on an M-series MacBook — no cloud GPUs, no paid compute.
Why this course¶ #
Millions interact with LLMs on a daily basis. But few bother to understand how these systems actually work. How does an LLM understand language? What is a "model" and how does it learn? How does a chat assistant "know" how to answer a question? How does matrix multiplication produce intelligent behavior? In this course we answer these questions from first principles.
Our goal is to go below the API. To understand every part of the LLM stack, end to end. You start with basic autodiff, and over twenty modules grow it: tensors, a neural net, a tokenizer, embeddings, attention, the transformer, pretraining, sampling, SFT, DPO, evaluation, RAG, tool use, an agent loop. By the final module you have a chat assistant you built end to end — tools, retrieval, and a swappable backend: the tiny model you trained yourself, or a stronger local open model when you want it to be genuinely useful. All on your laptop. All built by you.
The scaffolding is real but small: a tokenizer that takes ten minutes to train, a transformer with a few million parameters, a corpus that fits in RAM. Tiny is deliberate. Once you've built every layer once at toy scale, the production-scale versions stop being magic.
What's in it¶ #
Twenty modules plus a fast prerequisite review, in two parts. Each module is roughly one week of effort at the level of a rigorous elite-college course.
Part I — From gradients to a language model¶
Build the model itself. You end with a language model whose every layer you wrote, generating readable text on your laptop.
| # | Module | Group |
|---|---|---|
| 00 | ||
Scalar autodiffTensors and matmulA first neural networkTrainingTokenizationEmbeddings and positionsNext-token predictionSelf-attentionMulti-head attentionThe transformer blockPretrainingMilestone: TinyLLMSampling and decodingPart I ends here, and finishing it is a real accomplishment. You will have built a working language model from scalar derivatives up, with no black boxes in the path.
Part II — From a language model to ChatGPT¶
Build the system around a model. A different subject from Part I, not a harder one: less derivation, more systems engineering.
| # | Module | Group |
|---|---|---|
| 12 | ||
Instruction tuning (SFT)LoRAPreference tuning (DPO)Hallucination and evaluationLocal pretrained models and inferenceSynthetic dataRetrieval-augmented generationTool useAgent loopsCapstone: a tiny ChatGPTPart II can also be entered directly if you already know the fundamentals and want the systems material — see Module 12.
The syllabus lays out each module in detail and gives the full motivation for the ordering.
Who it's for¶ #
You'll get the most out of this if you're comfortable with Python, undergraduate calculus (chain rule, gradients), and basic linear algebra. You don't need prior deep learning experience — Module 0 covers the prerequisites and Modules 1–3 build the math substrate from scratch. You should be willing to read a paper now and then, and be willing to debug your own code without a framework hiding the failure mode.
If you've watched Karpathy's videos and wished for a structured curriculum with exercises, deliverables, and tests, this is that.
How the course works¶ #
Each module in the course has a lesson page and a set of deliverables combining a coding project and a problem set. The lesson pages are hosted on this site and also available as markdown in the course repo.
Each week, students will read the module lesson page. They'll then implement a new sub-package covering that week's topic inside the g2c/
Python package. Finally they'll complete a set of student exercises in a Jupyter notebook using the code they wrote that week. The lesson pages are the readable front door; the repo is where the code lives.
Get started¶ #
- Read the syllabusfor the full arc across both parts. - Clone the repositoryand follow the README quickstart to set up on your machine. - Start with Module 0: Prerequisite review, or jump straight intoModule 1: Scalar autodiff.
If you get stuck¶ #
Two things, in this order.
Keep moving. You don't have to debug backwards through five modules to make progress. Hand back the reference implementations for the modules you aren't working on and carry on with your own code everywhere else:
./notebook.sh 12 --solutions=01-07
Then tell us where it happened. Open a stuck report naming the module. Nothing in the course phones home, so a module that quietly loses people is invisible to us unless someone says so. It genuinely helps, and it takes a minute.