Book Review: Build a DeepSeek Model (From Scratch) #
I was offered to read Build a DeepSeek Model (From Scratch) by Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat, and Naman Dwivedi in exchange for an honest review. The book takes the innovations behind the DeepSeek model family and rebuilds them one at a time, scaled down to consumer hardware. If you just read papers about models with MoE and various attention mechanisms, it is easy to understand what each component does, but not why it exists. More than that, it is easy to skip the details of the design and implementation, and to miss the trade-offs that led to each choice. This book covers this gap and gives you the ability to explain the design of each component rather than just its function.
The overall structure
The book follows a four-stage structure:
- Stage 1 derives the key-value cache from the autoregressive loop and covers Multi-Query and Grouped-Query Attention as the two standard ways to reduce the memory cost.
- Stage 2 replaces both halves of the transformer block: Multi-Head Latent Attention with decoupled RoPE in place of standard attention, andDeepSeek-MoE in place of the feed-forward network. - Stage 3 covers multi-token prediction andFP8 training, assembles a MiniDeepSeek and trains it on TinyStories, and closes with DualPipe parallelism. - Stage 4 goes from policy gradients up to GRPO and reinforcement learning with verifiable rewards, then knowledge distillation.
What I liked
The book usually shows a naive version of a given appraoch, then shows why it fails and builds a better version of it.
Chapter 3 is a great example: before moving to RoPE, positional encoding is developed as a series of failed attempts (raw integers, binary vectors, sinusoids). By the time you are introduced to RoPE, you already know the problems and how they can be addressed.
I liked the observation that a binary encoding is already a multi-frequency signal (low bits oscillate fast, high bits slow), and that sinusoids are simply the continuous version of the same thing. I had used RoPE for a long time without ever seeing it explained from that point of view.
Chapter 4 has a similar approach: auxiliary loss, then load-balancing loss, then a hard capacity cap, with each one shown to interfere with the main training objective before DeepSeek’s bias-term update removes the extra loss term entirely.
Shared experts are motivated through two important problems: knowledge hybridity, where too few experts force each one to be a generalist, and knowledge redundancy, where several experts separately re-learn the same English grammar. Shared experts see every token and absorb the common knowledge, so the routed experts stop suffering from it.
Chapters 3, 4, and 8 each end with a small head-to-head experiment. Chapter 3 trains MHA, MQA, GQA, and MLA with parameter counts held nearly equal, and MLA matches full multi-head attention on validation loss while holding a quarter of the KV cache. Chapter 4 does the same for a standard MoE against DeepSeek-MoE.
The book is also consistent about the limits of its own evidence. The KV cache chapter measures a fivefold speedup on an ordinary CPU laptop and points out that number is not universal. Chapter 7 has a section called “Reading DeepSeek-R1 claims carefully” that separates the strong claims (GRPO with verifiable rewards induces reasoning behavior, and dropping the value model simplifies large-scale RL) from the overclaim that verifiable rewards solve alignment.
Finally, I liked the comparison at the end of Chapter 8. The same Qwen2.5-32B base is trained two ways: large-scale RL from scratch, or a couple of epochs of supervised fine-tuning on DeepSeek-R1’s reasoning traces. The distilled version wins by more than 25 points on AIME 2024, and it wins on every other benchmark as well.
What could have been better
A few small things:
- The head-to-head experiments run on TinyStories at small scale, so the margins are narrow and the conclusions are directional.
- The architecture improvements stop at V3 and R1, and attention design has evolved toward the sparse stacks in
Conclusion
This book is a good fit for people who read LLM papers and would like to implement modern architectures from scratch. It suits engineers who already work with transformers and want implementation-level understanding of latent attention, sparse expert routing, and the post-training pipeline, and it is a solid follow-on for anyone who has built a GPT from scratch and wants to know what changed after that.