A series of worked performance analyses of DeepSeek-V3
If you want to learn how to efficiently train an LLM on many GPUs, you
may have already heard of resources like [How to Scale Your Model](https://jax-ml.github.io/scaling-book/)
and The Ultra-Scale Playbook, which teach you all of the foundational concepts necessary to achieve good MFU. However, it is one thing to learn out of a textbook and another to actually do the analysis in a production setting on a model you are planning to train in a month.
The purpose of this post series is to really dig into what exactly these analyses would look like for DeepSeek-V3, a historically important and influential mixture-of-experts transformer model. In my opinion, DeepSeek-V3 has stood the test of time as an unusually clean exemplar of the "platonic MoE model." It has very few anachronisms--I could count them on one hand--the three initial dense layers, use of global attention everywhere, the built-in MTP, perhaps where exactly the norms are placed. I don't think you will be poorly served from closely studying DeepSeek-V3. It also helps that [MLPerf 6.0
added DeepSeek-V3 as a large-scale pretraining benchmark.](https://mlcommons.org/2026/05/deepseek-v3-training-v6-0/) My goal for this series is that we start with a roofline model of DeepSeek-V3: an overly simplified approximation of the behavior of DeepSeek-V3 which tells us what the absolute best, "speed of light" performance we could hope to achieve on it is. And then, we progressively refine this model with more real world correction factors, until we end up with a model that is reasonably able to predict what a PyTorch profile trace would look like if you had actually run it. We should be able to understand why NVIDIA chose the performance settings they did in their MLPerf submission. We should be able to understand the tradeoff space if we're planning to try to pretrain DeepSeek-V3.
AI disclosure: The graphics and visualizations are 100% vibe coded with Fable, but I have been reviewing the widgets themselves carefully. The text can be assumed to be human-written, except when otherwise disclosed.
Posts #
An infra-oriented diagram of the DeepSeek-V3 architecture— The rest of the posts in this series will rely on a reasonably good working understanding of DeepSeek-V3. Here, we condense the architecture into a diagram that fits on a slide, that is simultaneously detailed enough to. e.g., accurately report all the parameters and matrix multiplies in the model.Memory: a Hopper case study— How to fit it on 80 GiB. We identify all of the main uses of memory in training, and then walk through the impact of pipeline parallelism, activation checkpointing and low precision on your memory usage.