{"slug": "deepseek-v3-from-roofline-to-reality", "title": "DeepSeek-V3: From Roofline to Reality", "summary": "A new technical blog series by an unnamed author provides a roofline-to-reality performance analysis of DeepSeek-V3, a mixture-of-experts transformer model recently added to MLPerf 6.0 as a large-scale pretraining benchmark. The series aims to progressively refine a simplified roofline model with real-world correction factors to predict PyTorch profile traces and understand NVIDIA's MLPerf performance settings.", "body_md": "# DeepSeek-V3: from roofline to reality\n\nA series of worked performance analyses of DeepSeek-V3\n\nIf you want to learn how to efficiently train an LLM on many GPUs, you\nmay have already heard of resources like [How to Scale Your Model](https://jax-ml.github.io/scaling-book/)\nand [The\nUltra-Scale Playbook](https://huggingface.co/spaces/nanotron/ultrascale-playbook), which teach you all of the foundational concepts\nnecessary to achieve good MFU. However, it is one thing to learn\nout of a textbook and another to actually do the analysis in a production\nsetting on a model you are planning to train in a month.\n\nThe purpose of this post series is to really dig into what exactly these\nanalyses would look like for DeepSeek-V3, a historically important and\ninfluential mixture-of-experts transformer model. In my opinion,\nDeepSeek-V3 has stood the test of time as an unusually clean exemplar of\nthe \"platonic MoE model.\" It has very few anachronisms--I could count them\non one hand--the three initial dense layers, use of global attention\neverywhere, the built-in MTP, perhaps where exactly the norms are placed.\nI don't think you will be poorly served from closely studying DeepSeek-V3.\nIt also helps that [MLPerf 6.0\nadded DeepSeek-V3 as a large-scale pretraining benchmark.](https://mlcommons.org/2026/05/deepseek-v3-training-v6-0/)\n\nMy goal for this series is that we start with a roofline model of DeepSeek-V3: an overly simplified approximation of the behavior of DeepSeek-V3 which tells us what the absolute best, \"speed of light\" performance we could hope to achieve on it is. And then, we progressively refine this model with more real world correction factors, until we end up with a model that is reasonably able to predict what a PyTorch profile trace would look like if you had actually run it. We should be able to understand why NVIDIA chose the performance settings they did in their MLPerf submission. We should be able to understand the tradeoff space if we're planning to try to pretrain DeepSeek-V3.\n\n*AI disclosure:* The graphics and visualizations are 100% vibe\ncoded with Fable, but I have been reviewing the widgets themselves\ncarefully. The text can be assumed to be human-written, except when\notherwise disclosed.\n\n## Posts\n\n[An infra-oriented diagram of the DeepSeek-V3 architecture](./studies/01-deepseek-diagram.html)— The rest of the posts in this series will rely on a reasonably good working understanding of DeepSeek-V3. Here, we condense the architecture into a diagram that fits on a slide, that is simultaneously detailed enough to. e.g., accurately report all the parameters and matrix multiplies in the model.[Memory: a Hopper case study](./studies/02-hopper-memory.html)— How to fit it on 80 GiB. We identify all of the main uses of memory in training, and then walk through the impact of pipeline parallelism, activation checkpointing and low precision on your memory usage.", "url": "https://wpnews.pro/news/deepseek-v3-from-roofline-to-reality", "canonical_source": "https://deepseek-v3.ezyang.com/index.html", "published_at": "2026-09-01 19:42:14+00:00", "updated_at": "2026-09-01 19:52:29.957396+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["DeepSeek-V3", "NVIDIA", "MLPerf", "PyTorch", "Fable"], "alternates": {"html": "https://wpnews.pro/news/deepseek-v3-from-roofline-to-reality", "markdown": "https://wpnews.pro/news/deepseek-v3-from-roofline-to-reality.md", "text": "https://wpnews.pro/news/deepseek-v3-from-roofline-to-reality.txt", "jsonld": "https://wpnews.pro/news/deepseek-v3-from-roofline-to-reality.jsonld"}}