reiner-dwarkesh-transcript.md
type: NewsArticle
description: Based on the transcript, this is a blackboard-style interview between Dwarkesh Patel and Reiner Pope, CEO of chip startup MatX, discussing model architecture and machine learning infrastructure. The conversation focuses on how batch size affects token cost and speed in transformer models, using a roofline analysis of a Blackwell NVL72 GPU cluster to explain inference time and cost trade-offs. Topics also include Mixture of Experts (MoE) model layout, pipeline parallelism, and the implications of reinforcement learning on model over-training beyond Chinchilla-optimal scaling.
date published: 2026-04-29T16:58:29+00:00
date modified: 2026-05-22T13:48:35+00:00
language: en
is accessible for free: true
article body: ## Timestamps (00:00:00) – How batch size affects token cost and speed (00:32:09) – How MoE models are laid out across a GPU racks (00:47:12) – How pipeline parallelism moves model layers across racks (01:03:37) – Why Ilya said, “As we now know, pipelining is not wise.” (01:18:59) – Because of RL, models may be 100x over-trained beyond Chinchilla-optimal (01:33:02) – Deducing long context memory costs from API pricing (02:04:02) – Convergent evolution between neural nets and cryptography ## Transcript ### 00:00:00 – How batch size affects token cost and speed **Dwarkesh Patel** Today, I'm interviewing [Reiner Pope](https://reiner.org/), who is the CEO of [MatX](https://matx.com/), [which is a new chip startup](https://techcrunch.com/2026/02/24/nvidia-challenger-ai-chip-startup-matx-raised-500m/). Previously, he was doing [TPU](https://en.wikipedia.org/wiki/Tensor_Processing_Unit) architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We're going to get up in a second. We in fact built this whole new studio with specifically this format in mind, so it's a pleasure to get to inaugurate it with you. We're going to be talking about [model architecture](https://huggingface.co/blog/ProCreations/the-mega-article), [ML infra](https://www.sei.cmu.edu/blog/a-hitchhikers-guide-to-ml-training-infrastructure/), and many other things. The reason I think it's an important topic is because once you understand how [training](https://www.ibm.com/think/topics/model-training) and [inference](https://cloud.google.com/discover/what-is-ai-inference) work in a cluster, a lot of things—about why AI is the way it is, why AI architectures are the way they are, why API prices are the way they are, and fundamentally why AI progress is the way it is—start making sense. You need to understand the details to get there, and you need a blackboard to understand the details. Reiner, thank you so much for doing t…
publisher: Web Pulse
keywords: artificial-intelligence, machine-learning, large-language-models, semiconductor, hardware
mentions: Reiner Pope
mentions: MatX
mentions: Google
mentions: Dwarkesh Patel
citation: CreativeWork