# 🎙️The Race to Build the Next Trillion-Dollar AI Chip Company > Source: > Published: 2026-09-17 19:29:02+00:00 I recently joined David Goldman, a partner at Celesta Capital, on the TechSurge podcast to talk about the race to build the next trillion-dollar AI chip company. We debated things like: *What if you could get 10x more tokens, at a fixed interactivity, for a particular model, out of your megawatts than you could buying something off the shelf?* It was a great conversation; here’s the video and transcript. *Highly recommend reading / listening / AI’ing.* *This transcript is lightly edited for clarity.* ## Why AI sells as systems **David Goldman:** Austin, welcome to the show. **Austin Lyons:** Thank you for having me. **David Goldman:** AI infrastructure is in the middle of one of the biggest capital buildouts in history, but under the surface, even though the numbers are going up, there’s been a pretty big change as the dollar spend on inference has gotten bigger than the spending on training. This is a trend that you’ve covered a lot in your writing and something you have a very unique perspective on. So I want to spend a little bit of time unpacking the implications of this shift. Let’s start at the top with how these systems get sold and what system selling is. Historically, when cloud buyers went out to the semiconductor industry, they were trying to commoditize server hardware as much as possible. They would work with industry groups, they would try to get standard designs done so that they could drive down pricing and negotiate with each part of that design. But increasingly over the last few years, they’ve been buying whole systems. So maybe let’s start at the top and just say what goes in a system, and why is this trend happening? **Austin Lyons:** Yes. I like the idea of starting with systems when we’re talking about AI, because that’s really what it’s all about these days. It’s rack scale, all the way out to full data center clusters. The compute at the heart of these systems is ultimately GPUs, and that’s what gets the airtime — or AI accelerators — but it’s actually a full system. So stepping back, the question is how did we get here, and why are people buying a full system from one vendor and not commoditizing it? The workload that we all care about today is LLM inference. If you’re running at the frontier and you’re wanting to use the latest and greatest OpenAI model or Anthropic model, these are models that are two trillion or more parameters. What that means is, even as you quantize these and try to run them without using as much memory as possible, you might still need something like a terabyte, or two terabytes, or several terabytes of memory just for the model. Because with these models, as we saw with the scaling laws over late 2022, 2023, 2024 — the bigger the models and the more compute you have, the better answers you get. Way back in the day, we might have taken an AI model and put it on just one GPU. But if you think about today’s frontier models, you need maybe terabytes of memory, and yet a single GPU can only have, for example, 288 gigabytes of HBM on it to store these model weights. So quickly you say, wait a minute, one model can’t fit in a single GPU’s memory. So what do you have to do? You need several GPUs. You might say, okay, take that model and split it across four GPUs or eight GPUs. And that’s how we started to work our way into “this isn’t just a single chip anymore, this is a whole system.” That’s how we got to, for example, the Grace Blackwell NVL72 rack — a rack with 72 GPUs in it. We’ve gotten to a place where we’re talking about AI systems even just for inference. Even if you were only to have one rack, you’ve got GPUs, CPUs, networking, power, and cooling. So back to the question of why the industry isn’t commoditizing it: a big piece of where we’re at today is that Nvidia did a really good job of getting to this rack scale first. They said, hey, we can help design the whole system and make sure it works across 72 GPUs and 36 CPUs, and that they can all talk to each other. And by the way, it also takes software and compilers so that you, the model developer, can write your model and actually run it across all this. So even though the industry always wants commoditization, competition, and multiple suppliers, Nvidia did a good job of frankly getting there first and making this complicated system turnkey, so that for the software developer it’s as easy as writing their model, deploying it, and it “ just works”. **David Goldman:** So you need a very big, complex system to make these LLMs work in a cloud. But if I’m a cloud buyer — Nvidia has famously high margins, and they charge that on all of the different parts of the system, not just on the GPU. If you go out in the Valley, there are all sorts of companies offering one piece of this puzzle. If I’m a cloud buyer, why would I be willing to give so much margin to Nvidia instead of trying to piece each piece together — buy my GPU from Nvidia maybe, my networking from someone else, my software from someone else, my cooling from someone else? What kind of value do you get from getting it all at once? Is it around speed? Is it around simplicity? What are the trade-offs that people think about? **Austin Lyons:** Yeah. We are not yet in the era where people are trying to squeeze down their costs as much. We’ve still been in the era of speed to market. We saw that with Elon Musk and xAI, where he’s been able to stand up data centers very quickly using Nvidia systems. I don’t think Elon’s goal was to say “how can we do this as cheaply as possible,” but more “how can we stand up this compute and get productive tokens out of it as quickly as possible?” So even though a company can potentially pull the components from many different vendors, there’s just a desire to buy it and stand it up quickly. And to your other point, there’s definitely simplicity there. We see even AMD, with their Helios rack that they’re bringing to market, seeing that same pull from customers: I want to be able to buy the full rack, get it, stand it up as quickly as possible, and get software running as quickly as possible. **David Goldman:** Each of Nvidia and AMD have had decades to put together this puzzle, and tons of financial firepower to do M&A. Nvidia acquired Mellanox; AMD has done a bunch of acquisitions to build out these system-selling strategies. Is it possible for a startup to compete now that you have to sell systems? **Austin Lyons:** That is a tough and very interesting question. To your point, if you’re a startup now, you can’t necessarily just come in and say “I’m going to build a better GPU, a better AI ASIC.” People are going to say, okay, great — now what do I do with that? Are you going to make your end customers piece together the whole system? We just said they want to stand it up and deploy it as quickly as possible. Or are you, as the startup, now going to have to take on building the rest of the system? I think there are probably different approaches here. And by the way, this is why we see that it takes hundreds of millions of dollars now for a chip startup, when maybe back in the day you used to do several rounds of just a couple million dollars to prove out your little proof of concept. So can a startup compete here? Yes, but they have to be very smart. It could also be death by a thousand paper cuts if you’re trying to invent a new AI accelerator and you’re also trying to invent your own proprietary networking and maybe your own proprietary cooling. Actually, for example, Cerebras took a very interesting approach. They’re an AI accelerator startup — they’ve gone public — and they make what they call wafer-scale engines. For those listeners who aren’t as familiar: instead of taking a whole wafer of chips and dicing them up into individual chips, like two GPUs and a CPU that you package, they said, why don’t we just leave it at the wafer scale and make the whole wafer our compute engine? But then they had to invent how you make them all communicate, how you cool all of that, how you deliver power to all of that. They basically had to reinvent all of it, and that takes a lot of time. So I think any other AI ASIC company that followed could look at an example like Cerebras and say: wow, they did very innovative technical things here, but they had to invent everything, and that’s time-consuming and expensive. How should we do this differently? **David Goldman:** Yeah. I think you see some parallels with the buildout of internet infrastructure in the ‘90s and early 2000s. You start with proprietary, then as open standards proliferate, there’s more capability for more people to offer things, and you can get a little bit more mix and match. I know you’ve seen SambaNova, for example, has partnered with Intel to offer x86 CPUs alongside their system. So there seem to be some trends, even though incumbents clearly have an advantage in system selling. ## Prefill, decode, and the split inference workload **David Goldman:** One of those trends, which I think you’ve touched on a lot in your writing: as we move into this inference era, you’re starting to see the inference workload itself get split, into what’s called prefill and decode. Can you tell us what those two things are, and why you might want to split that workload onto different types of silicon? **Austin Lyons:** Sure. At the end of the day, if you just think of simply talking to an AI chatbot — you write a paragraph of explanation, “hey, I want you to go research this and give me an answer” — the first thing the AI system, the model under the hood, needs to do is read through the whole prompt that you gave it. That’s the prefill stage. What’s happening is, in parallel — one of the key innovations of the transformer model is this idea of attention — you work through everything in the long English paragraph, and you ask which of these words are connected to other words in the sentence. That way you can piece together the context of the whole paragraph and what each word is referring to. And you can do all of this in parallel. If there are a hundred words, you can look at all hundred words in parallel and compare them to the other hundred words and do these calculations to figure out whether they’re related or not. That’s prefill. You’re ultimately doing a bunch of linear algebra, a bunch of matrix multiplication. Now, when it’s time to give you an answer — especially if you remember back to when ChatGPT first launched and was a lot slower — you’d see the answer come out word by word. That’s because every word that I’m about to say depends on the words that I just said. And when I say that word, the next word also depends on those other words. This is decode. This is where you’re predicting the next tokens, or the next words if we’re talking about English. And that is sequential. You can’t do it in parallel. People running this inference at scale started to realize: hey, wait a minute. In prefill, I’ve got all these GPUs doing all of this parallel computation, and it actually doesn’t require a huge amount of memory. I’ve paid for this high-bandwidth memory — which is very expensive, and getting more expensive by the day — and in prefill, that HBM is just sitting there being underutilized. Interesting. Okay, now I’ve got these other GPUs that are running decode, and they’re actually not fully utilizing all of their FLOPS, all their units of compute, because decode is what’s called memory-bound. It’s a lot of “I made a prediction, and I need to get the weights and whatever I need from the KV cache, then do a little calculation, then go back and forth and back and forth,” one at a time. You’re not doing things in parallel; you’re really just waiting on memory. So at the highest level — even Nvidia led the way with this — they said: we’ve got a bunch of GPUs in the prefill phase being heavily utilized for compute with their memory underutilized, and a bunch of GPUs in the decode phase basically underutilizing their compute and totally utilizing their memory. Naturally, any engineer is going to look at that and say: huh, maybe we should disaggregate these. Maybe prefill should run on systems that have lots of compute but don’t necessarily need all that memory. And maybe on the decode side, we should really emphasize memory bandwidth — how quickly information can get shuttled around — and maybe it doesn’t even need quite as much compute; we should essentially overindex on the memory part. So naturally, we started to move into a world where Nvidia shipped a software layer for their systems called Dynamo, which helps orchestrate this across Nvidia GPUs. And that actually gave rise to the Groqs and the Cerebrases — these AI ASIC startups that were actually started even before transformer-based LLMs were the defining workload of our era. They had made architectural choices where they used a lot of this really fast on-chip memory called SRAM — where you use transistors to store the memory, instead of DRAM, which is capacitors and transistors, which is what HBM is made of. We won’t go way down into those memory details, but basically these early startups had made a bet on having really high memory bandwidth. And once the workload got separated into prefill and decode, they could raise their hand and go: oh wait, we’re actually really good at decode. In fact, we can go even faster than GPUs. This took us from the inference era where everything was on GPUs to saying: what if you could slot in one of these AI ASICs, heavily built on SRAM, that can maybe unlock a thousand tokens a second? Whereas a GPU running decode — just the way GPUs are more general-purpose in their design and their memory hierarchy decisions — maybe could only run at a fraction of that. **David Goldman:** Is that something you should always do? Do we always need to split up prefill and decode? Or are there just certain applications where it’s really good to have speed — I’m willing to pay a premium for speed, therefore I’m willing to go through the hassle of splitting these things up, having orchestration software, having different types of silicon and all the things associated with that — versus just running it on the GPUs or the system I’ve already bought? **Austin Lyons:** Yeah, there are so many different nuances here. Take the hyperscalers deploying tens of billions of dollars of GPUs. Early on, GPUs are very general-purpose, very flexible — they give you the freedom to change your workloads. But let’s say you’re OpenAI and you say: no, these are our specific models that we know we’re iterating on — the frontier one, the medium-sized one, the small one. You start to say, hey, we should really cater to this workload’s needs, and therefore it would make sense to deal with the complexities you pointed out — splitting up prefill and decode and orchestrating that. It might be worth it. Maybe that unlocks, for example, being able to sell an ultra-premium tier where you get really, really fast inference, and maybe there’s a small subset of users who would pay 10x more to get tokens that are 5x faster. So I think there are certain model labs and hyperscalers at scale that are saying the complexity is totally worth it — we’re willing to deal with different SKUs, different chips. On the other hand, I think there are going to be tons of enterprises, and arguably the long tail of consumers, who of course want inference, and they’re thinking about cost and speed, but they aren’t going to want to manage all of that complexity. For them, where we’re at today, they’re not going to need it. Of course, everyone always wants faster inference — so if there’s a way for them to get that outcome without dealing with all the complexity, they’re going to want it. When I’m trying to think about this space and ask whether it’s one-size-fits-all GPU or many different chips — where is this going to go in the end — I look to CPUs. When you look at any CPU vendor, or any cloud like Google Cloud, they don’t just deploy one CPU, even for customers who want to rent them from their cloud. They have a portfolio: hey, this one has a lot of memory in case you’re running a database; oh, this one’s actually pretty vanilla and it’s nice and cheap if you’re just running an API server. There are still shapes — this family of chip, that family of chip. I do think there’s a world where we get to a couple of different shapes of AI accelerators. Where we are now: we had this training era, then this inference era — those were all on GPUs. Then we’ve got this next era, which is a GPU plus a specific decode chip that’s made a good memory-hierarchy trade-off so you can have high interactivity, as they call it — really fast tokens. But where we are today, those are multi-vendor: Nvidia plus Groq — of course, Nvidia acquired Groq, so they’re trying to bring that all in-house — or someone else plus Cerebras — AMD plus Cerebras, or Trainium plus Cerebras, or whatever. My thought is that we will ultimately go to those SKUs living inside the same silicon vendor, because it starts to get complicated when your route to market depends on another company — I sell a GPU and they sell a decode-specific thing. But it is working right now. There’s definitely demand for it. **David Goldman:** It feels like there’s this inherent tension in the market. You want to buy systems — that’s what customers are saying, they want someone to do the work of putting all of these things together. But at the same time, they also want the right silicon for the right job. They want specialized things for decode, and they want the right proportion of CPUs for the workload. So it’s not a one-size-fits-all system. Ultimately, do you think you get more and more fragmentation here? Or are customers going to say: “Nvidia, please solve this problem for me — buy Groq, buy the next company, buy the next company, keep selling me systems. AMD — buy Talis [sp?], work with Cerebras, figure it out for me.” Because, to give you the counter side: there are separate companies for CPUs and GPUs. We’ve decided those are separate enough markets that they can have standalone companies. We can figure out the system with CPUs and GPUs from different vendors — that’s been a solved problem for a long time. So maybe that could be an end state, where you have decode silicon that is a completely different market, and we give it a catchy name like DPU or something. Well, not that one, because it’s already been used. **Austin Lyons:** Right, totally. Zooming way out, when you look at the semiconductor industry in the long run, there always seem to be three or four vendors in a certain thing. Whether you look at wafer fab equipment, foundries — it’s complicated, but we’re getting back to maybe having two or three — CPU vendors, GPU vendors. We are of course in an era, as happens whenever there’s drastic innovation, where a ton of competitors have popped up. So I think it would not be crazy to zoom out and say: in the grand arc, maybe only three or four people will shake out, and therefore there’ll be some sort of consolidation. But I personally think this isn’t as simple as “Nvidia buys [that startup], AMD buys them,” even though we’re seeing some of that — and of course there are regulatory things to talk about there. If you look up and down the stack, there’s actually a lot of competition. For example, at the neocloud layer: a neocloud might be incentivized to say, hey, I can tell the customer wants the right silicon for the right job, but they also don’t want to deal with the complexity. I could buy a bunch of different silicon, deal with all the complexity myself, sell them tokens as a service, and try to give them the speed or the cost they’re looking for — and differentiate from other neoclouds that way. So I do think there are still routes to market where startups can come in today and say: hey, I am the best decode solution, you should try me. Or maybe I’m a prefill solution. And figure out a way to get to customers while letting the end developer not deal with all that complexity. Then you keep pulling on that thread and play it forward: could an AI ASIC startup ever merge with a neocloud? I don’t know. Maybe it won’t just be Nvidia or AMD buying all of these companies. ## The rise of the neoclouds **David Goldman:** You touch on an interesting thing here, because I think neoclouds are really under-discussed when people talk about AI infrastructure. We tend to focus a lot on semiconductor companies and systems companies, and not as much on the cloud layer. And I sort of have a pet theory that it’s because most of the traditional VCs missed out on those as investments, and so we don’t like to spend too much time giving credit where we don’t get to claim any. **Austin Lyons:** Yeah. **David Goldman:** But the reality is that if you look at the handful of neoclouds that came up in this first wave, they’ve created a lot more equity value. The top three are worth something like $125 billion in public markets, which is a lot more than what we’ve been talking about with some of these chip startups. So why do you think investors had a lot of trouble understanding the first wave of neoclouds and missed out on those investments? **Austin Lyons:** Yeah. For listeners, by “neocloud” we mean a cloud company that started by just renting GPUs. I’ll talk through my hesitation when I first saw the idea of neoclouds, and maybe I’m a fair proxy for some investors — maybe they thought this way too. Let’s back up. Who’s renting from these neoclouds? Well, it turns out a lot of it is hyperscalers, which is ultimately driven by demand from the biggest model labs. So the question is: wait a minute, you’re saying that OpenAI is running on GPUs that Microsoft is renting from some neocloud? Doesn’t Microsoft just have their own data centers? And the answer is: of course they do, and they’re trying to build more, but at the end of the day we are limited by access to power. There are financial reasons to rent versus buy — maybe even go off balance sheet as the capex has increased year over year over year. So there are very legitimate reasons why even someone as sophisticated as Microsoft Azure might say: actually, I want to rent GPUs from someone, and I’m also building other data centers, but maybe it’s a stopgap. Looking at that, I said: okay, so you’re going to have a neocloud company — maybe they have access to power, and maybe they have a good way to raise capital against assets like this. Bitcoin miners, for example, historically have had this experience, and maybe they’re shifting into GPUs. And so I thought: huh, okay. They were doing Bitcoin mining, now GPUs are hot, so they’re going to do GPUs, but they’re just going to rent them to Azure. But Azure is also standing up their own data centers. So how is this sustainable? They’ve got huge customer concentration — literally maybe one customer. But to their credit, that’s how you get the financing: you say, I’ve got Microsoft who’s going to rent these from me, and so people will lend against that and believe in that. That’s part of why I missed it as these companies were popping up. There’s a legitimate need in the marketplace for people who have access to power, who can get financing, who can manage this, and who can move quickly — to stand up GPUs, run models on them, and rent it out as bare metal or maybe at a higher level of abstraction. And even though they have serious customer concentration risk, so does everyone else in the semiconductor industry right now. Who are Nvidia’s end customers? Even Nvidia, the biggest and best, is selling a lot of GPUs to a small set of customers. **David Goldman:** Yeah, it’s not uncommon to see someone go public with 90% customer concentration these days. **Austin Lyons:** Totally. That is the name of the game. Look at any component supplier — even in the interconnect space or the switching space. Look at Credo, who made these active electrical cables, a really awesome invention. Same thing: even when they went public, they were selling to a handful of customers. The name of the game for this era is watching how this unfolds and seeing how some of the people who win one big customer ultimately win a couple more, build out more of a portfolio, and reduce a little bit of that risk. But that’s just the way the industry works right now. **David Goldman:** If you wind the clock back five years, before these neoclouds got big — AWS is an incredible business; Amazon and Microsoft have fortress balance sheets, great relationships with Nvidia and all the semiconductor companies, expertise in setting up data centers, software, customer relationships. It seems to me they could have done this, and certainly most people thought they would, which is why so many missed out on these investments. Was it a strategic decision, where they thought “maybe this isn’t going to be a big enough market, I’m not sure I want to spend all the money and take the risk”? Or was there some special sauce in what these neoclouds were able to do — setting up quicker, being more creative in financing, converting old Bitcoin data centers? Were they doing something different from what Microsoft might have done, or was Microsoft ceding market share to them, or some other factor? Not to pick on Microsoft — insert anyone here. **Austin Lyons:** Maybe a little bit of everything. At the end of the day, this is risky business, because the investments might be $40 billion this year, $80 billion next year, a hundred billion the year after that. I don’t think it’s that the existing players didn’t believe the future we’re in now would manifest — I think it’s all about timing. And they’re probably not incentivized to just sprint out and stand up all these data centers. It’s a huge investment. The neoclouds were able to move faster and take on more risk. They can kind of go for broke: all right, cool, let’s convert, let’s buy a bunch of GPUs, let’s get moving. It’s a bit like the innovator’s dilemma. There are already three big clouds, they already have all the customers, and I think they were probably looking at each other — AWS and Google and Azure — saying: are we all believing this future is coming and investing in GPUs at the same rate as each other? But that could still not be enough supply to meet demand. Now, I do think demand went higher faster than everyone expected, especially once we got reasoning models, and now of course the agentic age. So even if they all looked around and said, “we think if we grow supply like this — it’s a little risky and these numbers feel really big, but we think the demand will be there” — demand skyrocketed past that, and it gave neoclouds an opportunity to come in and say: yeah, we’ll fill that gap. **David Goldman:** I think there was also maybe a difference in interest level. You heard some of the hyperscalers make comments about how these bare-metal GPU instances are low-margin and sort of commodity, and therefore they didn’t want to support them as much. Whereas the neoclouds read them as revenue, and therefore good. **Austin Lyons:** Yeah, you make a fair point. Anyone whose business was renting CPUs, or selling services on top of CPUs — the cost structure there is much better than GPUs. So you could see CFOs saying: wait a minute, we’re going to spend a ton of money and our margins are going to go down, even if our margin dollars go up. There are still conversations to be had that might make you slow down or hesitate a little bit. **David Goldman:** So is this a durable state of affairs? Obviously this is a very fast-moving market, but “first mover,” “more willing to take risk,” “willing to take on lower margin” — those are not necessarily durable advantages that will last a decade or more. What do you see playing out with these neoclouds? Do they get acquired by hyperscalers? Do they consolidate into a neo-hyperscaler or something? **Austin Lyons:** Yeah, it’s very interesting. I’m not sure there will always be a need for a hundred neoclouds, but I do think it’s real demand, and it will remain real. I don’t think the big three clouds will always meet everyone’s needs indefinitely. Not only that — there’s always going to be innovation. Take World Labs, Fei-Fei Li’s company. They’re coming out with world models. Who’s going to make the bet there? What if those don’t run well on the exact shape of hardware all these big clouds have invested in? I think there will always be new workloads or new demands popping into existence where a neocloud can pop up quickly and say: I can meet your need, and I can innovate there. That will always exist. There may not be enough cutting-edge frontier demand for a hundred neoclouds to hop on, but I definitely think there will always be a need for these more nimble, smaller GPU and AI ASIC rental companies that can innovate a lot closer to where the model labs and AI-enabled software companies are innovating. ## Circular financing **David Goldman:** I feel like we can’t talk about neoclouds without at least discussing circular financing risk. This is probably the thing I hear most from people who are skeptical about the durability of AI infrastructure. The argument goes: the neoclouds get equity investment from Nvidia in many cases. They use that equity to buy GPUs. They then use those GPUs as collateral to take on debt. A lot of that debt is sometimes backstopped by either their hyperscaler or by Nvidia itself. It all sort of perpetuates, and the revenue from the neocloud buying GPUs goes back to Nvidia, who invests in more neoclouds. People see a circular financing issue, like the vendor financing that happened in the dot-com bubble and was revealed to be an inflator of that bubble. On their last earnings call, Nvidia addressed this directly and said they see it differently. Do you agree with them, or do you think there are some concerns here? **Austin Lyons:** You know, I was definitely the type of person where, right away, I said: okay, this is different, this feels funny, I need to dig in and understand it and try to understand both sides. I can definitely see why it looks like circular financing — call it what you will. But going back to thinking through demand, supply, and the cost of capital: it is a fact that there’s just insatiable demand, especially now with agentic AI, where the barrier to entry for software development has gone as close to zero as possible. I’ve got a son who made a 70,000-line video game this summer, and he didn’t write those lines himself — he used Codex to do it. That’s amazing, it’s unreal. And just wait until he’s in high school and college and beyond. He’s going to use AI so much more intelligently than me. He’s going to use way more tokens than me. I can’t even believe what it’s going to be like in the future. You can look at every industry and see lots of people writing software, doing interesting things they couldn’t do before. So the demand is totally real. The supply is very fixed. At the end of the day, you might look at someone like TSMC — there are only so many wafers that come out, only so much CoWoS capacity. But even as GPUs get built, the question is who has the capital to buy them. Today we might be talking $5–10 million a rack, or more. Who has that kind of money lying around? So there is this cost-of-capital, financing thing that comes into play. If customers are saying “I just want inference, as fast as possible, as soon as possible, please make it happen” — who in the supply chain has the money to invest in standing up all these data centers and running them? It could be neoclouds. Okay — do they have access to the capital they need to make tens of billions of dollars of investment, or even a few billion? A lot of these might be early companies, or former Bitcoin miners. They might have some access to capital, but not enough. If you’re Nvidia, sitting there with all this money as the world’s biggest company, and you’re saying demand is incredible and supply is what it is — but it’s not just building the GPUs, it’s getting them powered up, financed, stood up — of course it makes sense from their perspective to help get this stood up. The banks aren’t so sure they want to lend to the neoclouds, but if Microsoft or Google or AWS says “I’ll be the offtaker — don’t think about the neocloud when you’re lending, think about me,” and if Nvidia can also come in and say “we want to help make this happen, can we put our brand behind it, backstop it, whatever” — I can see why Nvidia would want to do that. Now, does that mean they also benefit from it? Of course, totally. It’s customers, and maybe even revenue sharing — a new source of revenue for Nvidia. **David Goldman:** Well, that’s capitalism. They’re not going to do it if they don’t benefit. **Austin Lyons:** Exactly. But as sort of a techno-optimist: if it’s going to feel funny, but there are ways to get more compute stood up faster so that more people around the world can do the awesome things they’re trying to do, then I’d say, all right, I can get behind that. **David Goldman:** I think it ultimately boils down to differences of opinion on the durability of the cash flow that comes from these assets. If you went out and said “I’m going to build a toll road,” you can get a lot of financing for that. You don’t have to put in a lot of equity, and you can get a lot of debt, because people know this road is going to have X number of cars, we know the traffic patterns, you’ll collect this amount of money. It’s very safe. You can raise lots of debt at very low rates for projects like that, even if you’re a new company. Obviously there’s a big difference between a toll road and a GPU-based AI factory, as some people are calling them. But from Nvidia’s perspective, and from some of the hyperscalers’ perspectives, these are fairly safe assets that will be completely paid back in two or three years, and then there’s a stream of cash flows coming out of them. Even if demand goes down a little or doesn’t grow at the same rate, you’ll still get value out of them, and they’re not going to depreciate super quickly. Nvidia has that view and is willing to put their balance sheet behind it, the hyperscalers feel the same way — and not everyone else has the same view. That’s where the rubber is going to meet the road. The flip side, though, is that you have the existing neocloud set who are now very embroiled with Nvidia — they’re in lockstep, they rely on them for financing. To the extent that customers are going to demand more and different silicon, that creates an opportunity for new neoclouds who can figure out how to make it all work together and choose the right silicon for what customers want. Do you think that’s effective counter-positioning? Will we see another wave of neoclouds like the first time, or will the existing set — the CoreWeaves, the Nebiuses, companies like that — figure this out and just start using Nvidia plus Groq, or AMD plus whoever? **Austin Lyons:** Right. I think both will exist, but you make a good point about the trade-off. It’s kind of like golden handcuffs. The trade-off for a neocloud is: hey, they’re backstopped by Nvidia, maybe they got financing, and they’re obviously getting allocation for GPUs. So those particular neoclouds might feel like, if there’s different silicon out there that’s very competitive, even for a subset of workloads, we may not feel like we can go buy it and offer it — because what if we don’t get as much allocation in the future? Don’t bite the hand that feeds you. So I do think there will be opportunities for other neoclouds to come in and say: we’ve got a bunch of different silicon, and maybe we can abstract it and run your workloads across it so you don’t need to worry about it. There will be opportunities for someone to come in and counter-position. ## Four conditions for the next trillion-dollar chip company **David Goldman:** In the last couple of years there have really been two big outcomes — we’ve touched on both — in the semiconductor space: Groq, which sold to Nvidia, and Cerebras, which went public. But neither one really won a hyperscaler before they were able to do this, and in the case of Groq, they sold to Nvidia. You wrote an article about what you think the conditions are for the next trillion-dollar chip company, and you had four conditions: capable of running trillion-plus-parameter models; rack-scale chips, which we’ve touched on; beating an incumbent on a KPI; and landing a frontier anchor. Points one and four — running trillion-parameter models and landing a frontier anchor — are correlated around the idea that the frontier matters most. Why do you think that’s the case, and a prerequisite to being the next big breakout company in silicon? **Austin Lyons:** Very interesting question. Ultimately, today, it’s the best models — especially if you can run them at fast enough speeds — where my belief is the outsized value will accrue. Yes, there are lots of use cases where you can use older models, smaller models, and you don’t have to run them as fast. I think that pie will continue to expand. I just don’t think people will pay a premium for it. So if you’re an AI accelerator company and you’re trying to put as much muscle behind a few arrows as possible, you’d want to compete at the frontier. Today it’s software developers saying: yes, I will pay — not $200 a month; we’ll pay for tokens, thousands of dollars a month, tens of thousands a month — if we can get it fast, if we can get Claude Fable, for example, or the latest OpenAI model. And of course the frontier will always keep getting better. It also feels like the frontier is where there will be the least competition, because GPUs, for example, can’t get there today on speed, and we know that. If you’re aiming at a 70-billion-parameter Llama 3 and going after all those workloads that are valuable but don’t need the highest intelligence, there’s going to be a lot of competition there too — and it could literally be old Hoppers or old Amperes from Nvidia. When you look at that Pareto frontier curve we always see — I’ll describe it for people who are just listening: the x-axis is interactivity, which is how fast the tokens come, tokens per second per user; the y-axis is throughput. The slower you go, the more tokens you can generate concurrently; but way out there on the far right, going fast, even if you can’t serve as many users — that’s where the value is accruing today. And it’s hard to see a world where that changes. **David Goldman:** That graph gets shown a lot, particularly when Jensen or people from Nvidia talk about what they’re going to be able to do with Groq plus Nvidia. But is tokens per second per user — that interactivity KPI — still the right one for startups to think about? Or are there changing needs because of power constraints, cost constraints, new workloads like agentic coding? Is it still all about speed? **Austin Lyons:** It’s not all about speed. In my opinion, the way to compare people is at a fixed interactivity, for a given unit of power. We are power-constrained. Ultimately, if a neocloud gets access to 100 megawatts, they’re going to have to ask themselves: how can I get as much revenue as possible out of this 100 megawatts? They might say, I want to bet on allocating some of my megawatts to really fast tokens, because I think we can charge more for them. So if you’re going way to the right on the interactivity curve — let’s say they’re aiming for 800 tokens per second or higher, because they feel they can charge a premium for that, and they’ve only got so many megawatts — then they’re going to ask: how many concurrent users can I serve? What is my token throughput? So I think it’s token throughput at a fixed interactivity, normalized by power. But again, not every workload needs that. **David Goldman:** I didn’t hear you say the word “cost.” One of the ways you get better interactivity is by using more expensive memory, using SRAM. So how much does cost play into that equation? **Austin Lyons:** Cost plays into it. If you’re in that use case — you’re a neocloud, you’ve got 100 megawatts, you’re trying to generate as many tokens at a fixed interactivity as you can — you also have a fixed budget to spend on compute. If you buy an Nvidia Vera Rubin rack, the latest and greatest, plus nine accompanying Groq LPU racks, you might get really high on that interactivity, and it might be pretty good power-normalized — but you might have spent half your budget, or all of your budget, right there. So cost absolutely comes into play. When I’m thinking about the user experience, I’m thinking about interactivity and how many people can be served. But if you’re a neocloud or any buyer of compute, you’re definitely thinking about cost. And cost could mean: I can get the same performance out of two racks from this vendor versus eleven racks from that one. Or, for the same fixed cost, what if I could get eleven racks from this new competitor — and therefore five or ten times more tokens at that interactivity? ## Enterprise and on-prem AI **David Goldman:** This kind of buyer thinking is very emblematic of a cloud or hyperscaler with a huge instance they’re trying to spread over lots of users. I personally always struggle with holding two ideas in my head at the same time. On the one hand, all of the initial demand and value has been going to the frontier labs, served by a combination of hyperscalers and neoclouds. On the other hand, we and many other people believe AI is going to be something as big as the internet. It’s going to diffuse into businesses all over the world. Every company is going to have some AI element, in the same way every company has a website now — there are no more dot-com versions of companies; everyone’s got a website, everyone’s got an app. Soon everyone will have some AI element in their business. And in today’s internet world, more workloads exist on-prem than in the cloud. If I’m a coffee shop, I might have a server in my coffee shop; I’m probably not going to have an AWS account. So ultimately, does enterprise actually become the big market here instead of cloud? **Austin Lyons:** That is a very good question. I think they’re both going to be massive. Let’s talk enterprise, because I don’t think people appreciate that enough. Even Nvidia is trying to get ahead of it — they changed their reporting and their business units so that for data center it’s essentially hyperscaler and non-hyperscaler. It gets a little fuzzy, but they’re saying that, by the way, the non-hyperscaler segment is growing faster than the hyperscaler one, and right now the revenue is pretty close on both — a little fuzzy because they put neoclouds in there. So the question is: what workloads are going to go to the enterprise, and why? You’re totally right that there is going to be diffusion of generative AI across every industry, and I definitely don’t think we’re there yet. Marc Andreessen said fifteen years ago that software is eating the world — every company is going to be a software company. And to some extent that was right: logistics, manufacturing, healthcare — they’re all using software, even if it’s just internal tools. I think we’re going to a world where agentic AI eats the world, and every company is going to be an agentic AI company. Like I said, pointing at my son: fast-forward fifteen years, and yes, they’re all going to be agentic. Okay — so if agentic AI is core to how businesses run, are they all going to have hundreds of millions of dollars to spend on tokens every year? Totally not. There are going to be all sorts of reasons for companies to want to deploy workloads on premises. One: cost. Two: owning your own data, and figuring out how you even differentiate in a world like that. It won’t be all-or-nothing — go to the cloud for the frontier workloads, and do as much as you can of the older, smaller models on premises. By the way, you could also read Nvidia’s hyperscaler/non-hyperscaler split as a proxy for frontier closed models versus open-source models, because if you’re running enterprise AI locally today — on premises, on a server, on your desktop — it’s got to be an open-source model. That’s a little bit why we’re in the world we’re in today: if you want the best model, you have to go to the cloud. And for everything else — do you want to pay for tokens, or for token generators? I think a lot of people, if they can afford it, would rather own token generators. But fast-forward a little: what happens if open-source frontier models can keep up, or be good enough? I think there will continue to be a rise in the amount of workloads run on premises. There are all sorts of reasons. Of course you can point to regulated industries that will have to run on premises. But even if I’m not regulated: maybe I think the labeled data my humans generate is valuable. Say I’m an insurance company. We have all this agentic stuff doing claims processing, and my humans are going in and correcting it — that was good, that was good, that was wrong. Let’s keep that data internally and fine-tune our own model so it gets it right in the future. Maybe I want to run that locally because I’m in charge of the model, I’m in charge of the data, I keep it, it’s all my intellectual property. And maybe I see that as how I differentiate in the future — I’ve got better agents than my insurance competitor. Can you do all this stuff in the cloud and feel like it’s secure? You totally can. But at the end of the day, when we’re talking about diffusion, we want every engineer at every company to be able to tinker with it, touch it, play with it, use it themselves. Sometimes when stuff’s in the cloud, you get the convenience, but you lose the ability to get under the hood — whether it’s a closed model or even an open model in the cloud. **David Goldman:** It’s funny you say that — I almost feel the opposite way about enterprise. To me it seems gated a little bit by software. So many of these companies haven’t moved workloads even to the cloud, for data sovereignty, regulatory, privacy, IP protection reasons. They’re very concerned about stuff leaving their IT premises, and they would gladly do more in AI, but they don’t have engineers in-house who know how to post-train a model or tinker with this stuff. Those people are expensive, and they don’t want to hire them. So they want to do it on-prem in the enterprise, but no one’s quite figured out how to help them do that yet. That, to me, feels like the missing piece that would unlock a lot of enterprise hardware sales. **Austin Lyons:** I definitely agree with you — that’s where we are today. Back up eight years: everyone wanted to be a software company, but they didn’t have software engineers. I live in Iowa, so if you were a software engineer there and willing to work in insurance or ag or retail, you were a rock star. You could walk in and they’d say: yes, thank you, we need you, we didn’t have this capability before. Fast-forward to now, and anyone can vibe-code, which is actually pretty awesome, because now the domain experts — the person in insurance who knows insurance really well — can actually build the solution they want. I think we’re now where we were eight years ago, but for fine-tuning: I’m an insurance expert and I can vibe-code a thing, and I’ve got some software people here, but none of us know how to fine-tune yet. That’s the education piece. If my children had to go to college today and pick a major, I’d say: pick anything, plus machine learning, and learn how to fine-tune, because then you can understand a domain and also understand how to actually apply AI. So I think there will probably be a rise of AI engineers, if you will. Maybe agents will do this for you and bring that cost down to zero faster than agents brought software down to zero — which took, you know, forty years. But it is a pain point today that companies don’t have generative AI familiarity yet. I don’t think that pain point will be there forever. Maybe it’s five years, maybe more, I don’t know. But I don’t think it will always be a blocker. If every company became a software company with software-literate people on staff, I think eventually everyone will have LLM-fine-tuning-literate people on staff too. **David Goldman:** So when that day comes, is this just a huge unlock for Nvidia, and they get that much more revenue? Or do you think anyone else has a chance at that market? **Austin Lyons:** That’s a great question. What happened to IBM, you know? There are giants, and they’re first, and they ride a huge wave. And then, to all the points we’ve talked about — zooming out, there are three or four winners, and people want competition. The more people can tinker, the more this diffuses, the more opportunity there is. One company cannot meet everyone’s needs at the right price point, at the right speed. They can meet lots of people’s needs, but there will always be people trying to do some interesting bespoke thing who say the off-the-shelf stuff uses too much power — I know I’ve got this crazy setup, but I can’t do 130 kilowatts, I can only do 50. There will always be emerging workloads where the off-the-shelf stuff just doesn’t fit. And the tough part is, when you’re Nvidia and you’ve got these huge hyperscalers, you’re not necessarily incentivized to go find those little people. You’re not interested in picking up pennies; you’re interested in picking up billion-dollar bills. So I think Nvidia will be totally fine — they have great solutions, and they’re always going to have customers coming to them for the latest and greatest, deployed quickly. But I do think there will continue to be new opportunities popping up, especially as this diffuses, where people can compete. ## Clean-sheet silicon for LLMs **David Goldman:** Circling back to this idea of the next trillion-dollar company. There’s an explosion of opportunities in different workloads, and also, as we’re hearing, different markets. Does that mean there’s an opportunity for a trillion-dollar company? Or are we going to get $500 billion companies? AMD is still not even a trillion-dollar company, and they’ve been around for a really long time and have a lot of pieces of this puzzle. **Austin Lyons:** I just think about the size of the market. Here’s the deal, and part of why I came to that conclusion. GPUs obviously have a history in graphics and doing things in parallel, and that has been shifting toward AI-centric. Nvidia is saying: yes, these data center GPUs — you’re not going to run Doom on them. They used to support FP64, and they still do, but we’re going to spend all of our transistors, as we do a node shrink, on FP4 and FP8, the lower precision that AI models really want. But at the same time, it was still a general-purpose GPU. That’s why I said we went from training with GPUs, to inference with GPUs, to this next era where LLMs are *the* workload — and we still haven’t had silicon designed specifically for LLMs. We had GPUs that morphed from their early roots to fit the shape of what we’re doing. Let’s take a GPU and slap on this SRAM thing. But no one yet has really brought a chip to market that was designed specifically for LLMs. So the question is: what if you can be the first one that can stand up a gigawatt’s worth? Which is not simple. **David Goldman:** Well, to some degree, the Nvidia GPUs of the last couple of cycles *are* chips designed specifically for LLMs. It’s not like you can take Grace Blackwell and play a video game on it easily. It’s highly specialized for this type of workload. And in particular, when you get into these combination GPU-LPU systems, or the AMD Helios, these are really designed specifically for LLM workloads. **Austin Lyons:** They are *morphed* specifically for LLM workloads. They have legacy ways of doing the networking, legacy ways of thinking about the memory hierarchy. I definitely agree that they are iterating toward what is best for the inference workloads they serve. But the question is: what if you started with a blank sheet? We actually saw this from OpenAI with their Jalapeño chip, which they presented at Hot Chips recently. They said: we started with a blank design, and we’re thinking very differently about it, making very different architectural decisions. They said, instead of shipping all this KV cache around and having all this shared memory and all this contention — the data is never in the right place when we want it, and our compute is always sitting around — what if every accelerator had its own little HBM slice? What if we mapped the workload and rethought it so we don’t have all this contention and shipping data around? Tensordyne is another example. They said: should it be matrix multiplication, or could we do log math? That would turn multiplies into adds, which are really fast in silicon. So I actually do think there are architectural knobs. Nvidia’s next GPU still has to have backward support for the software — not exactly, but to some extent; they want to support all the workloads, so if it ran on Hopper they want it to mostly run on Blackwell. But what if you could start with a clean sheet and make different architectural decisions — about the way you do the compute, the memory hierarchy, the way you network it up? Etched said: let’s do low-voltage inference. What would happen if we ran this at lower voltage, instead of putting in a lot of oomph so it can go really fast? If power is fixed, maybe there are benefits, even with trade-offs — it doesn’t run as fast, but it’s significantly lower power. So I do think there’s opportunity to make clean-sheet designs, make different architectural decisions, and therefore unlock that KPI I talked about. What if you could get 10x more tokens, at that fixed interactivity, for that particular model, out of your 100 megawatts, than you could buying something off the shelf from Nvidia or AMD? **David Goldman:** You hit on another interesting tension. On the one hand, you have someone like OpenAI, who is a large customer of Nvidia but is also now designing their own chip. When you hear them talk about how they did that design, they claim it was very AI-optimized — that they were able to do it much quicker because of AI acceleration. And then you also have companies like Tensordyne, who have an entirely new paradigm, a whole new way of approaching this problem, that a customer probably wouldn’t have thought of on their own, because they’re more focused on their specific workload than on new ideas from first principles. How do you think about the trade-off between custom silicon companies using AI to create something specific for what they need — maybe working with a Broadcom or Marvell to help them finish and get it into production — versus these companies with entirely new approaches? **Austin Lyons:** It’s so interesting to think about OpenAI buying compute from the big vendors, mostly GPUs, while also building their own silicon — with the advantage of their chip designers working hand-in-hand with their software team, co-designing for specific workloads. Off-the-shelf silicon vendors are not inventing in a vacuum; they’re talking with their biggest customers, asking where their roadmaps are going and how to make sure their silicon meets those needs. But that’s different from OpenAI internally having their ML team and their chip team working closely together and co-designing. That said, they’re going to land on a particular set of trade-offs. Everything in engineering is about trade-offs. Do you want more HBM or more SRAM? It’s going to cost you something either way. Do you want more die size, or do you stack it higher and take the thermal trade-offs? The merchant vendors made a particular set of trade-offs — and they need to sell their chips to as many customers as possible, even if it’s only a handful these days. The internal teams will make their own set of trade-offs, given what they know about the workloads they’re running. But to think those two sets of trade-offs are all you’ll ever need? I think there’s opportunity for someone like a Tensordyne to say: what if there are these totally crazy trade-offs, and we take the risk of doing the R&D on them? The log math stuff — surely it’s crossed the minds of people at the merchant silicon vendors or the internal XPU teams; they’ve seen a paper. But they may not be incentivized to take the risk. If you’re OpenAI’s XPU team making your first chip, are you going to play around with log math? Or are you going to say: no, no, let’s make these other interesting but proven decisions, like HBM slices that have been used elsewhere in industry? **David Goldman:** Yeah. People forget there are people involved in these decisions who have career risk. **Austin Lyons:** Exactly. **David Goldman:** If you can make a decision that’s high-probability and still works, but maybe isn’t the 10x — that’s probably better for you if you’re at a big company, particularly delivering your first version of something. You don’t want to screw up. **Austin Lyons:** And it’s expensive, too. You’re going to tape it out, and you might be all-in a billion dollars or something. You don’t want to get that wrong, or have it get cancelled before you can stand it up. So yes, there’s this whole human side, these incentives. And again, that continues to create opportunities for startups to say: we’re going to take that risk. Or, in Tensordyne’s case: we’ve been taking that risk — we were trying the log math in a different market, and now we’re ready to bring it to this one. Could an OpenAI say “that’s super interesting, let’s try a couple racks”? Absolutely — these are very sophisticated buyers, and they’re always watching what else is out there, because they completely understand that designs made at a particular point in time may not fit exactly what they need a couple of years from now. ## AI-designed chips and the falling barrier to custom silicon **David Goldman:** Are more companies looking at custom chips? OpenAI has a lot of money and a lot of really talented engineers — even without AI, they probably could have designed their own chip. But as AI makes designing a chip easier — it’s not at the level of your son creating a video game with Codex, but you talk to teams and they’re seeing lots of improvements — there’s some floor to how cheap it can get, because you ultimately need to tape out and do things in the physical world. Do you think we’ll see a big proliferation of companies that never would have tried custom silicon giving it a shot? **Austin Lyons:** I definitely think so, and you can already look at examples where it’s happening. Why would you design your own silicon? If you know your workload really well, what you bought off the shelf — where design decisions were already made — might not map to your workload perfectly. That might be okay at first. But take Rivian, for example. They went through a couple of different merchant silicon vendors, mapping their workload to the parts, and it was fine enough. But if you’re running an electric vehicle and trying to do autonomous driving, you have very specific needs. You want to use as little power as possible, because otherwise you’re taking battery away from the customer being able to drive another couple of miles. On the other hand, you need to run as fast as possible, because if it takes too long to make a decision, that’s another 20 meters of braking distance while you were thinking. Now all of a sudden they’ve got lidar and cameras and all this data flowing around that they need to do inference on as fast as possible. And by the way, it used to be convolutional neural networks; now they’re doing end-to-end VLA — vision-language-action — models. The workload has been changing, and in this world of real-time inference on big models with power under control, plus a particular interconnect bandwidth and memory capacity and memory bandwidth — they’re saying: man, this stuff off the shelf just doesn’t fit our needs and our cost profile. So the answer could be: design your own chip. Well, there’s a cost to that. You need engineers familiar with front-end design, back-end — maybe you can partner for testing — and at the end of the day, that might take you three years of development. You’re paying those engineers for years, there’s a roadmap, it’s a very big investment, and then there’s the cost to tape out and actually get it built. But what if, with the help of AI, it still takes a hundred people but instead of three years it takes one? Maybe your cost is cut by two-thirds. Where before, even though the performance and the headroom would let you do interesting things, your CFO was saying “we just can’t add another $2,000 to the bill of materials” — maybe now you come back and say it’s only going to be $700 on the BOM, and they say: okay, that’s really interesting, we think we can hack it. So I think adding AI to chip design — speeding up time to market, doing more with the same number of people — will ultimately be net good, and will reduce that cost barrier to entry. Or it could be a talent thing: how do I go find a hundred people? Maybe you only need fifty. I just think it will reduce barriers to entry, and we’ll see all sorts of cases where people who never thought about making their own chip will say: that’s actually something we could do. **David Goldman:** It’s an interesting dynamic, because the same speed-up available to these companies is also available to the merchant silicon teams — who are probably even better positioned to use these tools, because an experienced engineer who understands the trade-offs is going to use the tool better than someone approaching their first chip design. So maybe they can start proliferating their number of SKUs and serve more customers with more semi-custom things within what is somewhat merchant. I just think it’s really interesting. I don’t know how it’s going to work. **Austin Lyons:** Yeah. I would hope that merchant silicon companies — the thing is, when you’re a merchant silicon company making a particular product, there has to be a big enough market to capture enough customers for it to be worth your time and investment. There’s probably a point where the ROI didn’t make sense: the market’s only this big, the SKU costs this much to develop — not worth it. But if the cost can come down by half, maybe it clears the internal rate of return you needed. So yes, I’d expect big companies to be able to create more innovations as well. **David Goldman:** Awesome. Well, I don’t think anyone really knows how it’s going to play out. This is such an exciting time. Thank you so much — this has been an incredible conversation. I’ve really enjoyed it. **Austin Lyons:** Yes, thank you. This was fun. Let’s do it again. **David Goldman:** Absolutely.