{"slug": "the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can", "title": "The picks and shovels are the story. Here's the chip nobody outside AI infra can explain.", "summary": "A developer argues that the real money in AI is in the physical infrastructure layer—chips, servers, power, and cooling—not the chatbots. Using Groq as an example, the piece explains the 'memory wall' bottleneck in inference, where decode speed is limited by memory bandwidth, and notes that electricity is becoming the binding constraint as demand for compute has no natural ceiling.", "body_md": "There's a line from a recent David Lin conversation with Clem Chambers that stuck with me: the real money in AI isn't the chatbots. It's the physical layer. Chips, servers, drives, cables, cooling, power, buildings, and the banks writing the cheques for all of it.\n\nYou can't buy OpenAI or Anthropic directly. You can buy the people selling them electricity and silicon.\n\nThat framing is easy to nod along to and hard to act on, because most people arguing about AI hardware are actually arguing about one company's stock price. So I want to go a level down. Not \"buy the picks and shovels\" as a slogan, but: what is the actual bottleneck inside the box, who is attacking it from which angle, and what does a founder do with that information on a Tuesday morning.\n\nI'll use Groq as the worked example, because it's the clearest case of a company that ignored the obvious problem and went after a different one.\n\nThe podcast made four points about infrastructure that I think are correct, and one that I'd push back on.\n\n**Demand for compute has no natural ceiling.** This is underrated. A social network saturates when everyone has an account. Compute doesn't saturate, because every improvement in model quality creates new things worth computing. Reasoning models burn 10-50x the tokens of a single-shot answer for the same question. Agents burn more again, because one user action becomes fifteen chained model calls. The unit of demand keeps getting bigger even when the user count stays flat.\n\n**There's no prize for second place.** Correct at the lab level, and it's why capex runs hot for years rather than months. If you believe you might have the best model in eighteen months, underspending now is how you guarantee you won't.\n\n**Electricity is the binding constraint.** This one I'd put first, not fourth. You can order more chips. You cannot order more grid. A gigawatt of new generation plus interconnect is a multi-year permitting problem in the US, and that timeline doesn't compress with money. Which is why \"performance per watt\" stopped being a spec-sheet vanity metric and became a business constraint. If your data centre is power-capped rather than budget-capped, the chip that does more inference per watt literally lets you serve more customers from the same building.\n\n**Where I'd push back:** \"spread across the AI supply chain, hold ten years, ignore the news\" treats the supply chain as one thing. It isn't. Power generation, transmission, and cooling are genuinely long-cycle businesses with physical moats. Silicon is a knife fight where architectures get obsoleted in 24-month cycles. Holding both for ten years with the same conviction is treating two different risk profiles as one bet.\n\nWhich brings me to the interesting part.\n\n**Running a language model is two completely different jobs wearing one name.**\n\nJob one is **prefill**: reading your prompt. The model processes all your input tokens at once, in parallel. This is compute-heavy. Lots of maths, all of it independent, all of it able to happen simultaneously. GPUs are exceptional at this. It's essentially what they were built for.\n\nJob two is **decode**: writing the answer. The model produces one token, looks at everything it has written so far, then produces the next one. It cannot parallelise. Token 50 needs token 49 to exist first.\n\nAnd here's the catch that makes decode a hardware problem rather than a maths problem: **to generate each single token, the chip has to read the entire model's weights out of memory.** All of them. For a 70-billion-parameter model, that's roughly 140GB moved from memory to compute, per token.\n\nSo the compute units sit mostly idle, waiting. The pipe from memory is the ceiling. People in the field call this the memory wall.\n\nAn H100 has around 3.35 TB/s of memory bandwidth. Do the arithmetic on 140GB per token and you can see why GPU decode speeds land where they land. The chip isn't thinking slowly. It's queuing.\n\n**Every serious inference chip company is an answer to this one question:** how do you stop the weights from having to travel so far?\n\nGroq was founded in 2016 by Jonathan Ross, who led the team that built Google's first TPU. He left with a specific prediction: memory bandwidth, not compute, would become the thing that decides AI performance. He was early by about six years.\n\nTheir answer has three ideas. All three are simple once you see them.\n\nA GPU keeps model weights in HBM, which is memory that sits *next to* the processor and gets read across a bus. Fast, but it's still a journey.\n\nGroq's LPU puts hundreds of megabytes of SRAM directly on the die and uses it as primary weight storage rather than as a cache. The weights don't live somewhere else and get fetched. They live *on the chip*, already there, permanently.\n\nThe Groq 3 generation carries around 500MB of on-chip SRAM with roughly 150 TB/s of bandwidth, against the H100's 3.35 TB/s. That's not a 10% improvement. It's a different order of magnitude, and it's the whole ballgame for decode.\n\n**The analogy I use with non-technical people:** a GPU is a chef with a huge, well-stocked walk-in freezer down the hall. Every dish, they walk to the freezer and back. Fast walker, big freezer, but the hallway is the constraint. The LPU is a chef whose ingredients are all pre-laid out on the counter within arm's reach. Tiny counter. Nothing in the freezer. But no walking.\n\nA GPU decides at runtime what happens next. Threads get scheduled, caches hit or miss, memory access patterns vary. This is flexible and it's why GPUs run anything. It also means the exact same request takes 40ms sometimes and 95ms other times.\n\nThe LPU throws that out. There are no caches, no dynamic scheduling, no out-of-order execution. A compiler works out the entire schedule ahead of time, down to which data arrives at which unit on which clock cycle. The chip then just executes the score.\n\nGroq describes this as a programmable assembly line, and the metaphor is exact. On an assembly line nobody decides at 2pm what station three does next. It was decided when the line was designed, and the part arrives at station three at the moment station three is ready for it.\n\nNotably, Groq wrote the compiler *before* they designed the chip. The hardware was shaped to fit the software rather than the other way round.\n\nBecause 500MB is small, a real model doesn't fit on a single LPU. It gets split across many chips, which then have to coordinate.\n\nOn GPU clusters this coordination is a well-known source of unpredictable delay. Switches, routing, packets arriving in odd orders. Groq extends the static schedule across the whole rack. The compiler knows when data will arrive at each chip, so chips connect directly with no switches deciding anything at runtime. Hundreds of chips act as one large processor.\n\n**Speed**, obviously. Roughly 5-10x GPU decode throughput depending on model and generation.\n\n**Tail latency**, which matters more and gets discussed less. Deterministic execution means the same work takes the same time every time. On a GPU your median response might be 200ms and your 99th percentile 900ms. When one user action fans out into fifteen sequential model calls, you're not exposed to the median. You're exposed to the worst one in the chain, fifteen times over. This is why agent products feel unpredictably slow in production while feeling fine in demos.\n\n**Energy per token.** No HBM stacks, no complex cooling. Groq's racks are air-cooled by design. Circle back to the electricity bottleneck from part one and you can see why this is a commercial argument, not an environmental one.\n\nI'd rather you hear this from me than find out during a migration.\n\n**It can't train.** Training needs backpropagation, gradient accumulation, optimiser states. Wrong shape entirely for a statically-scheduled inference pipeline. You still need GPUs upstream.\n\n**Compile time is real.** Solving that scheduling puzzle for a new model takes serious time. Great for a model you'll serve for six months. Painful for research where the architecture changes weekly.\n\n**Small memory per chip means you need lots of chips**, which is why Groq serves a curated catalogue of open models rather than letting you upload anything.\n\n**Prefill isn't its strength.** Long-context, document-heavy work leans on the phase GPUs win. The emerging pattern is hybrid: GPU does prefill, hands the KV cache to the LPU, LPU does decode. That works but it adds real infrastructure complexity.\n\nFive different bets on the same problem. I've kept this to what changes a decision.\n\nNvidia GPU (H200/B200) |\nGroq LPU |\nCerebras WSE |\nSambaNova RDU |\nGoogle TPU / AWS Trainium |\n|\n|---|---|---|---|---|---|\nCore bet |\nGeneral-purpose parallel compute wins everything | Kill the memory wall with on-chip SRAM + static scheduling | One wafer-sized chip removes chip-to-chip overhead | Reconfigurable dataflow, sits between GPU and fixed ASIC | Vertical integration; own the silicon under your own cloud |\nTrain or infer |\nBoth | Inference only | Both | Both | Both |\nSpeed vs GPU |\nBaseline | ~5-10x on decode | ~15x claimed | ~5x claimed (SN50) | Competitive, varies |\nWhere it wins |\nAnything and everything; unmatched ecosystem | Real-time decode, agent loops, voice | Very large models, long context, HPC | On-prem enterprise, very large models | Cost at hyperscale inside one cloud |\nWhere it loses |\nCost per token; latency variance | No training, weak prefill, curated models only | Cost, footprint, niche software | Smaller ecosystem | Locked to GCP / AWS |\nHow you access it |\nBuy, or any cloud | API (GroqCloud) or on-prem racks | API or on-prem | Mostly on-prem | Only that cloud |\nData residency |\nFully yours if on-prem | Vendor infra unless you buy racks | Either | On-prem strength | Vendor cloud |\nSwitching cost |\n— | Near zero, OpenAI-compatible API | Low, API-based | High, on-prem commitment | Medium, cloud lock-in |\nCompany risk |\nInvestment grade | Independent post-Nvidia deal, $1.75B raised | Public (NASDAQ: CBRS) | Private, $1.5B raised | Backed by trillion-dollar parents |\n\nTwo patterns worth pulling out of that table.\n\n**Nobody is beating Nvidia at being Nvidia.** Every challenger wins a specific dimension and loses the general case. Nvidia's roughly 80% share isn't inertia. It's that one architecture credibly does training and inference and vision and research, with a software ecosystem fifteen years deep. The challengers aren't trying to replace that. They're trying to peel off the workload where specialisation pays.\n\n**Inference is where challengers can actually win**, because inference rewards specialisation and grows with *usage* rather than with lab budgets. Training demand is concentrated in maybe a dozen buyers. Inference demand is every company shipping a feature.\n\nThis is the connection most coverage misses.\n\nGroq doesn't serve Claude or GPT. It can't. Those weights aren't available. The catalogue is open-weight models: the GPT-OSS family, Qwen, Llama, Whisper for speech.\n\nThat's usually described as a limitation. I'd argue it's the entire strategic point.\n\n**The economics of open models only work if inference is cheap and fast.** Open weights are free to download and expensive to serve. That's why \"just self-host Llama\" is advice that sounds better than it works. The moment someone specialises the hardware for exactly those models, the calculus shifts. Prices land around $0.05 per million input tokens on smaller models, against a couple of dollars for frontier proprietary ones. Meta partnered with Groq for the official Llama API. That's an ecosystem forming, not a vendor selling boxes.\n\n**Where this leaves a founder:** the honest answer is not \"switch to Groq.\" It's tiering.\n\nMost products have a boring tier and a hard tier, and most teams route everything to the hard tier because it's one API key. Classification, routing, extraction, summarisation, reformatting — that's the boring tier, and an 8B open model on fast cheap hardware handles it. Genuine multi-step reasoning, ambiguous judgement, anything customer-facing where quality is the product — that's the hard tier, and a frontier model earns its price.\n\nTwo providers, a router in between. Because both speak an OpenAI-compatible API, this is a config change, not a rewrite. That's the whole migration.\n\nAnd there's a category where the speed isn't an optimisation but the product itself. Voice agents live or die on a sub-second turn budget split between speech-to-text, model, and text-to-speech. At GPU decode speeds the loop doesn't close and the conversation feels stilted. At LPU speeds it does. Same for live translation, coding autocomplete, and anything where a human is watching tokens appear.\n\nI'd be doing you a disservice if I stopped at the technology.\n\nIn late 2025 Nvidia paid roughly $20B for access to Groq's inference architecture. Jonathan Ross and president Sunny Madra went to Nvidia. Simon Edwards is now Groq's CEO. Nvidia showed a Groq 3 LPX inference accelerator at GTC 2026. Reporting still differs on whether that was a non-exclusive licence or an acquisition of the inference unit.\n\nGroq raised $650M in June 2026 and GroqCloud continues to operate independently. And in June 2026, Groq deprecated most of the model IDs that every LangChain tutorial written before that date depends on. I found that out the way everyone does, with a 400 error that says the model has been decommissioned.\n\nWhich is the real lesson, and it isn't about Groq. **Speed is a feature you rent. Portability is a feature you own.** Write your code against an OpenAI-compatible interface. Keep model IDs in config, never in source. Query the provider's model list programmatically instead of hardcoding what a tutorial told you eighteen months ago. Then the question \"which chip is fastest this quarter\" becomes a pricing decision rather than an engineering project.\n\nThe podcast's advice was to hold the physical layer for ten years and ignore the daily noise. For investors, maybe. For builders, the opposite applies. The silicon underneath you will get replaced two or three times in that window. Build so you don't care which one is winning.\n\n*David Lin Podcast: https://www.youtube.com/watch?v=12GGEIB4Tco*\n\n*Sources: Groq's LPU architecture documentation and engineering blog, Groq's TSP papers from ISCA 2020 and 2022, arXiv work on prefill/decode-aware accelerator evaluation, and current vendor benchmarks. Performance figures are largely vendor-supplied and should be treated accordingly. Model availability on GroqCloud changes frequently — check the live catalogue before building against any specific model ID.*", "url": "https://wpnews.pro/news/the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can", "canonical_source": "https://dev.to/asfiya_tech/the-picks-and-shovels-are-the-story-heres-the-chip-nobody-outside-ai-infra-can-explain-4cn3", "published_at": "2026-07-22 10:21:39+00:00", "updated_at": "2026-07-22 11:00:34.265515+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-chips", "ai-products", "ai-research"], "entities": ["Groq", "David Lin", "Clem Chambers", "OpenAI", "Anthropic", "H100"], "alternates": {"html": "https://wpnews.pro/news/the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can", "markdown": "https://wpnews.pro/news/the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can.md", "text": "https://wpnews.pro/news/the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can.txt", "jsonld": "https://wpnews.pro/news/the-picks-and-shovels-are-the-story-here-s-the-chip-nobody-outside-ai-infra-can.jsonld"}}