# Nvidia and Cerebras are selling performance their customers will never see > Source: > Published: 2026-08-27 23:27:04+00:00 Shots were fired at the Hot Chips conference in California this week as Nvidia announced its new Groq-3-based LPX racks had entered production, with early tests showing the systems churning an eye-watering 3,400 tokens a second in Gemma 4 31B. That's four times faster than rival Cerebras. A day later, Cerebras fired back, touting nearly equivalent performance from its next-gen CS-4 accelerators revealed last week. These top-line performance figures make inference feel instantaneous relative to the chatbots we've grown accustomed to over the past four years. But the two companies are essentially arguing over numbers their customers will probably never see in production. REG AD To be clear, neither is lying. If you wanted to recreate these results, you certainly could — Artificial Analysis, the team responsible for both sets of benchmarks, knows what they're doing — but beyond a marketing gimmick, no inference-as-a-service model operator in their right mind would run either system this way. Not unless you've somehow figured out how to make a profit serving one request at a time. REG AD In reality, these numbers are more like the top speed on a race car. It makes for great marketing but you probably aren't driving that fast on a regular basis, and if you did, you wouldn't get very far before your tank runs dry. These aren't the numbers that matter Unsurprisingly, the economics of "premium" or "ultra-low latency" inference are a bit more nuanced, but ultimately come down to how efficiently you can scale that performance at the rightmost end of the Pareto frontier. We've shown the graphic below before. It charts the performance characteristics of how different Nvidia B300 configurations perform across a Pareto front. As you can see, GPUs are great for high-throughput, low-interactivity applications, but run out of steam quite quickly as per-user generation rates climb. Chips like Nvidia's Groq-3 LPUs or Cerebras' dinner plate-sized accelerators offer massive bandwidth thanks to their SRAM-heavy architectures, so they can extend and push the Pareto curve out to the right. Unfortunately, the same thing that makes them so fast also limits their ability to scale that performance to large numbers of users. To understand why we need to walk through a little math. We're going to pick on Gemma 4 31B, in part because we have rough numbers for both Nvidia and Cerebras, but also because it's a small model and therefore presents a best-case scenario. Each Groq-3 accelerator is equipped with 500 MB of on-chip memory capacity. Nvidia achieved its 3,400 tok/s per user run at 8-bit precision. So to run a 31 billion-parameter model, we need about 31 GB of memory or about 64 LPUs. But that's just to hold the model's weights. Since it's running as a standalone system, we also have to account for the KV caches generated during the prefill stage in which the prompt is processed. According to LMcache's KV calculator, at 8-bit precision, a 100,000-token input sequence is going to chew up another 8 GB of memory, and that's per sequence. So for every additional 100,000 token input, that's another 16 LPUs worth of memory. REG AD By our estimate, that means the most a single LPX rack with its 256 LPUs can manage on its own before running out of memory is a batch size of 12 at a 100,000 token input length each. The numbers look a little different for Cerebras' CS4, but the conclusion is largely unchanged. Each CS4 rack contains three massive WSE-3T accelerators with 44 GB of SRAM apiece, totaling 132 GB of capacity. That leaves the system a bit more headroom, but we're still looking at a maximum batch size of 12. To be clear, that's not a compute limitation. The systems simply don't have enough memory to support larger batches without adding additional racks or cutting the prompt size. Sure, we're oversimplifying a bit. We're assuming batch inference with fixed input sequences, not concurrent users hitting the systems asynchronously, and with different input and output sequences each time. For a chatbot, especially one with a limited context window — think the model's short term memory — a single CS4 or LPX rack could conceivably serve far more users than the 12 noted here. Lifting the curve Even if both systems can hit 3,400 tok/s, the better platform is going to be the one that delivers better scale at a lower cost and power consumption. The benchmark figures touted by Nvidia and Cerebras this week are really measuring the top speed the systems could run on, not the speed customers will actually run them. In fact, customers probably won’t run these systems on their own. These SRAM-heavy architectures work a lot better as decode accelerators, which is exactly why Nvidia spent $20B to license Groq's tech and poach its engineering team. It's also why Cerebras recently teamed up with AWS and AMD. REG AD We've discussed these kinds of heterogeneous compute architectures in the past, but in a nutshell they involve GPUs or other compute heavy XPUs, like Amazon's Trainium chips, with something like Groq's LPUs, Cerebras' waferscale accelerators, or SambaNova's SN-series parts. This combination largely eliminates, or at least minimizes, the KV cache and compute overheads that made scaling Cerebras' and Groq's chips so difficult. As decode accelerators, they only really need to hold the model weights, as almost everything else can get on just fine on a GPU. Again, top line AI performance makes for great headlines, but the far right of the Pareto is rarely where you'll want to be. As is usually the case with Pareto curves, the sweet spot usually presents itself as a notable hump or knee pushing toward the upper right corner of the chart. Here you'll be trading some per-user interactivity for better concurrency, throughput, and economics. Combining GPUs with Cerebras or Groq accelerators is what Nvidia, AMD, and AWS are all counting on to make "premium inference" cost-effective. There's a better benchmark for this We've yet to see benchmarks combining Cerebras' or Groq's chips with Nvidia, AMD, or AWS' respective GPUs and XPU platforms, but these are the metrics that are actually going to matter. SemiAnalysis' InferenceX offers a better representation of whether or not adding a CS4 or LPX rack would even help, as we can see exactly how each system performs across the full Pareto, ranging from high-volume low-interactivity "bulk" tokens on the left, to ultra-low-latency but also lower-volume tokens on the right. In fact, while Nvidia and Cerebras opted for Artificial Analysis' simpler test methodology, OpenAI actually opted for InferenceX to show how its spicy new Jalapeño accelerators perform across bulk and premium inference regimes. So why didn't Nvidia and Cerebras do the same? If we had to guess, it's because their LPX and CS4 racks are still best suited toward low-latency inference and wouldn't look as impressive on the left and middle sections of the Pareto curve. We fully expect that both companies will share InferenceX results for their combined GPU and LPX/CS4 systems when they're ready, but no marketer ever got a bonus for making their product look worse than it really is, so we'll just have to wait. In any case, the next time you see someone boasting of some impossibly fast inference accelerator, it's worth asking where on the Pareto that number came from. ®