Prefill and Decode Want Different Computers AWS is pairing its Trainium chips with Cerebras CS-3 systems to split transformer inference into prefill and decode phases, with Trainium handling prefill and Cerebras handling decode, shipping as a premium tier on Bedrock. AMD is making the same split with Helios rack-scale systems and the Cerebras Wafer-Scale Engine, claiming 5x higher tokens per second per watt, with Helios entering Cerebras datacenters before end of 2026. NVIDIA is doing the same in-house with Rubin CPX, a chip with 128 GB of GDDR7 for prefill, because decode is memory-bound and prefill is compute-bound, a gap that grows with model size. Prefill and Decode Want Different Computers Three announcements inside a year, all making the same architectural bet. AWS is pairing Trainium with Cerebras: Trainium runs prefill, a Cerebras CS-3 runs decode, and the result ships as a premium tier on Bedrock. 1 AMD is pairing Helios rack-scale systems with the Cerebras Wafer-Scale Engine on the same split, claiming 5x higher tokens per second per watt, with Helios going into Cerebras datacenters before the end of 2026. And NVIDIA is doing the same arbitrage in-house with Rubin CPX, a part built specifically for the prefill phase, carrying 128 GB of GDDR7 instead of HBM. 2 fn:2 3 fn:3 None of this is fashion. It falls out of a property of the transformer decode loop that has been true since the first autoregressive model and gets worse with every hardware generation. This post is about why the split is inevitable. The four that follow are about why programming the result is so much harder than the press releases suggest. The two phases are not the same computation Serving one request has two phases with almost nothing in common. Prefill processes the whole prompt at once. For a prompt of $N$ tokens and hidden dimension $d$, each projection in each layer is a matrix multiply of shape $ N \times d \times d \times d $. That is $2Nd^2$ floating-point operations against $d^2$ weights read from memory. At two bytes per parameter, arithmetic intensity is For a prompt of a few thousand tokens, that is a few thousand FLOP per byte. Every accelerator built in the last decade is compute-bound in that regime. Decode emits one token per sequence per step. The same projection becomes $ 1 \times d \times d \times d $ — a matrix-vector product. You read the entire weight matrix to produce a single token: That is the whole problem in one line. Prefill and decode differ in arithmetic intensity by three orders of magnitude, on identical weights, in the same model, for the same request. Batching is the standard answer and it is a partial one. Running $B$ sequences together turns the GEMV back into a GEMM of shape $ B \times d \times d \times d $, so intensity rises to roughly $B$. To approach the compute-bound regime on a modern GPU you need $B$ in the hundreds. Two things push back. Larger batches raise per-token latency for every sequence in the batch, which is the metric users feel most directly. And the KV cache scales with $B$ times context length, so the batch you want for arithmetic intensity is often the batch you cannot fit. Attention makes it worse. At decode step $t$, attention reads the entire KV cache accumulated so far and does $O t $ work on $O t $ bytes. There is no reuse to find. That component is memory-bound at every batch size, and it grows linearly with context, which is precisely the direction the industry is moving. So decode is bandwidth-starved by construction, and long context starves it further. The ceiling, in one division The abstraction becomes concrete fast. At batch 1, generating a token requires reading every weight in the model exactly once. So the upper bound on single-stream decode throughput is a division: \ \text{tokens/sec} \le \frac{\text{memory bandwidth}}{\text{bytes of weights}}\ Nothing about the kernel, the framework, or the compiler enters that expression. An 8B model at fp16 is 16 GB. On an accelerator with 3 TB/s of bandwidth, batch-1 decode cannot exceed roughly 190 tokens per second no matter what you do to the software. A 70B model at fp16 is 140 GB, which puts the same ceiling near 21 tokens per second and does not fit in one H100’s memory in the first place. That division is why decode latency is a hardware property rather than an optimization target. You can improve the constant factor with quantization, which shrinks the numerator’s denominator, or with speculative decoding, which amortizes one weight sweep across several accepted tokens. Both are real and both are bounded. What you cannot do is make a batch-1 GEMV compute-bound. It is also why 21 PB/s is the number Cerebras leads with. Put the weights in SRAM and the division comes out somewhere else entirely. The design has a cost that belongs in the same paragraph. Those 44 GB are distributed across 900,000 cores as small local memories rather than pooled the way HBM is, and a model whose weights exceed 44 GB needs more than one wafer. For frontier-scale models the decode side is a multi-wafer system with its own partitioning problem, which is a real constraint rather than a footnote. One chip cannot be right for both Put the two phases on the same accelerator and you buy hardware that is wrong for one of them. An H100 pairs roughly 3 TB/s of HBM3 bandwidth with a very large amount of compute. Run prefill on it and the FLOPs are the binding constraint while most of the bandwidth sits unused. Run decode on it and the bandwidth is the binding constraint while most of the FLOPs sit unused. You paid for both. Each phase uses about half of what you bought. The interesting part is that the gap is widening. Splitwise measured it directly across one generation: from A100 to H100, compute grew 3.43x while memory bandwidth grew only 1.64x. 4 Prefill got more than twice as much of what it needs as decode did. Extrapolate two more generations and a single balanced SKU is not a compromise, it is a chip that is mostly idle no matter which phase you run on it. That is the reasoning behind the recent silicon, and the specifications make the intent obvious. Rubin CPX , built for prefill, delivers 30 PFLOPS at NVFP4 from a monolithic die with 128 GB of GDDR7 rather than HBM. 3 Reporting puts the resulting bandwidth near 1.8 TB/s on a 512-bit interface, which is an estimate from the memory configuration rather than a published figure. Less bandwidth than an H100, far more compute. NVIDIA’s framing is that prefill “heavily utilizes compute and only lightly uses memory bandwidth,” so the right part is skinny on bandwidth and fat on compute. Cerebras WSE-3 , which the AWS and AMD deals put on decode, is the exact inverse. 900,000 cores and 44 GB of SRAM distributed across a 46,225 mm² wafer, 4 trillion transistors, 125 PFLOPS of peak AI compute, and 21 PB/s of memory bandwidth. 5 There is no HBM anywhere in the design. That bandwidth figure is roughly seven thousand times an H100’s, which is a number that only makes sense once you accept that decode is a bandwidth problem wearing a compute problem’s clothing. graph TB subgraph P "Prefill: compute-bound, intensity ~ N" P1 "whole prompt at once" -- P2 "GEMM N x d x d x d " P2 -- P3 "wants FLOPs