Tokens per Second Is a Memory Bandwidth Number

Tokens per second for local LLM generation is primarily a memory bandwidth number, not a measure of silicon cleverness, according to an analysis by an unnamed author. At batch size one, generation speed equals memory bandwidth divided by bytes read per token, with model weights dominating the byte count. For example, a 32B model at 4-bit quantization (~19 GB) has a ceiling of ~53 tok/s on an RTX 4090 (~1000 GB/s) and ~21 tok/s on a MacBook Pro M1 Max (~400 GB/s). Quantization is thus a speed feature, and prefill and decode are effectively different machines.

Tokens per Second Is a Memory Bandwidth Number People compare local models by tokens per second the way they once compared CPUs by clock speed, as if it were a property of the silicon's cleverness. It mostly is not. For generation at batch size one, tokens per second is memory bandwidth divided by the bytes you read per token, and the biggest term by far is the model's weights. You can predict your speed before the download finishes, with division. One token, one full read of the weights A transformer answers in two phases. During prefill it processes your whole prompt at once. During decode it produces output tokens one at a time, and each new token depends on the one before it, so there is no batching your way out of the sequence on a single stream. Here is the fact the benchmark charts skip: to produce one token, the model multiplies activations against essentially every weight matrix it has. Each weight participates in one multiply. Which means each weight must travel from memory to the compute units once per generated token. The compute itself is trivial for modern hardware; a multiply-accumulate per loaded byte is nowhere near enough work to hide the loading. The chip spends the step waiting on memory. decode step, batch size 1: weights in RAM/VRAM ──────────────► compute ──► 1 token read ~all of them, every step tokens/sec ≤ memory bandwidth / bytes per token bytes per token ≈ weight bytes + KV cache read That inequality is the whole story. Everything else in this post is corollaries. The arithmetic, with real machines Weight bytes are parameters times bits-per-weight over eight. A 4-bit K-quant is roughly 4.9 bits-per-weight, so an 8B model is about 5 GB, a 32B about 19 GB, a 70B about 42 GB. Now divide your machine's bandwidth by that. | Machine | Bandwidth | 32B @ Q4 ~19 GB | 8B @ Q4 ~5 GB | |---|---|---|---| | MacBook Pro M1 Max | ~400 GB/s | ~21 tok/s ceiling | ~80 tok/s ceiling | | RTX 4090 | ~1000 GB/s | ~53 tok/s ceiling | ~200 tok/s ceiling | | Mac Studio Ultra | ~800 GB/s | ~42 tok/s ceiling | ~160 tok/s ceiling | | Desktop DDR5, CPU only | ~80 GB/s | ~4 tok/s ceiling | ~16 tok/s ceiling | These are ceilings, not promises. Real systems land somewhere between half and most of the ceiling, depending on kernel quality, how well the quant format streams, and the KV cache term we will get to. But run the division for your own machine and model, then compare against any published benchmark: the ranking it predicts is almost always the ranking you observe. When someone reports a number far above the ceiling, batching or speculative decoding is involved. When far below, something is spilling. Corollary one: quantization is a speed feature Quantization gets discussed as a memory-fit compromise, quality traded for size. The same bytes govern speed. Halving bits-per-weight halves the bytes each token must read, which roughly doubles the generation ceiling on the same hardware. An 8-bit 32B and a 4-bit 32B are not just different disk sizes; one is nearly twice as fast to generate with before quality even enters the discussion. This is why a 4-bit quant of a bigger model frequently beats an 8-bit quant of a smaller one on both quality and speed at equal memory. Bytes, not parameters, are the budget. Corollary two: prefill and decode are different machines Prefill processes all prompt tokens in parallel, so the weight read is amortized across every token of the prompt at once. Arithmetic dominates, and the GPU actually gets to work. Decode reads everything for one token at a time. This is why the same setup can chew through a 20,000-token prompt quickly and then type the answer at 20 tokens per second, and why "time to first token" and "tokens per second" respond to completely different tuning. More compute helps prefill. Only more bandwidth, or fewer bytes, helps decode. Corollary three: why MoE offload lands at a few tokens per second A mixture-of-experts model reads only its active parameters per token. A 1T-total model with 32B active reads about as many bytes per token as a dense 32B, which is what makes trillion-parameter models locally plausible at all. But when the experts live in system RAM, offloaded from a 24 GB GPU, the bandwidth that matters for those bytes is the RAM's, not the GPU's. 32B active @ ~4 bpw ≈ 16 GB read per token dual-channel DDR5 ≈ 80 GB/s 80 / 16 ≈ 5 tokens per second Every large-MoE guide that says "expect a few tokens per second on single-GPU-plus-RAM" is reporting this division, whether the author knows it or not. It also tells you what upgrades work: faster RAM and more memory channels move this number. A faster GPU does not, because the GPU was never the constraint. Corollary four: the KV cache tax grows with context Weights are not the only thing read per token. Attention also reads the KV cache, the stored keys and values for every token already in context, and that read grows linearly as the conversation grows. At short context it is noise. At 100K tokens of context it can be several gigabytes per step on a large model, which is why generation visibly slows as a long session fills, on the same model, same machine, same quant. Quantizing the cache q8 0 keys, or FP8 in server stacks and flash-attention kernels shrink the tax. Nothing removes it. A model that starts at 25 tokens per second and ends the session at 14 has not degraded; it is reading more per token, exactly as the arithmetic says it must. Why datacenter numbers look nothing like yours A serving stack runs decode for dozens of requests in one batch. The weights are read once per step and shared across every sequence in the batch, so the per-request cost of the weight read divides by the batch size. That is the entire trick behind high aggregate throughput, and it is unavailable to you, because you are one user generating one stream. Speculative decoding is the single-stream version of the same amortization: a small draft model proposes several tokens, the big model verifies them in one parallel pass, one weight read pays for several tokens when the guesses hold. When it works you beat the naive ceiling; when the draft misses often, you do not. What this predicts Apple Silicon is a bandwidth play. An Ultra-class Mac at ~800 GB/s with hundreds of gigabytes of unified memory is slower per byte than a 4090 but can hold models the 4090 cannot see. For big models, capacity at decent bandwidth beats speed at small capacity. The spill cliff is real. The moment a model exceeds VRAM and layers fall to system RAM, those layers are read at RAM speed. The slowdown is not gradual; it is proportional to the fraction of bytes that moved to the slow pool. "Faster GPU" is often the wrong purchase. If your bytes live in system RAM, buy RAM channels. If they live in VRAM, a bandwidth-heavy card matters more than a compute-heavy one. Small models feel disproportionately fast because five gigabytes divides into any bandwidth generously. That snappiness is bytes, not magic. I keep a hardware guide /guides/hardware/ with the capacity side of this, what fits which machine. This is the speed side, and it is the same variable. Before you download forty gigabytes tonight, do the division: your bandwidth, over the file size. The number that comes out is very close to the number you will watch typing at you tomorrow.