Nobody publishes tokens per second for your model on your RAM, and any page that gives you one has either measured a machine that is not yours or made it up. What can be established is a ceiling, from two numbers you can look up in five minutes.
A request has a prefill phase, where the whole prompt goes through the model in one parallel pass, and a decode phase, where each output token requires its own pass. They are limited by different things, and conflating them is why CPU inference advice is so often useless.
Decode is the one people mean when they say a local model is slow. To produce one token, the CPU must read every active weight in the model out of RAM and into cache, do a small amount of arithmetic with each, and move on. At a batch size of one there is nothing to amortise that read against — one pass over all the weights buys exactly one token. So decode is bound by memory bandwidth, and the arithmetic units are mostly idle waiting for data.
If each token requires reading the weights once, then:
tokens/second <= usable memory bandwidth (bytes/s) / weight bytes
The numerator is derivable from the DDR standard your machine uses. A DDR4 or DDR5 channel is 64 bits wide, so it moves 8 bytes per transfer, and the number in the name is the transfer rate:
DDR4-3200, one channel: 3200e6 transfers/s x 8 B = 25.6 GB/s
DDR4-3200, dual channel: = 51.2 GB/s
DDR5-5600, one DIMM: 5600e6 transfers/s x 8 B = 44.8 GB/s
DDR5-5600, two DIMMs: = 89.6 GB/s
The denominator is the size of the file on disk, which you can read off the repository listing before down. It is a good proxy for the bytes streamed per token in a dense model, because a GGUF file is almost entirely tensor data.
Apple publishes unified memory bandwidth directly rather than leaving you to derive it: in Apple’s October 2024 announcement of the M4 Pro and M4 Max the figures given are 273 GB/s for the M4 Pro and 546 GB/s for the M4 Max. That is the single reason Apple Silicon does well at local inference — not the CPU, the bus.
Dividing the bandwidths above by real published file sizes gives the following ceilings. Every file size is from the corresponding Hugging Face GGUF repository listing, checked 2026-08-11; every bandwidth is either derived above or published by Apple. Every cell is bandwidth divided by file size and nothing else.
Model (Q4_K_M) size DDR4-3200 DDR5-5600 M4 Pro M4 Max
51.2 GB/s 89.6 GB/s 273 GB/s 546 GB/s
Phi-3.5-mini-instruct 2.39 GB 21.4 37.4 114.1 228.1
Meta-Llama-3.1-8B-Instr. 4.92 GB 10.4 18.2 55.5 111.0
Qwen2.5-Coder-14B-Instr. 8.99 GB 5.7 10.0 30.4 60.7
Meta-Llama-3.1-70B-Instr. 42.52 GB 1.2 2.1 6.4 12.8
units: tokens per second, upper bound only
Read the bottom row carefully, because it is the useful one. A 70B at Q4_K_M on a dual-channel DDR5 desktop cannot exceed roughly two tokens per second no matter what CPU is attached, because 42.5 GB has to cross the memory bus for every single token. No amount of cores fixes that. It is also why the 70B row is the one where people conclude local inference does not work, when what does not work is that specific ratio.
These are ceilings, not predictions, and they are the only kind of number that can honestly be given without running your machine. A real figure requires a measurement on your hardware — see benchmarking your own tokens per second.
Achieved bandwidth is always below peak, and on a general-purpose desktop it is well below. The gap has named causes, all of which you can check on your own machine rather than take on trust:
dmidecode
on Linux or wmic memorychip
on Windows.-t
often peaks below the core count, and why the right value is found by sweeping it rather than by setting it to the number of cores.Prefill is not bandwidth-bound, because the whole prompt is available at once and every weight read is reused across all of its tokens. It is bound by arithmetic instead, at roughly two floating-point operations per parameter per token:
8.03e9 params x 2 FLOP = 16.06 GFLOP per token of prompt
512-token prompt = 8.2 TFLOP total
Your CPU’s peak throughput is derivable the same way: cores x FMA units x SIMD lanes x 2 x clock. A sixteen-core machine with two AVX-512 FMA units doing sixteen FP32 lanes at 3.5 GHz peaks at about 3.6 TFLOP/s, which puts that 512-token prefill at roughly 2.3 seconds before any inefficiency — several seconds of silence before the first token, on a machine that then generates at a perfectly usable rate.
The practical consequence is that CPU-only inference degrades on prompt length far more sharply than on output length, which is the opposite of the intuition people bring from hosted APIs. Short prompts and a small model are comfortable. Long documents are not, and no quantization level fixes that, because prefill cost scales with parameters and tokens, not with bits per weight.