Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill

A developer's analysis shows that INT4 weight-only quantization speeds up decode but not prefill, because prefill is compute-bound while decode is memory-bound. The crossover point where a GEMM becomes compute-bound is lower for INT4 than for BF16, meaning quantization helps only at low concurrency. The developer provides a formula and code to estimate the crossover for GPUs like H100 and A100.

You benchmark a 70B model with batch size=1 , one prompt, one stream. FP16 gives you 18 tokens/sec. You swap in an AWQ INT4 checkpoint and get 55 tokens/sec. Three times faster, same GPU, ~1 point of accuracy lost. You ship it. Then production traffic arrives: 8k-token RAG prompts, 40 concurrent users. Time-to-first-token gets worse than the FP16 build, and your throughput at high concurrency is flat or slightly down. Nothing is broken. INT4 weight-only quantization did exactly what it does — it reduced bytes moved, and prefill was never bottlenecked on bytes. Because decode and prefill sit on opposite sides of the roofline, and INT4 weight-only quantization only moves the memory axis. A transformer's linear layers are the whole story here. During decode with B concurrent sequences, each weight matrix K, N is read from HBM once and used for B token-vectors: 2 · B · K · N K · N · b where b = bytes per weight 2 for BF16, 0.5 for INT4 Arithmetic intensity is therefore 2B / b FLOP/byte. At B = 1 , BF16 gives you 1 FLOP per byte . An H100 wants ~295. You are running the tensor cores at roughly a third of a percent of peak and burning the entire kernel time on HBM reads. Shrink the weights 4x and the kernel gets ~4x faster — the math never had to change. Prefill runs the same weight matrix against S tokens at once. With S = 2048 , intensity is 2·2048/2 ≈ 2048 FLOP/byte — an order of magnitude past the ridge point. The weight read is amortized to nothing. You are limited by tensor-core throughput, and W4A16 does not raise tensor-core throughput. It lowers it slightly, because you now spend cycles unpacking nibbles, applying per-group scales and zero-points, and writing FP16 into the MMA path. That is the sentence to remember: W4A16 buys bytes, not FLOPs. Roughly the roofline ridge point scaled by bytes-per-weight. On an H100 SXM it is about 74 concurrent decode tokens for INT4 versus ~295 for BF16. Where does a weight-stationary GEMM stop being memory-bound? intensity = 2 B K N / K N bytes per weight = 2B / bytes per weight SPECS = { dense tensor-core FLOPS, HBM bandwidth "H100-SXM": 990e12, 3.35e12 , BF16 "A100-80G": 312e12, 2.039e12 , BF16 } def crossover gpu, bytes per weight : flops, bw = SPECS gpu ridge = flops / bw FLOP/byte the GPU wants return ridge bytes per weight / 2 concurrent tokens B for gpu in SPECS: for name, b in "BF16", 2 , "FP8", 1 , "INT4", 0.5 : print f"{gpu:10s} {name:5s} weights - compute-bound at B ~= {crossover gpu, b :5.0f}" H100-SXM BF16 weights - compute-bound at B ~= 296 H100-SXM FP8 weights - compute-bound at B ~= 148 H100-SXM INT4 weights - compute-bound at B ~= 74 A100-80G BF16 weights - compute-bound at B ~= 153 A100-80G INT4 weights - compute-bound at B ~= 38 This model ignores activation traffic and assumes large K, N — fine for a 70B's 8192×28672 MLP projections, less fine for tiny models. Treat it as the right order of magnitude, not a promise. The practical reading: quantization lowers the concurrency at which you become compute-bound. That is counterintuitive until you see the formula. Fewer bytes per FLOP means you hit the compute wall sooner. So the regime where INT4 gives its headline speedup — small B — is exactly the regime a well-utilized production server tries to leave. If your serving fleet runs at 60+ concurrent decodes on H100, you have already crossed the line. Your INT4 build is doing the same FLOPs as FP16 plus dequantization. Three reasons, in descending order of impact. Dequantization is real work in the inner loop. A W4A16 kernel loads packed nibbles, unpacks, multiplies by a per-group FP16 scale group size 128 is the common default , subtracts a zero-point, then feeds FP16 MMA. In the memory-bound regime that work hides behind HBM latency for free. In the compute-bound regime there is nothing to hide behind — it lands directly on the critical path. Kernel maturity and shape sensitivity. Marlin-class kernels closed most of the large- M gap for GPTQ/AWQ weights, but they still tune for specific tile shapes. Feed them a 6k-token prefill with an odd hidden size and you can fall off the fast path into a slower fallback. FP16 cuBLAS/CUTLASS GEMMs have no such cliff. No fused epilogue advantage. FP8 paths on Hopper get native tensor-core support and can keep activations in 8-bit through the layer. W4A16 has to materialize FP16 activations regardless, so it saves nothing on the activation side of prefill, where activations are large. Net effect in practice: TTFT moves the wrong way by a modest but measurable amount, while inter-token latency improves substantially. If your SLO is dominated by TTFT on long RAG prompts, INT4 is a regression dressed up as an optimization. Yes, structurally. Chunked prefill exists to stop long prompts from stalling decode, and it works by slicing a prompt into chunks and co-scheduling them with in-flight decode tokens in a single batched forward pass. That merged batch has a token count of B decode + chunk size . With a 512-token chunk, every batch is at B ≥ 512 — far past the ~74-token crossover. The GEMMs are compute-bound essentially all the time, so the weight-byte savings buy nothing, and the dequantization overhead is paid on every step. You can still win here, but the win comes from the freed HBM bigger KV cache → more concurrency → better throughput , not from faster matmuls. Often yes — as a capacity decision rather than a latency one. Three concrete payoffs: None of those are "prefill is faster." Be honest about which one you are buying. Quantize the activations, not just the weights. vLLM, H100 fleet, high concurrency, long RAG prompts. FP8 W8A8 uses native Hopper tensor cores: halves weight bytes AND roughly doubles compute peak vs BF16. model: meta-llama/Llama-3.3-70B-Instruct quantization: fp8 W8A8, native on Hopper/Blackwell kv cache dtype: fp8 e4m3 also halves KV traffic during decode max model len: 32768 enable chunked prefill: true max num batched tokens: 2048 tune against TTFT vs ITL SLOs gpu memory utilization: 0.92 Rules of thumb: B ≈ 1–8 : INT4 weight-only is the right call. This is where the 3x single-stream number is real.Stop benchmarking with batch size=1 . Sweep concurrency and separate the two phases: Sweep concurrency; watch TTFT prefill and ITL decode separately. for c in 1 4 16 64 128; do vllm bench serve \ --model $MODEL --dataset-name random \ --random-input-len 4096 --random-output-len 256 \ --max-concurrency $c --num-prompts $ c 8 done Log TTFT p50/p99 and inter-token latency p50/p99 as separate series, then plot each against concurrency for both checkpoints. The signature of the failure mode is unmistakable: the INT4 ITL curve sits well below FP16 at low concurrency and converges to it somewhere in the 50–100 range, while the INT4 TTFT curve sits above FP16 everywhere. If you only ever look at end-to-end throughput at one concurrency level, both effects average into a single number that tells you nothing. INT4 weight-only quantization doesn't speed up prefill because prefill is compute-bound and W4A16 only reduces memory traffic — the kernel still dequantizes to FP16 and runs the same tensor-core matmuls, so it does the same FLOPs plus unpacking overhead. Decode at low concurrency has an arithmetic intensity near 1 FLOP/byte against a GPU that wants ~295, so removing 75% of the weight bytes gives a near-linear speedup there; prefill runs at hundreds or thousands of FLOP/byte, where those bytes were already free. The crossover is around 74 concurrent decode tokens on an H100, and chunked prefill pushes every batch past it. Use INT4 for single-stream latency and for memory capacity; use FP8 or another activation-quantized scheme when you need prefill and high-concurrency throughput to actually get faster.