# Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill > Source: > Published: 2026-07-31 20:57:34+00:00 You benchmark a 70B model with `batch_size=1` , one prompt, one stream. FP16 gives you 18 tokens/sec. You swap in an AWQ INT4 checkpoint and get 55 tokens/sec. Three times faster, same GPU, ~1 point of accuracy lost. You ship it. Then production traffic arrives: 8k-token RAG prompts, 40 concurrent users. Time-to-first-token gets *worse* than the FP16 build, and your throughput at high concurrency is flat or slightly down. Nothing is broken. INT4 weight-only quantization did exactly what it does — it reduced bytes moved, and prefill was never bottlenecked on bytes. Because decode and prefill sit on opposite sides of the roofline, and INT4 weight-only quantization only moves the memory axis. A transformer's linear layers are the whole story here. During decode with `B` concurrent sequences, each weight matrix `[K, N]` is read from HBM once and used for `B` token-vectors: `2 · B · K · N` `K · N · b` where `b` = bytes per weight (2 for BF16, 0.5 for INT4)Arithmetic intensity is therefore `2B / b` FLOP/byte. At `B = 1` , BF16 gives you **1 FLOP per byte**. An H100 wants ~295. You are running the tensor cores at roughly a third of a percent of peak and burning the entire kernel time on HBM reads. Shrink the weights 4x and the kernel gets ~4x faster — the math never had to change. Prefill runs the same weight matrix against `S` tokens at once. With `S = 2048` , intensity is `2·2048/2 ≈ 2048` FLOP/byte — an order of magnitude past the ridge point. The weight read is amortized to nothing. You are limited by tensor-core throughput, and W4A16 does not raise tensor-core throughput. It lowers it slightly, because you now spend cycles unpacking nibbles, applying per-group scales and zero-points, and writing FP16 into the MMA path. That is the sentence to remember: **W4A16 buys bytes, not FLOPs.** Roughly the roofline ridge point scaled by bytes-per-weight. On an H100 SXM it is about 74 concurrent decode tokens for INT4 versus ~295 for BF16. ``` # Where does a weight-stationary GEMM stop being memory-bound? # intensity = 2*B*K*N / (K*N*bytes_per_weight) = 2B / bytes_per_weight SPECS = { # dense tensor-core FLOPS, HBM bandwidth "H100-SXM": (990e12, 3.35e12), # BF16 "A100-80G": (312e12, 2.039e12), # BF16 } def crossover(gpu, bytes_per_weight): flops, bw = SPECS[gpu] ridge = flops / bw # FLOP/byte the GPU wants return ridge * bytes_per_weight / 2 # concurrent tokens B for gpu in SPECS: for name, b in [("BF16", 2), ("FP8", 1), ("INT4", 0.5)]: print(f"{gpu:10s} {name:5s} weights -> compute-bound at B ~= {crossover(gpu, b):5.0f}") # H100-SXM BF16 weights -> compute-bound at B ~= 296 # H100-SXM FP8 weights -> compute-bound at B ~= 148 # H100-SXM INT4 weights -> compute-bound at B ~= 74 # A100-80G BF16 weights -> compute-bound at B ~= 153 # A100-80G INT4 weights -> compute-bound at B ~= 38 ``` This model ignores activation traffic and assumes large `K, N` — fine for a 70B's 8192×28672 MLP projections, less fine for tiny models. Treat it as the right order of magnitude, not a promise. The practical reading: quantization *lowers* the concurrency at which you become compute-bound. That is counterintuitive until you see the formula. Fewer bytes per FLOP means you hit the compute wall sooner. So the regime where INT4 gives its headline speedup — small `B` — is exactly the regime a well-utilized production server tries to leave. If your serving fleet runs at 60+ concurrent decodes on H100, you have already crossed the line. Your INT4 build is doing the same FLOPs as FP16 plus dequantization. Three reasons, in descending order of impact. **Dequantization is real work in the inner loop.** A W4A16 kernel loads packed nibbles, unpacks, multiplies by a per-group FP16 scale (group size 128 is the common default), subtracts a zero-point, then feeds FP16 MMA. In the memory-bound regime that work hides behind HBM latency for free. In the compute-bound regime there is nothing to hide behind — it lands directly on the critical path. **Kernel maturity and shape sensitivity.** Marlin-class kernels closed most of the large-`M` gap for GPTQ/AWQ weights, but they still tune for specific tile shapes. Feed them a 6k-token prefill with an odd hidden size and you can fall off the fast path into a slower fallback. FP16 cuBLAS/CUTLASS GEMMs have no such cliff. **No fused epilogue advantage.** FP8 paths on Hopper get native tensor-core support and can keep activations in 8-bit through the layer. W4A16 has to materialize FP16 activations regardless, so it saves nothing on the activation side of prefill, where activations are large. Net effect in practice: TTFT moves the wrong way by a modest but measurable amount, while inter-token latency improves substantially. If your SLO is dominated by TTFT on long RAG prompts, INT4 is a regression dressed up as an optimization. Yes, structurally. Chunked prefill exists to stop long prompts from stalling decode, and it works by slicing a prompt into chunks and co-scheduling them with in-flight decode tokens in a single batched forward pass. That merged batch has a token count of `B_decode + chunk_size` . With a 512-token chunk, every batch is at `B ≥ 512` — far past the ~74-token crossover. The GEMMs are compute-bound essentially all the time, so the weight-byte savings buy nothing, and the dequantization overhead is paid on every step. You can still win here, but the win comes from the freed HBM (bigger KV cache → more concurrency → better throughput), not from faster matmuls. Often yes — as a **capacity** decision rather than a latency one. Three concrete payoffs: None of those are "prefill is faster." Be honest about which one you are buying. Quantize the activations, not just the weights. ``` # vLLM, H100 fleet, high concurrency, long RAG prompts. # FP8 (W8A8) uses native Hopper tensor cores: halves weight bytes AND # roughly doubles compute peak vs BF16. model: meta-llama/Llama-3.3-70B-Instruct quantization: fp8 # W8A8, native on Hopper/Blackwell kv_cache_dtype: fp8_e4m3 # also halves KV traffic during decode max_model_len: 32768 enable_chunked_prefill: true max_num_batched_tokens: 2048 # tune against TTFT vs ITL SLOs gpu_memory_utilization: 0.92 ``` Rules of thumb: `B ≈ 1–8` ): INT4 weight-only is the right call. This is where the 3x single-stream number is real.Stop benchmarking with `batch_size=1` . Sweep concurrency and separate the two phases: ``` # Sweep concurrency; watch TTFT (prefill) and ITL (decode) separately. for c in 1 4 16 64 128; do vllm bench serve \ --model $MODEL --dataset-name random \ --random-input-len 4096 --random-output-len 256 \ --max-concurrency $c --num-prompts $((c * 8)) done ``` Log **TTFT p50/p99** and **inter-token latency p50/p99** as separate series, then plot each against concurrency for both checkpoints. The signature of the failure mode is unmistakable: the INT4 ITL curve sits well below FP16 at low concurrency and converges to it somewhere in the 50–100 range, while the INT4 TTFT curve sits *above* FP16 everywhere. If you only ever look at end-to-end throughput at one concurrency level, both effects average into a single number that tells you nothing. INT4 weight-only quantization doesn't speed up prefill because prefill is compute-bound and W4A16 only reduces memory traffic — the kernel still dequantizes to FP16 and runs the same tensor-core matmuls, so it does the same FLOPs plus unpacking overhead. Decode at low concurrency has an arithmetic intensity near 1 FLOP/byte against a GPU that wants ~295, so removing 75% of the weight bytes gives a near-linear speedup there; prefill runs at hundreds or thousands of FLOP/byte, where those bytes were already free. The crossover is around 74 concurrent decode tokens on an H100, and chunked prefill pushes every batch past it. Use INT4 for single-stream latency and for memory capacity; use FP8 or another activation-quantized scheme when you need prefill and high-concurrency throughput to actually get faster.