{"slug": "why-int4-weight-only-quantization-doesn-t-speed-up-prefill", "title": "Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill", "summary": "A developer's analysis shows that INT4 weight-only quantization speeds up decode but not prefill, because prefill is compute-bound while decode is memory-bound. The crossover point where a GEMM becomes compute-bound is lower for INT4 than for BF16, meaning quantization helps only at low concurrency. The developer provides a formula and code to estimate the crossover for GPUs like H100 and A100.", "body_md": "You benchmark a 70B model with `batch_size=1`\n\n, one prompt, one stream. FP16 gives you 18 tokens/sec. You swap in an AWQ INT4 checkpoint and get 55 tokens/sec. Three times faster, same GPU, ~1 point of accuracy lost. You ship it.\n\nThen production traffic arrives: 8k-token RAG prompts, 40 concurrent users. Time-to-first-token gets *worse* than the FP16 build, and your throughput at high concurrency is flat or slightly down. Nothing is broken. INT4 weight-only quantization did exactly what it does — it reduced bytes moved, and prefill was never bottlenecked on bytes.\n\nBecause decode and prefill sit on opposite sides of the roofline, and INT4 weight-only quantization only moves the memory axis.\n\nA transformer's linear layers are the whole story here. During decode with `B`\n\nconcurrent sequences, each weight matrix `[K, N]`\n\nis read from HBM once and used for `B`\n\ntoken-vectors:\n\n`2 · B · K · N`\n\n`K · N · b`\n\nwhere `b`\n\n= bytes per weight (2 for BF16, 0.5 for INT4)Arithmetic intensity is therefore `2B / b`\n\nFLOP/byte. At `B = 1`\n\n, BF16 gives you **1 FLOP per byte**. An H100 wants ~295. You are running the tensor cores at roughly a third of a percent of peak and burning the entire kernel time on HBM reads. Shrink the weights 4x and the kernel gets ~4x faster — the math never had to change.\n\nPrefill runs the same weight matrix against `S`\n\ntokens at once. With `S = 2048`\n\n, intensity is `2·2048/2 ≈ 2048`\n\nFLOP/byte — an order of magnitude past the ridge point. The weight read is amortized to nothing. You are limited by tensor-core throughput, and W4A16 does not raise tensor-core throughput. It lowers it slightly, because you now spend cycles unpacking nibbles, applying per-group scales and zero-points, and writing FP16 into the MMA path.\n\nThat is the sentence to remember: **W4A16 buys bytes, not FLOPs.**\n\nRoughly the roofline ridge point scaled by bytes-per-weight. On an H100 SXM it is about 74 concurrent decode tokens for INT4 versus ~295 for BF16.\n\n```\n# Where does a weight-stationary GEMM stop being memory-bound?\n# intensity = 2*B*K*N / (K*N*bytes_per_weight) = 2B / bytes_per_weight\n\nSPECS = {                      # dense tensor-core FLOPS, HBM bandwidth\n    \"H100-SXM\": (990e12, 3.35e12),   # BF16\n    \"A100-80G\": (312e12, 2.039e12),  # BF16\n}\n\ndef crossover(gpu, bytes_per_weight):\n    flops, bw = SPECS[gpu]\n    ridge = flops / bw                       # FLOP/byte the GPU wants\n    return ridge * bytes_per_weight / 2      # concurrent tokens B\n\nfor gpu in SPECS:\n    for name, b in [(\"BF16\", 2), (\"FP8\", 1), (\"INT4\", 0.5)]:\n        print(f\"{gpu:10s} {name:5s} weights -> compute-bound at B ~= {crossover(gpu, b):5.0f}\")\n\n# H100-SXM  BF16  weights -> compute-bound at B ~=   296\n# H100-SXM  FP8   weights -> compute-bound at B ~=   148\n# H100-SXM  INT4  weights -> compute-bound at B ~=    74\n# A100-80G  BF16  weights -> compute-bound at B ~=   153\n# A100-80G  INT4  weights -> compute-bound at B ~=    38\n```\n\nThis model ignores activation traffic and assumes large `K, N`\n\n— fine for a 70B's 8192×28672 MLP projections, less fine for tiny models. Treat it as the right order of magnitude, not a promise.\n\nThe practical reading: quantization *lowers* the concurrency at which you become compute-bound. That is counterintuitive until you see the formula. Fewer bytes per FLOP means you hit the compute wall sooner. So the regime where INT4 gives its headline speedup — small `B`\n\n— is exactly the regime a well-utilized production server tries to leave.\n\nIf your serving fleet runs at 60+ concurrent decodes on H100, you have already crossed the line. Your INT4 build is doing the same FLOPs as FP16 plus dequantization.\n\nThree reasons, in descending order of impact.\n\n**Dequantization is real work in the inner loop.** A W4A16 kernel loads packed nibbles, unpacks, multiplies by a per-group FP16 scale (group size 128 is the common default), subtracts a zero-point, then feeds FP16 MMA. In the memory-bound regime that work hides behind HBM latency for free. In the compute-bound regime there is nothing to hide behind — it lands directly on the critical path.\n\n**Kernel maturity and shape sensitivity.** Marlin-class kernels closed most of the large-`M`\n\ngap for GPTQ/AWQ weights, but they still tune for specific tile shapes. Feed them a 6k-token prefill with an odd hidden size and you can fall off the fast path into a slower fallback. FP16 cuBLAS/CUTLASS GEMMs have no such cliff.\n\n**No fused epilogue advantage.** FP8 paths on Hopper get native tensor-core support and can keep activations in 8-bit through the layer. W4A16 has to materialize FP16 activations regardless, so it saves nothing on the activation side of prefill, where activations are large.\n\nNet effect in practice: TTFT moves the wrong way by a modest but measurable amount, while inter-token latency improves substantially. If your SLO is dominated by TTFT on long RAG prompts, INT4 is a regression dressed up as an optimization.\n\nYes, structurally. Chunked prefill exists to stop long prompts from stalling decode, and it works by slicing a prompt into chunks and co-scheduling them with in-flight decode tokens in a single batched forward pass.\n\nThat merged batch has a token count of `B_decode + chunk_size`\n\n. With a 512-token chunk, every batch is at `B ≥ 512`\n\n— far past the ~74-token crossover. The GEMMs are compute-bound essentially all the time, so the weight-byte savings buy nothing, and the dequantization overhead is paid on every step.\n\nYou can still win here, but the win comes from the freed HBM (bigger KV cache → more concurrency → better throughput), not from faster matmuls.\n\nOften yes — as a **capacity** decision rather than a latency one. Three concrete payoffs:\n\nNone of those are \"prefill is faster.\" Be honest about which one you are buying.\n\nQuantize the activations, not just the weights.\n\n```\n# vLLM, H100 fleet, high concurrency, long RAG prompts.\n# FP8 (W8A8) uses native Hopper tensor cores: halves weight bytes AND\n# roughly doubles compute peak vs BF16.\nmodel: meta-llama/Llama-3.3-70B-Instruct\nquantization: fp8              # W8A8, native on Hopper/Blackwell\nkv_cache_dtype: fp8_e4m3       # also halves KV traffic during decode\nmax_model_len: 32768\nenable_chunked_prefill: true\nmax_num_batched_tokens: 2048   # tune against TTFT vs ITL SLOs\ngpu_memory_utilization: 0.92\n```\n\nRules of thumb:\n\n`B ≈ 1–8`\n\n): INT4 weight-only is the right call. This is where the 3x single-stream number is real.Stop benchmarking with `batch_size=1`\n\n. Sweep concurrency and separate the two phases:\n\n```\n# Sweep concurrency; watch TTFT (prefill) and ITL (decode) separately.\nfor c in 1 4 16 64 128; do\n  vllm bench serve \\\n    --model $MODEL --dataset-name random \\\n    --random-input-len 4096 --random-output-len 256 \\\n    --max-concurrency $c --num-prompts $((c * 8))\ndone\n```\n\nLog **TTFT p50/p99** and **inter-token latency p50/p99** as separate series, then plot each against concurrency for both checkpoints. The signature of the failure mode is unmistakable: the INT4 ITL curve sits well below FP16 at low concurrency and converges to it somewhere in the 50–100 range, while the INT4 TTFT curve sits *above* FP16 everywhere. If you only ever look at end-to-end throughput at one concurrency level, both effects average into a single number that tells you nothing.\n\nINT4 weight-only quantization doesn't speed up prefill because prefill is compute-bound and W4A16 only reduces memory traffic — the kernel still dequantizes to FP16 and runs the same tensor-core matmuls, so it does the same FLOPs plus unpacking overhead. Decode at low concurrency has an arithmetic intensity near 1 FLOP/byte against a GPU that wants ~295, so removing 75% of the weight bytes gives a near-linear speedup there; prefill runs at hundreds or thousands of FLOP/byte, where those bytes were already free. The crossover is around 74 concurrent decode tokens on an H100, and chunked prefill pushes every batch past it. Use INT4 for single-stream latency and for memory capacity; use FP8 or another activation-quantized scheme when you need prefill and high-concurrency throughput to actually get faster.", "url": "https://wpnews.pro/news/why-int4-weight-only-quantization-doesn-t-speed-up-prefill", "canonical_source": "https://dev.to/ji_ai/why-int4-weight-only-quantization-doesnt-speed-up-prefill-1b45", "published_at": "2026-07-31 20:57:34+00:00", "updated_at": "2026-07-31 21:14:27.282214+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["H100", "A100", "AWQ", "INT4", "BF16"], "alternates": {"html": "https://wpnews.pro/news/why-int4-weight-only-quantization-doesn-t-speed-up-prefill", "markdown": "https://wpnews.pro/news/why-int4-weight-only-quantization-doesn-t-speed-up-prefill.md", "text": "https://wpnews.pro/news/why-int4-weight-only-quantization-doesn-t-speed-up-prefill.txt", "jsonld": "https://wpnews.pro/news/why-int4-weight-only-quantization-doesn-t-speed-up-prefill.jsonld"}}