# LLM Inference Latency: Why Your 7B Model Gets 15 tok/s on a T4 but 3,500 tok/s on an H100 > Source: > Published: 2026-07-13 02:16:01+00:00 `NVIDIA's spec sheet says the H100 delivers **989 TFLOPS** of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster. So a 7-billion-parameter LLM should be 15× faster on an H100, right? **It's 150× faster.** The T4 struggles at **~15 tok/s**. The H100 cruises at **~2,200 tok/s** — and with continuous batching, north of **3,500**. The 15× TFLOPS gap doesn't explain the 150× throughput gap. The missing variable is the one thing NVIDIA's marketing pages bury on line three of the spec table: Memory bandwidth. The T4 has 300 GB/s. The H100 has 3,350 GB/s. That's an 11× gap in raw bandwidth, and closer to 100–150× in effective throughput once cache size and clock speed are factored in. This post traces the arithmetic from first principles: *why* memory bandwidth is the bottleneck, *how* quantization turns it into a lever, and *what* tok/s you should actually expect across 12 real GPU × model combinations. Autoregressive LLM inference has two phases: | Phase | What Happens | Bottleneck | |---|---|---| Prefill | Process the input prompt (all tokens in parallel) | Compute | Decode | Generate output tokens (one at a time) | Memory bandwidth | The decode phase dominates total latency for any response longer than a few tokens. And in decode, the GPU spends 98%+ of its time doing nothing — waiting for the next chunk of model weights to arrive from VRAM. Each generated token requires reading **every single parameter** from VRAM. For a 7B model at FP16: ``plaintext` Weights: 7,000,000,000 params × 2 bytes = 14 GB per token H100: 14 GB ÷ 3,350 GB/s = 4.18 ms → 239 tok/s (theoretical) T4: 14 GB ÷ 300 GB/s = 46.7 ms → 21 tok/s (theoretical) ` Now consider the compute cost: ~14 TFLOPs per token. The H100's 989 TFLOPS could execute that **70 times** in 4.18 ms. The compute units finish their work and sit idle, waiting for the next 14 GB of weights to trickle in. You cannot compute faster than you can read. Every meaningful LLM optimization — quantization, KV cache compression, FlashAttention, speculative decoding — is fundamentally about **reducing bytes moved per token**. | GPU | Bandwidth | 7B FP16 | 7B INT4 | 70B INT4 | VRAM | |---|---|---|---|---|---| NVIDIA B200 | 8,000 GB/s | 571 | 2,286 | 229 | 192 GB | NVIDIA H200 | 4,800 GB/s | 343 | 1,371 | 137 | 141 GB | NVIDIA H100 | 3,350 GB/s | 239 | 957 | 96 | 80 GB | NVIDIA A100-80GB | 2,039 GB/s | 146 | 583 | 58 | 80 GB | NVIDIA RTX 4090 | 1,008 GB/s | 72 | 288 | 28.8 | 24 GB | NVIDIA L40S | 864 GB/s | 62 | 247 | — | 48 GB | NVIDIA T4 | 300 GB/s | 21 | 86 | — | 16 GB | Theoretical tok/s at 100% bandwidth utilization. Real-world: multiply by 0.6–0.85. "—" = doesn't fit in VRAM. If your bottleneck is bytes-per-token, and you can't change the GPU's memory bandwidth, there's exactly one lever left: **reduce bytes per parameter**. | Precision | Bytes/Param | 7B Model | 70B Model | Speedup | Quality | |---|---|---|---|---|---| | FP16 | 2.0 | 14 GB | 140 GB | 1× | Reference | | INT8 | 1.0 | 7 GB | 70 GB | ~1.8× | Negligible | INT4 (GPTQ/AWQ) | 0.5 | 3.5 GB | 35 GB | ~3.5× | 1–3% perplexity | | INT3 | 0.375 | 2.6 GB | 26 GB | ~4.5× | 3–8% loss | **The rule of thumb:** the larger the model, the harder it is to break with quantization. A 405B model at INT2 often outperforms a 70B model at FP16 on knowledge tasks — despite using fewer bytes per token. **What quantization doesn't improve:** time-to-first-token (prefill latency). Prefill is compute-bound. Quantization can even make it *slower* due to dequantization overhead. The gains apply almost entirely to the decode phase. Theoretical ceilings are clean math. Reality includes framework overhead, attention kernel efficiency, and KV cache management. These numbers are from **vLLM 0.6.x**, continuous batching, bare-metal H100 instances — mid-2026. | Model | Precision | GPU(s) | Batch 1 | Batch 8 | Batch 32 | |---|---|---|---|---|---| Llama 4 Scout (8B) | FP16 | 1× H100 | 185 | 1,200 | 3,200 | Llama 4 Scout (8B) | INT4 | 1× H100 | 620 | 3,800 | 8,500 | Llama 4 Scout (8B) | INT4 | 1× RTX 4090 | 95 | 310 | — | Mistral Small 3 (7B) | FP16 | 1× H100 | 195 | 1,250 | 3,500 | Llama 4 Maverick (70B) | FP16 | 2× H100 | 35 | 190 | 480 | Llama 4 Maverick (70B) | INT4 | 1× H100 | 78 | 440 | 1,050 | | Mistral Large 2 (123B) | FP16 | 4× H100 | 15 | 70 | 140 | | DeepSeek-V3 (671B MoE) | INT8 | 8× H100 | 18 | 85 | 180 | | Qwen 2.5 (72B) | INT4 | 1× H100 | 72 | 410 | 960 | Phi-4 (14B) | INT4 | 1× RTX 4090 | 78 | — | — | Gemma 3 (27B) | INT4 | 1× RTX 4090 | 55 | — | — | "TP=N" = Tensor Parallelism across N GPUs. "—" = VRAM insufficient at listed context length. Output token decode only. **The headline:** INT4 turns a 70B model from "needs 2 GPUs and is still slow" into "runs on 1 GPU and is faster than FP16 on 2." Batching fills the idle time between memory reads. While one request waits for weights, another's compute can run. But per-request latency climbs with batch size. | Workload | Batch | Latency Target | |---|---|---| | Real-time chat / copilot | 1–4 | <200ms TTFT | | Code completion | 1–2 | <20ms/tok | | Support chatbot | 4–16 | <1s TTFT | | Summarization (async) | 16–32 | Throughput priority | | Dataset labeling | 32–128 | Throughput only | Continuous batching(vLLM, TensorRT-LLM, SGLang) is the critical innovation. Traditional static batching waits forallrequests to finish. Continuous batching evicts completed requests and admits new ones atevery decode step— eliminating the "one slow request holds up 31 others" problem. **If you deploy without continuous batching, you're leaving 50–70% of your GPU throughput on the table.** When one GPU isn't enough, you split the model with **tensor parallelism**. But all-reduce communication between GPUs eats into your gains: | Configuration | Batch 32 tok/s | Per-GPU Efficiency | NCCL Tax | |---|---|---|---| | 2× A100 (TP=2) | 380 | 92% | ~8% | | 4× A100 (TP=4) | 610 | 80% | ~20% | | 8× A100 (TP=8) | 840 | 63% | ~37% | **At TP=8, nearly 40% of interconnect bandwidth goes to synchronization.** This is why **expert-parallel** inference for MoE models is revolutionary: each expert lives on a dedicated GPU, and only the *active* experts participate in each forward pass. DeepSeek-V3 has 671B total parameters but only 37B active — so it's faster than a 70B dense model despite having 10× more total weights. LLM inference throughput = GPU memory bandwidth ÷ bytes per token. Everything else is overhead management. *Try the interactive calculator at jslet.com/llm-inference-latency — plug in your model, GPU, and quantization level. All 100% client-side, no signup.*