LLM Inference Latency: Why Your 7B Model Gets 15 tok/s on a T4 but 3,500 tok/s on an H100

`NVIDIA's spec sheet says the H100 delivers 989 TFLOPS of FP16 compute. The A100: 312. The T4: 65. Simple arithmetic says the H100 is 15× faster.

So a 7-billion-parameter LLM should be 15× faster on an H100, right?

It's 150× faster.

The T4 struggles at ~15 tok/s. The H100 cruises at ~2,200 tok/s — and with continuous batching, north of 3,500.

The 15× TFLOPS gap doesn't explain the 150× throughput gap. The missing variable is the one thing NVIDIA's marketing pages bury on line three of the spec table:

Memory bandwidth.

The T4 has 300 GB/s. The H100 has 3,350 GB/s. That's an 11× gap in raw bandwidth, and closer to 100–150× in effective throughput once cache size and clock speed are factored in.

This post traces the arithmetic from first principles: why memory bandwidth is the bottleneck, how quantization turns it into a lever, and what tok/s you should actually expect across 12 real GPU × model combinations.

Autoregressive LLM inference has two phases:

The decode phase dominates total latency for any response longer than a few tokens. And in decode, the GPU spends 98%+ of its time doing nothing — waiting for the next chunk of model weights to arrive from VRAM.

Each generated token requires reading every single parameter from VRAM. For a 7B model at FP16:

``plaintext`

Weights: 7,000,000,000 params × 2 bytes = 14 GB per token

H100: 14 GB ÷ 3,350 GB/s = 4.18 ms → 239 tok/s (theoretical)

T4: 14 GB ÷ 300 GB/s = 46.7 ms → 21 tok/s (theoretical)

`

Now consider the compute cost: ~14 TFLOPs per token. The H100's 989 TFLOPS could execute that 70 times in 4.18 ms. The compute units finish their work and sit idle, waiting for the next 14 GB of weights to trickle in.

You cannot compute faster than you can read.

Every meaningful LLM optimization — quantization, KV cache compression, FlashAttention, speculative decoding — is fundamentally about reducing bytes moved per token.

Theoretical tok/s at 100% bandwidth utilization. Real-world: multiply by 0.6–0.85. "—" = doesn't fit in VRAM.

If your bottleneck is bytes-per-token, and you can't change the GPU's memory bandwidth, there's exactly one lever left: reduce bytes per parameter.

The rule of thumb: the larger the model, the harder it is to break with quantization. A 405B model at INT2 often outperforms a 70B model at FP16 on knowledge tasks — despite using fewer bytes per token.

What quantization doesn't improve: time-to-first-token (prefill latency). Prefill is compute-bound. Quantization can even make it slower due to dequantization overhead. The gains apply almost entirely to the decode phase.

Theoretical ceilings are clean math. Reality includes framework overhead, attention kernel efficiency, and KV cache management. These numbers are from vLLM 0.6.x, continuous batching, bare-metal H100 instances — mid-2026.

| Model | Precision | GPU(s) | Batch 1 | Batch 8 | Batch 32 |

|---|---|---|---|---|---|
Llama 4 Scout (8B) |

FP16 | 1× H100 | 185 | 1,200 | 3,200 | Llama 4 Scout (8B) | INT4 | 1× H100 | 620 | 3,800 | 8,500 | Llama 4 Scout (8B) | INT4 | 1× RTX 4090 | 95 | 310 | — | Mistral Small 3 (7B) | FP16 | 1× H100 | 195 | 1,250 | 3,500 | Llama 4 Maverick (70B) | FP16 | 2× H100 | 35 | 190 | 480 | Llama 4 Maverick (70B) | INT4 | 1× H100 | 78 | 440 | 1,050 | | Mistral Large 2 (123B) | FP16 | 4× H100 | 15 | 70 | 140 | | DeepSeek-V3 (671B MoE) | INT8 | 8× H100 | 18 | 85 | 180 | | Qwen 2.5 (72B) | INT4 | 1× H100 | 72 | 410 | 960 | Phi-4 (14B) | INT4 | 1× RTX 4090 | 78 | — | — | Gemma 3 (27B) | INT4 | 1× RTX 4090 | 55 | — | — |

"TP=N" = Tensor Parallelism across N GPUs. "—" = VRAM insufficient at listed context length. Output token decode only.

The headline: INT4 turns a 70B model from "needs 2 GPUs and is still slow" into "runs on 1 GPU and is faster than FP16 on 2."

Batching fills the idle time between memory reads. While one request waits for weights, another's compute can run. But per-request latency climbs with batch size.

Continuous batching(vLLM, TensorRT-LLM, SGLang) is the critical innovation. Traditional static batching waits forallrequests to finish. Continuous batching evicts completed requests and admits new ones atevery decode step— eliminating the "one slow request holds up 31 others" problem.

If you deploy without continuous batching, you're leaving 50–70% of your GPU throughput on the table.

When one GPU isn't enough, you split the model with tensor parallelism. But all-reduce communication between GPUs eats into your gains:

| Configuration | Batch 32 tok/s | Per-GPU Efficiency | NCCL Tax |

|---|---|---|---|
| 2× A100 (TP=2) | 380 | 92% |

~8% | | 4× A100 (TP=4) | 610 | 80% | ~20% | | 8× A100 (TP=8) | 840 | 63% | ~37% |

At TP=8, nearly 40% of interconnect bandwidth goes to synchronization.

This is why expert-parallel inference for MoE models is revolutionary: each expert lives on a dedicated GPU, and only the active experts participate in each forward pass. DeepSeek-V3 has 671B total parameters but only 37B active — so it's faster than a 70B dense model despite having 10× more total weights.

LLM inference throughput = GPU memory bandwidth ÷ bytes per token. Everything else is overhead management.

Try the interactive calculator at jslet.com/llm-inference-latency — plug in your model, GPU, and quantization level. All 100% client-side, no signup.