Same A100, similar throughput: would both pass a 1-second first-token target?

A Hugging Face forum benchmark by Exulux found that two A100 SXM 80 GB rentals running Llama 3.1 8B Instruct in BF16 at concurrency 32 delivered nearly identical throughput of about 1,135–1,139 output tokens/s across six runs, but diverged sharply on first-token p95 latency: 1,251–1,261 ms on RunPod Secure in Kansas versus 752–990 ms on Massed Compute via Shadeform in Des Moines. Against a 1,000 ms first-token target, the Des Moines allocation passed all three runs and the Kansas allocation missed all three, with the slowest passing run holding only about 10 ms of headroom. Exulux noted the results are same-instance measurements covering one allocation each with different hosts, regions and drivers, so application and network latency remain untested.

Exulux https://discuss.huggingface.co/u/Exulux 1 We tested Llama 3.1 8B Instruct on two A100 SXM 80 GB rentals: RunPod Secure in Kansas and Massed Compute via Shadeform in Des Moines. Both ran BF16 at concurrency 32, with 2,000 input tokens and 500 requested output tokens. Throughput was about 1,135–1,139 output tokens/s across the six measured runs. First-token p95 was 1,251–1,261 ms on the Kansas allocation and 752–990 ms on the Des Moines allocation. At a 1,000 ms target, that difference matters: one allocation passed all three runs, and the other missed all three. The slowest passing run had only about 10 ms of headroom. These were same-instance measurements, so application and network latency still need to be tested. Each result represents one allocation, with different hosts, regions and drivers. The hourly quotes in the case study are historical. Full results, method and limitations https://exuluxcompute.com/compare/runpod-vs-shadeform-a100-llama-3-1-8b/ If you’re sizing a deployment now, what first-token p95 limit and peak concurrency does the GPU need to handle? DHEIVER https://discuss.huggingface.co/u/DHEIVER 2 @Exulux https://discuss.huggingface.co/u/exulux The variance in first-token latency 1.25s vs 0.75s despite identical throughput points to host-level scheduling contention. During the prefill phase, the system is memory-bound and sensitive to OS scheduling noise or hypervisor throttling. Even with GPU compute saturated, delays in memory allocation or context switching directly impact the time to first token, while the decode phase masks this instability until context fills. The issue lies in how the host manages memory pressure during the prefill spike, not the GPU’s raw throughput.