# How many tokens per second should an LLM serve on one GPU?

> Source: <https://dev.to/viswaretas_kotra/how-many-tokens-per-second-should-an-llm-serve-on-one-gpu-27b8>
> Published: 2026-11-18 21:35:18+00:00

**Recommendation:** Size capacity by total system throughput at your real prompt lengths, not single-stream speed, and batch requests until per-user output drops to your 30-50 tokens-per-second chat floor.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

One NVIDIA H100 serving an 8B model at full load produces about 1,300 to 26,000 output tokens per second in total, depending on prompt length, while any single user sees only a few dozen to a few hundred, capped by memory bandwidth. On an H100 SXM listed at $2.60/hr on the Nodus pricing page, NVIDIA's published 14,992 tokens per second for Llama 3.1 8B works out to about $0.05 per million output tokens, and serving one user at a time costs about 72 times more. So the right target is not one number: set a per-user floor for chat (around 30 to 50 tokens per second), then pack in as many users as the GPU allows while that floor holds.

It measures two different things, and most confusion comes from mixing them. NVIDIA's [LLM benchmarking metrics guide](https://docs.nvidia.com/nim/benchmarking/llm/metrics) defines both.

Total tokens per second (TPS) per system is all output tokens across every simultaneous request, divided by the benchmark's wall time. This is what decides your cost.

TPS per user is one request's output length divided by its end-to-end latency. It approaches 1 divided by the inter-token latency (ITL), the gap between consecutive tokens. This is what a person watching a chat window feels.

The guide states the tradeoff plainly: as concurrent requests increase, total system TPS rises while TPS per user falls. Two more numbers complete the picture:

| Metric | What it tells you | Who cares | 
|---|---|---|
| Time to first token (TTFT) | Queueing plus prefill plus network before the first token | Chat users, voice agents | 
| Inter-token latency (ITL) | Time between tokens while streaming | Chat users | 
| TPS per user | Output speed for one request | Chat, agents waiting on a reply | 
| Total TPS per system | Output across all requests | Batch jobs, whoever pays the bill | 

Definitions from [NVIDIA's metrics guide](https://docs.nvidia.com/nim/benchmarking/llm/metrics). Tools differ on whether TTFT is included in ITL, so compare numbers only when the definitions match.

Generating each new token means reading every model weight from GPU memory once. With a single request, there is nothing else to share that read with, so the ceiling is roughly memory bandwidth divided by the bytes of weights. Compute barely matters at this point.

Here is that ceiling worked out for Llama 3.1 8B at 16-bit precision, about 16 GB of weights, using bandwidth from NVIDIA's own spec pages:

Bandwidth figures are from NVIDIA's pages for the [L4](https://www.nvidia.com/en-us/data-center/l4/), [L40S](https://www.nvidia.com/en-us/data-center/l40s/), [A100](https://www.nvidia.com/en-us/data-center/a100/), [H100](https://www.nvidia.com/en-us/data-center/h100/) and [H200](https://www.nvidia.com/en-us/data-center/h200/). These are upper bounds we calculated, not measurements. Real decoding also reads the KV cache and pays kernel overhead, so expect less.

*Single-stream speed tracks memory bandwidth, so an H200 can stream one user about 16 times faster than an L4.*

The same arithmetic gives two quick rules. Halving the bytes per weight, for example by serving in 8-bit (FP8), doubles the ceiling: 418.75 tokens per second for the 8B model on an H100. A 70B model in FP8 is about 70 GB, which leaves an H100 a ceiling of 3,350 / 70 = 47.9 tokens per second for one user, before any KV cache traffic.

Far more than the single-user ceiling, because a batch of requests shares each read of the weights. NVIDIA's [TensorRT-LLM performance overview](https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html) publishes total output throughput at maximum load. For Llama 3.1 8B in FP8 on one H100 SXM:

| Input, output tokens per request | Output tokens per second, one H100 | 
|---|---|
| 128, 128 | 26,401 | 
| 1,000, 1,000 | 14,992 | 
| 5,000, 500 | 3,276 | 
| 20,000, 2,000 | 1,341 | 

Source: [TensorRT-LLM performance overview](https://nvidia.github.io/TensorRT-LLM/performance/perf-overview.html), throughput tables, checked November 18, 2026. NVIDIA describes these as reference points, not peak performance.

*Long prompts cut the same GPU's output throughput by up to 20 times, so ask any benchmark what input length it used.*

Three things in that table matter more than the headline number. Prompt length dominates: long inputs spend GPU time on prefill and fill memory with KV cache, so fewer requests fit in a batch. Bigger models scale down fast: the same page lists Llama 3.3 70B FP8 at 4,181 tokens per second on two H100s at 1,000 in and 1,000 out, or about 2,091 per GPU. A faster GPU helps less than its spec suggests: the H200 row for 8B at 1,000 and 1,000 is 17,162, only 14% above the H100.

These are maximum-throughput runs, and they hide the user's experience. The example output on the same page reports 5,531 total output tokens per second alongside a per-user output speed of 5.47 tokens per second and an average TTFT of 52.7 seconds. That is a fine result for an overnight batch job and an unusable one for chat. NVIDIA's [parameters guide](https://docs.nvidia.com/nim/benchmarking/llm/parameters) explains why: throughput saturates near the maximum batch size while latency keeps rising.

For one person reading a streamed chat reply, yes. At NVIDIA's [rule of thumb](https://docs.nvidia.com/nim/benchmarking/llm/overview) of about 0.75 English words per token, 40 tokens per second is 30 words per second, with 25 ms between tokens. That is comfortably faster than people read, so pushing a chat endpoint much higher per user mostly costs money.

It is not good in two other cases. Reasoning models and agents generate tokens nobody reads: a 2,000-token reasoning trace at 40 tokens per second is 50 seconds of waiting before the answer starts. For those, per-user speed is latency, and faster is worth paying for.

It is also a poor number if it is your whole GPU. One H100 at $2.60/hr producing 40 tokens per second costs $2.60 x 1,000,000 / (40 x 3,600) = $18.06 per million output tokens. The same GPU at NVIDIA's 14,992 tokens per second costs $0.05.

The decision rule we use:

Divide the hourly price by the tokens produced in an hour. We use the H100 SXM listed at $2.60/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/), checked November 18, 2026, with NVIDIA's published 8B throughput:

*Batching, not the hourly rate, sets the price: the same H100 hour costs 72 times more per token when it serves one user.*

The 70B case costs more because it needs two GPUs: two H100s at $5.20/hr and 4,181 tokens per second come to $0.35 per million output tokens. The H200, listed at $3.593/hr, comes to $0.058 at 17,162 tokens per second, slightly more per token than the H100 here because its price rises faster than this throughput.

These figures assume the GPU stays saturated and count output tokens only. Real traffic has idle hours and input tokens. Our post on [LLM inference cost per million tokens](https://www.nodus-compute.ai/blog/llm-inference-cost-per-token/) covers input, cached and reasoning token pricing, and [whether self-hosting is cheaper than an API](https://www.nodus-compute.ai/blog/self-hosting-llm-vs-api-cost/) works through break-even volume and how busy the GPU must be.

Run a concurrency sweep with your real prompt and output lengths, not synthetic 128-token prompts. NVIDIA's [parameters guide](https://docs.nvidia.com/nim/benchmarking/llm/parameters) recommends controlling load by concurrency, the number of requests in flight, and recording TTFT, ITL and total TPS at each level. The useful point is the highest concurrency where your per-user floor and TTFT limit still hold. Total TPS at that point, divided into the hourly price, is your real cost per token.

To stand up the server itself, our guide to [hosting your own LLM behind an OpenAI-style API](https://www.nodus-compute.ai/blog/host-your-own-llm-api/) has the vLLM commands, and [vLLM vs SGLang](https://www.nodus-compute.ai/blog/vllm-vs-sglang/) compares the two engines. On Nodus, a [Sweep](https://www.nodus-compute.ai/docs/guides/sweeps/) runs the same Job across GPU types and parameter values, then reports each cell's cost, wall time and throughput, so a single `nodus apply -f sweep.yaml` can be set up to compare, say, an L40S against an H100 at several concurrency levels under a `maxCostUSD` cap. We have not published such a sweep ourselves yet. Throughput there counts the units your program reports, so log one unit per request and compute tokens per second from your benchmark client.

They usually differ in input and output length, precision (16-bit, FP8, FP4), engine, concurrency and whether they report per-user or total throughput. NVIDIA's own H100 table spans 1,341 to 26,401 tokens per second for one model on one GPU just by changing lengths.

In NVIDIA's published numbers, yes. The same TensorRT-LLM page lists Llama 3.3 70B in FP4 at 6,434 output tokens per second on one B200 at 1,000 in and 1,000 out. At the B200's listed $5.50/hr on Nodus, that is $0.24 per million output tokens, against $0.35 for FP8 on two H100s, provided FP4 quality passes your evals.

For a small model and few users, yes, if the model is quantized. The 16 GB 8B model has an 18.75 tokens per second single-user ceiling on an L4, below a comfortable chat floor, while 8-bit weights double that to 37.5. The L4 is listed at $0.430/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/).

*Originally published on the [Nodus blog](https://www.nodus-compute.ai/blog/llm-tokens-per-second/).*
