Oh. It depends on the backend you’re using, but that seems to be about the expected speed:
482 tokens in about 12 seconds works out to roughly 40 tokens/s. If those 12 seconds are the actual generation/decode time, that number does not look suspiciously high for Qwen3.8-27B on a 3090.
As a sanity check rather than an exact apples-to-apples benchmark, the current crowd-sourced llamabench.ai results for Qwen3.8-27B on an RTX 3090 include non-speculative runs at 38.2, 40.0, 41.1, and 45.6 tok/s. Those runs use llama.cpp and somewhat different quant/cache settings, so I would not treat them as a reproduction of your Ollama run, but your ~40.2 tok/s sits right in that range.
The cheapest way to verify what you are seeing is probably:
ollama run <your-model> --verbose
ollama ps
For the first command, the useful lines are approximately:
prompt eval count: ...
prompt eval duration: ...
prompt eval rate: ...
eval count: ...
eval duration: ...
eval rate: ... tokens/s
Ollama’s API usage documentation makes the distinction explicit:
prompt_eval_*
is processing the input prompt.eval_count
is the number of generated output tokens.eval_duration
is the time spent generating those output tokens.total_duration
also includes other work such as model and prompt processing.So if Ollama itself reports an eval rate
around 40 tok/s, there is not much ambiguity left: you really are getting about 40 generated tokens/s.
ollama ps
gives the other high-value check. According to the Ollama FAQ, 100% GPU
means the model is loaded entirely on the GPU, while a CPU/GPU percentage indicates that it is split between system RAM and VRAM.
A useful decision tree is therefore:
Is ~12 s the generation time / Ollama eval_duration?
|
+-- Yes
| |
| +-- ~482 / ~12 s = ~40 tok/s
| |
| +-- ollama ps says 100% GPU
| | |
| | +-- This looks quite normal for this model/GPU class.
| |
| +-- CPU/GPU split
| |
| +-- Then context size/off becomes important
| before comparing the number with other benchmarks.
|
+-- No / not sure
|
+-- Use --verbose (or the API metrics) once and compare eval_rate,
rather than dividing visible tokens by wall-clock time.
In other words, I would not worry that the 3090 has somehow produced an impossible result. Its age is a bit misleading here: 24 GB of VRAM is still a very useful configuration for local inference, because a quantized model of this size can fit without having to spill a large part of it into CPU memory.
So, for the result you posted, my default interpretation would simply be:
yes, ~40 tok/s is believable.
If ollama run ... --verbose
reports roughly that eval rate
and ollama ps
says 100% GPU
, I would consider the basic mystery solved. The fact that a 3090 is an older card does not prevent it from still being a very capable local-LLM card when a ~27B quantized model fits comfortably inside its 24 GB VRAM.