nvidia-smi said our H100 was 35% busy while users queued for it. Counting every kernel and memory copy found the cause, 643 MB copied off the GPU per request.
- H100 and L4 · llama.cpp and vLLM
- llama.cpp
- vLLM
- GPU utilisation
- CUPTI
We put an LLM server on an H100, pointed sixteen clients at it, and opened nvidia-smi. It said 35% busy. Throughput was 58 tokens per second, and users waited almost two seconds for a first token.
So the card sat idle two thirds of the time while people queued for it. None of the usual tools could say where the rest of the time went.
This is how we found out, and the small tool we built to do it.
The percentage says less than you think #
“GPU busy” sounds like “how hard the GPU is working”. It is not. NVIDIA documents it as the share of each sampling period in which at least one kernel was running. A kernel is one small program the GPU executes, and an inference server launches a steady stream of them.
So 35% only means that for 65% of the time nothing was running at all. It does not say why. It also does not say how much of the card the running kernels used, which matters later in this post.
So we counted instead #
We build Tensward, an open-source tool for analysing and tuning LLM inference, and a percentage was not enough for us. So we wrote a census: a small library that sits inside the inference server and counts what the GPU is asked to do. Every kernel launch and how long it really ran. Every memory copy, how long it took, how many bytes it moved and in which direction.
You do not patch or rebuild the engine. NVIDIA ships a profiling interface, CUPTI, and an environment variable, CUDA_INJECTION64_PATH, that loads a library into any CUDA program. You set it, name an output folder in a second variable, start your server as usual, and the census keeps running totals. Divide those by the number of requests served and you get what one request costs the GPU.
643 megabytes per request #
Our server was llama.cpp and the model was Qwen3.8-27B. It is a hybrid model: besides the usual cache it carries recurrent state, a running memory of the conversation.
The first census report had a number we did not expect. Every request copied 643 MB from the GPU back to host memory. A regular 8B model on the same prompts copied 49 MB.
That pointed at state saving. By default llama-server keeps a prompt cache in host memory and takes checkpoints of each conversation. It is a trade: copy a conversation’s state off the GPU now, and if that conversation comes back, load the copy instead of computing the whole prompt again. The copy is paid on every request, whether or not the conversation ever returns. For this model it came to hundreds of megabytes each time.
The bytes are not the whole cost. We think the copy runs on the one thread that serves every client, so while one conversation is being saved, all sixteen wait. When we sampled that thread, it was inside a memory copy 16 times out of 20.
Two flags switch the saving off:
--cache-ram 0 --ctx-checkpoints 0
Throughput went from 58 to 110 tokens per second, and busy went from 35% to 88%. We did not quite believe it, so we ran it again on fresh machines with a separately written client. The gain was 87%, against 89% and 94% the first time.
One warning before you copy those flags. They also throw away prompt reuse, which is the other side of the trade. On a regular 8B model serving returning conversations (an L4, the older build) they made things about 20% slower. Treat them as a way to test the diagnosis, not as a recipe.
The number that did not move #
Later we tried a newer llama.cpp build (b12246, from 10 November; ours was b11669, from 15 October). With default settings it ran the same workload 67 to 75% faster. A benchmark would stop there: new version, faster.
The census had more to say. Bytes copied per request were 643 MB before and 644 MB after, so the copy was just as big. But kernels now ran 45% of the time instead of 30%. The copy had apparently become cheaper, not smaller.
That fits what happened upstream. Between the two builds, three merged pull requests changed how state is copied and stored: #30881, #31391 and #30137. A user had already reported a slow state save in issue #30252, in a different setup from ours. We contributed to none of these and did not build the commits in between, so which change explains the gain is our guess.
Busy is not full, and busy is not always work #
Two more cases where the percentage misled.
On an L4, llama.cpp served a mixture-of-experts model with part of its weights left on the CPU. nvidia-smi read 55% busy. The census said kernels ran 25% of the time and copies of weights from the CPU took about 30%. DCGM, NVIDIA’s data-center monitoring tool, also read 25%. Our reading is that the percentage counted copy time as busy, which its documentation does not say. We did not prove it.
The other direction: with vLLM on an H100, nvidia-smi read 95 to 97% busy, while DCGM’s SM occupancy (DCGM_FI_PROF_SM_OCCUPANCY) was 8 to 12%. Something was always running, and it used a small part of the card. The census cannot see this one yet. It would only have said “busy”, and we want it to report this too.
What it costs, and what it cannot do yet #
The census is research tooling today, not a finished product. Here is what we know about it.
- Cost. On an L4, recording cost about 4% of throughput on llama.cpp. On vLLM it was well under 1% for an 8B model and about 4% for a tiny model under heavy load. For an embeddings server we could not show it was cheap: throughput barely moved, but tail latency was too noisy to tell. We have not measured it on an H100 yet. For that reason every throughput number in this post comes from runs without the census; the bytes and kernel-time figures come from a separate run of the same load with it loaded.
- Crashes. With the hybrid model on the older llama.cpp build, the first version of the census crashed or hung the server in 5 of 79 test runs, and in every run that put four servers behind a proxy. On the newer build there were none in 84 runs, all on one H100 with one model. We did not find the cause, and we have not re-measured the current version.
- Engines. It has been attached, with no change to the engine, to llama.cpp, vLLM 0.30.0, Ollama 0.40.15 and SGLang 0.5.21. The last two we tested on an L4 only.
- Other tools. NVIDIA’s nsys profiler can report copied bytes too, and we used it to check our figure (it said 639 MB). It needs a separate trace session, though. The census counts while the server runs normally, which is how we noticed the copy in the first place.
What is next #
The census is planned to become part of Tensward 0.5. There is no date yet.