Paste a long document into ChatGPT and hit enter.
Nothing happens for a second or two. Then the answer starts appearing, word by word, at a steady pace until it finishes.
You have seen this hundreds of times. Most people never think about it.
But those are two completely different things happening inside one request, running on two different parts of the same GPU, limited by two different bottlenecks. Once you know what they are, a lot of confusing things about serving AI models stop being confusing.
Including why a faster GPU sometimes makes no difference at all.
This is part two of a series. Part one covered the split between VRAM and cores. This post covers what each of them does when a request arrives.
I am going to trace one question the whole way through:
what is the capital of france
Simple, short, and it produces a short answer. Perfect for watching the machinery.
Before anything happens, the model server has already loaded the model's weights into VRAM. That happened at startup and they stay there. Nothing about your request loads a model.
Your words get chopped into tokens, which are just pieces of words turned into numbers, and placed in VRAM alongside the weights.
Now the work begins.
Before the model can write anything, it has to read everything you gave it.
Here is the important part, and it is the opposite of what most people assume: it reads your entire prompt at once. Not word by word. All six words go to the cores together, in a single pass.
That is possible because your whole prompt is already there. Nothing has to wait for anything else. So thousands of cores fire simultaneously and chew through all of it in one burst.
flowchart LR
A["what is the capital of france<br/>all 6 tokens together"] --> C
W["16 GB of model weights"] --> C
C["GPU CORES<br/>every core busy"] --> O1["First word: 'The'"]
C --> O2["Notes saved to VRAM<br/>the KV cache"]
Two things come out of that single pass:
The moment that first word appears is the moment the ends.
How long is the ? It depends entirely on how much you gave it to read. Six words is nothing, so you would not notice. Paste a hundred-page contract and that is tens of thousands of tokens pushed through in one burst, and now the is real.
This phase has a name: prefill. And the has a name too: TTFT, time to first token. It is one of the two numbers every serving benchmark reports.
What is the bottleneck here? The cores. There is a mountain of maths to do and they are all doing it. This phase is compute-heavy.
Now the model writes the rest of the answer. One word per pass.
To produce each word, the cores need three things: the notes from before, the word that was just produced, and the model's weights. All of it read out of VRAM.
flowchart LR
N["Notes in VRAM<br/>the KV cache"] --> C
L["The word just produced"] --> C
W["16 GB of weights<br/>read again, in full"] --> C
C["GPU CORES<br/>barely used"] --> O["One new word"]
O -.->|"repeat"| N
That loop runs once per word until the answer is done:
Lap 1 → "The"
Lap 2 → "capital"
Lap 3 → "is"
Lap 4 → "Paris"
This is why the answer types itself out instead of appearing all at once. You are watching each lap finish. Streaming is not a feature someone built. It is just what happens when each word is sent the moment it exists.
This phase is called decode, and the gap between words is TPOT, time per output token. That is the second benchmark number.
What is the bottleneck here? Not the cores. Look at what has to happen for a single word: the entire model, all 16 GB of it, has to travel out of VRAM and past the cores. The cores glance at it, do a small amount of maths, and produce one word.
Then it all happens again for the next word.
The cores are mostly idle. The memory pipe is flat out. Decode is limited by how fast the GPU can read its own memory.
Here is the question I got stuck on, and it is worth its own section.
To predict word 3, the model needs to know everything before it. Your prompt, plus word 1, plus word 2. So does it read all of that again?
If it did, every single word would have its own . The whole thing would be unusable.
It does not, because of the notes.
Every word that gets produced is added to the notes as it goes. So by the time lap 3 starts, the notes already contain your prompt, "The", and "capital". The model reads those notes and only has to process the one word that just came out.
Lap 3 reads: notes (prompt + "The" + "capital") + the word "capital"
Lap 3 does: process one word
Lap 3 gives: "is"
Nothing gets forgotten and nothing gets reprocessed. The model always sees the full history. It just never redoes the work.
You pay for the once, not once per word.
Here is where this gets practical.
Every word requires one full read of the model out of VRAM. So the maximum speed of a model on a given card is a division:
How fast can the card read its own memory?
───────────────────────────────────────── = words per second
How much is there to read?
Let me do it with real numbers.
Take an AWS g6e.xlarge
. The GPU inside is a single NVIDIA L40S with 48 GB of VRAM. NVIDIA's datasheet gives its memory bandwidth as 864 GB/s.
Now put a model on it. Qwen3.8-27B is a 28-billion-parameter model. In its published form each weight takes 2 bytes, so:
28 billion × 2 bytes = 56 GB
That does not fit in 48 GB. So we use the Quantization version FP8 , where each weight takes 1 byte:
28 billion × 1 byte = 28 GB
Now the division:
864 GB/s ÷ 28 GB ≈ 31 words per second
That is the ceiling. Two numbers off two spec sheets, and you know roughly how fast that model will generate on that card. No deployment required to calculate this.
And notice what shrinking the model did. At 56 GB the ceiling would have been about 15 words per second. Halving the weights doubled the speed, because there is half as much to read every single lap.
A caveat worth stating plainly: that number is a theoretical ceiling for one user, and real throughput lands meaningfully below it. The notes have to be read too, and no software achieves perfect memory utilisation. Treat it as a sanity check, not a promise.
Here is something that caught me out, and I suspect it catches out a lot of people.
That bandwidth figure, the number that decides your generation speed, is not on AWS GPU Instance page.
Look at an EC2 GPU instance table and you get the card name, GPU memory, vCPUs, system memory, and storage. Some tables have a column called "EBS Bandwidth" and another called "Network Bandwidth", which look like the answer and are not. Those are disk and network throughput. Neither has anything to do with VRAM.
The three bandwidths worth separating:
| Bandwidth | Between what | Typical scale |
|---|---|---|
| Memory bandwidth | VRAM and cores | hundreds of GB/s |
| PCIe | GPU card and server | tens of GB/s |
| Network / EBS | Server and the outside world | Gbps |
Only the first one governs how fast words come out, and AWS does not publish it.
So the workflow is: read the card name off the AWS table, then go to NVIDIA's datasheet for that card and find the field labelled Memory Bandwidth.
You might reasonably be looking at "31 words per second" and thinking that sounds poor for a server. And it would be, if that was all it could do.
But look again at what happens during decode. The GPU reads the entire model out of VRAM to produce one word for one person, and the cores sit mostly idle while it happens.
So why not use that same read to produce a word for fifty people at once?
That is exactly what real serving software does, and it is why the economics of hosting a model work at all. It also introduces a new limit, a different kind of bottleneck, and the reason ordinary load balancing falls apart completely for AI workloads.
That will cover in upcoming posts.