What Happens in 1 LLM API Call? Measured on GPU With Qwen3–8B

A benchmark by Arun Prabhakaran of LatentAILabs measured a 10-turn DevOps chat on Qwen/Qwen3-8B in bfloat16 on an NVIDIA A100-SXM4-40GB and found that re-prefilling the full transcript processes 5,549 tokens to read a 963-token conversation, 5.55x the 999 tokens a prefix-cached mode computes, with summed prefill time of 868 ms versus 642 ms (1.35x) across the 10 turns. The time saving only appears once the prompt passes roughly 500 tokens, because below that every forward pass costs about 65 ms regardless of length. The author also reported that Qwen3's chat template drops the empty reasoning block when a turn becomes history, breaking the shared prefix 4 tokens before the reply and defeating prefix caching until the benchmark was re-run with a token-by-token prefix check.

By Arun Prabhakaran · @LatentAILabs https://www.youtube.com/@LatentAILabs · Interactive Hub https://latentailabs.github.io/ · Medium https://medium.com/@arunp0 · Hugging Face https://huggingface.co/arunp0 · GitHub https://github.com/latentailabs/episodes/tree/main/ep11-llm-api-call · LinkedIn https://www.linkedin.com/in/arunp0/ Every AI agent framework, whether it is LangGraph, the OpenAI Agents SDK, Claude Code or a custom MCP host, is a loop around one stateless request to an LLM. Before building the loop, it pays to know what happens inside that one request on the GPU. Many agent failures trace back to four facts about that single call: This is Episode 1 of How AI Agents Actually Work . Every number below comes from one setup: Qwen/Qwen3–8B in bfloat16 on an NVIDIA A100-SXM4–40GB PyTorch 2.11.0+cu130 , Hugging Face transformers with eager attention, batch size 1, no CUDA graphs or torch.compile. This is a simple research setup, not a tuned inference server, so absolute latencies would be lower on vLLM or SGLang. The code and raw results are on GitHub https://github.com/latentailabs/episodes/tree/main/ep11-llm-api-call . A chat UI feels like a session that remembers you. The model itself does not. The 15.3 GiB of weights in Qwen/Qwen3-8B are read-only during inference, and nothing about your request is written back into them. Anything that looks like memory is either text your application re-sends or a server-side cache of that text. Here is the demonstration on our GPU : Your application, not the model, keeps the conversation and re-sends the whole transcript on every turn. I timed a 10-turn DevOps conversation in which each turn adds a server log and asks for a one-line summary. By Turn 10 the prompt is 963 tokens long. Two modes: Each number is the median of 5 timed runs of one forward pass plus picking the first token prefill time to first token, which excludes tokenization, scheduling and network : Over all 10 turns, re-prefilling processes 5,549 tokens to read a 963-token conversation, 5.55x the 999 tokens the cached mode computes 868 ms vs 642 ms summed over the 10 turns, 1.35x . The token saving is large, but in this setup the time saving only shows once the prompt passes roughly 500 tokens: below that, every forward pass costs about the same ~65 ms, whatever its length more on that floor in Section 3 . One trap we hit. Qwen3’s chat template ends every prompt with an empty reasoning block, <|im start| assistant\n<think \n\n</think \n\n. When that turn becomes history on the next request, the template writes <|im start| assistant\n<reply instead, without the think block. So the previous prompt and the new prompt stop matching 4 tokens before the reply, and a prefix cache can only reuse the tokens before that point. Our first version of this benchmark missed it. The numbers above come from a re-run that checks the shared prefix token by token, and the cached and full re-prefill runs pick the same next token on every turn. Why is reuse safe at all? Causal attention only lets token i look at tokens 0..i, so appending new tokens never changes the Keys and Values of earlier ones. If the first k token IDs are identical, their cached K/V tensors are what a full re-prefill would compute. In bfloat16, different kernel shapes give small numerical differences: in our run the last-token logits differed by at most 1.19 on logits around 50–60 , and the chosen next token was the same on every turn. This small PyTorch check shows it: python import torch def verify causal prefix kv invariance seq len turn1=16, seq len delta=8, d model=64 : """Single attention layer: attention outputs for the new tokens are the same whether the prefix is recomputed or its K/V are reused from a cache.""" torch.manual seed 42 total = seq len turn1 + seq len delta wq = torch.nn.Linear d model, d model, bias=False wk = torch.nn.Linear d model, d model, bias=False wv = torch.nn.Linear d model, d model, bias=False x prefix = torch.randn 1, seq len turn1, d model x delta = torch.randn 1, seq len delta, d model x full = torch.cat x prefix, x delta , dim=1 Mode 1: recompute everything q full, k full, v full = wq x full , wk x full , wv x full scores full = q full @ k full.transpose -2, -1 / d model 0.5 causal mask = torch.triu torch.full total, total , float "-inf" , diagonal=1 attn full = torch.softmax scores full + causal mask, dim=-1 @ v full Mode 2: reuse cached K/V for the prefix, compute only the new tokens k cached = torch.cat wk x prefix , wk x delta , dim=1 v cached = torch.cat wv x prefix , wv x delta , dim=1 q delta = wq x delta scores delta = q delta @ k cached.transpose -2, -1 / d model 0.5 delta mask = torch.triu torch.full seq len delta, total , float "-inf" , diagonal=seq len turn1 + 1 attn delta cached = torch.softmax scores delta + delta mask, dim=-1 @ v cached max diff = attn full :, seq len turn1:, : - attn delta cached .abs .max .item print f"Max absolute difference, full re-prefill vs cached prefix: {max diff:.2e}" assert max diff < 1e-5 verify causal prefix kv invariance Rule 1 : keep the start of the prompt fixed. A timestamp or request ID at the top of the system prompt changes an early token, and everything after it misses the cache, so the server re-prefills the whole history on every agent step. Put the system prompt and tool definitions first and unchanged, and append anything dynamic at the end. Check your model’s chat template too: as we saw with Qwen3, the template itself can rewrite earlier turns. client.chat.completions.create messages= ... takes a list of role/content dictionaries. The Transformer never sees those dictionaries. Before the first layer runs, the server calls tokenizer.apply chat template , which turns the list into one flat sequence of integer token IDs with special control tokens marking each turn. With enable thinking=False, a system message "Answer with the 3-digit HTTP status code only." and the user question "What is the HTTP status code for Not Found?" become exactly 38 tokens: I asked five short questions with known answers HTTP Not Found → 404, HTTPS port → 443, 17 × 6 → 102, symbol for gold → Au, bits in a byte → 8 , each formatted three ways, greedy decoding, at most 32 new tokens. A reply counts as “clean” only if it is the short answer and then stops. Where does the model look? I recorded the attention weights output attentions=True in all 36 layers for the 38-token prompt above, at the last prompt position 37 , just before the model writes 404. Averaged over all heads and layers, 59.1% of that position's attention lands on the 7 positions that hold the four special-token IDs 151644, 151645, 151667, 151668 . By layer it reaches 73.8–77.3% in layers 7–9 and peaks at 91.4% in layer 30 88.7% in layer 29, 87.9% in layer 32 . Those 7 positions are 18.4% of the prompt, so this is far above an even spread. Two caveats. Position 0 is <|im start| , and the first token of a sequence is a well-known attention sink that soaks up attention in most LLMs Xiao et al., arXiv:2309.17453 , so part of this mass is sink behaviour. And attention weight is not causal influence. Read the result as "the model attends heavily to the template's scaffolding" on this one prompt, not as proof that those tokens steer the answer. Rule 2: always format requests with the model’s own chat template, apply chat template or the server's chat endpoint. Never hand-build prompt strings or reuse another model family's tags. Once the prompt is a sequence of token IDs, the GPU runs the call in two phases. Prefill processes all input tokens in one forward pass and produces the first output token. Decode then produces one token per forward pass, because each new token depends on the previous one. For each prompt length N in {64, 256, 512, 1,024, 2,048}, the script prefills N tokens and then decodes 64 tokens m3 prefill vs decode two phases : Two things stand out. Prefilling 512 input tokens took 78.0 ms, while one output token took 63.2 ms. And at N = 1,024, input went through at 7,672 tokens/s and output at 15.5 tokens/s, 494x slower per token . Why? The textbook answer is the shape of the math. In prefill, every weight matrix multiplies a block of N token vectors at once, so each weight read from memory is reused N times. In decode there is a single new token vector, so each step has to read all 15.3 GiB of weights to do very little arithmetic. Decode is bound by memory bandwidth. Our numbers add a second, setup-specific reason. If decode were limited only by reading the weights, the A100’s 1,555 GB/s of memory bandwidth NVIDIA datasheet would allow about 10 ms per token. We measured about 64 ms, an effective rate of only 237–242 GiB/s, about 255–260 GB/s weights divided by time per token, not a profiled memory rate , roughly 16% of the spec. Decode time also barely moved with context length 64.3 ms at N = 64, 63.3 ms at N = 2,048 , and prefilling 64 or 256 tokens cost about the same as one decode step. So in this Hugging Face eager, batch-1 setup, most of each forward pass is a fixed overhead from launching many small GPU kernels, on top of the weight reads. Serving engines such as vLLM and SGLang cut that overhead with CUDA graphs and batch many users per step, so the exact ratio depends on the stack. The direction does not: per token, decode costs far more than prefill. For a whole call, decode dominates. With a 64-token reply, decode was 96.9% of the call at N = 1,024 and 92.8% at N = 2,048. A 500-token reply at 64 ms per token is about 32 seconds in this setup. Rule 3: budget latency around output length. Keep structured outputs compact, don’t ask for prose fields you don’t need, and turn off long reasoning enable thinking=False on Qwen3 for simple extraction steps. At the end of each decode step, the final hidden state is multiplied by the output matrix lm head, shape 4096, 151936 to give 151,936 logits , one score per vocabulary entry. The sampler divides them by the temperature T and applies softmax. T = 0 means always take the top token. Higher T flattens the distribution so lower-ranked tokens get picked more often. In a chat window, a different word now and then is harmless. In an agent, the reply often goes straight into json.loads or a Pydantic model, and one stray character breaks the pipeline. The task: extract "service" string , "port" integer and "healthy" boolean from 8 server log lines. Each mode ran 16 calls at each of T ∈ {0.0, 0.3, 0.7, 1.0, 1.3} the 8 cases twice, and top p = 0.95 when T 0 , so 240 local generate calls in total. Pass means strict json.loads on the raw reply, the exact three keys, and the right types. Field values were not scored. What does the prompt change? The right panel of Figure 4 shows the raw step-0 logits for the strict prompt. The top candidates are '{\n' 51.0, 97.1% , '{"' 47.5, 2.93% and '{' 39.8 . At rank 4 sits the Markdown fence ' ' with a logit of 36.5. That is 14.5 points below the top token, so its probability is tiny, about 5 in 10 million, but it is not zero. With harder schemas, long inputs or adversarial text in the input, such low-probability tokens can still be sampled. A prompt only shifts the scores. Every token in the 151,936-entry vocabulary stays possible. How do OpenAI Structured Outputs, vLLM guided decoding, Outlines and XGrammar make invalid JSON impossible instead of unlikely? They turn the JSON schema into a grammar a finite-state machine, or a pushdown automaton for nested schemas and plug a logits processor into the decoding loop, just before softmax. At each step, the grammar looks at the text generated so far and works out which tokens can legally come next. Every other token’s logit is set to -inf. Because exp -inf = 0, those tokens have probability zero at any temperature. Our grammar is deliberately simple and hand-written: compact single-line JSON of the form {"service": "<name ", "port": <int , "healthy": <bool }, where <name must be one of the 8 service names in the test set. Here is the step trace from the run constrained step traces while it generated {"service": "auth-api", "port": 8081, "healthy": false}: A minimal logits processor that does the same thing, and a check on the real step-0 logits from our run: python import torchfrom transformers import LogitsProcessor class PrefixGrammarLogitsProcessor LogitsProcessor : """Masks every token that cannot extend the text generated so far into one of allowed strings. Toy version: scans the whole vocab each step.""" def init self, prompt len: int, id to piece: dict, allowed strings: list : self.prompt len = prompt len self.id to piece = id to piece self.allowed strings = allowed strings def call self, input ids: torch.LongTensor, scores: torch.FloatTensor - torch.FloatTensor: gen ids = input ids 0, self.prompt len: .tolist prefix = "".join self.id to piece i for i in gen ids mask = torch.full like scores, float "-inf" for tok id, piece in self.id to piece.items : if any target.startswith prefix + piece for target in self.allowed strings : mask 0, tok id = 0.0 return scores + mask Step-0 top-5 logits from our A100 runvocab = {0: "{\n", 1: '{"', 2: "{", 3: "With the mask, 80 of 80 calls produced valid JSON at every temperature from 0.0 to 1.3 median 2,424–2,472 ms per call . The extra time over the strict prompt comes from our slow Python scanner, which checks the whole vocabulary at every step, and from different output lengths. Production engines such as XGrammar precompute token masks per grammar state, which makes the per-step cost far smaller Dong et al., arXiv:2411.15100 . Two caveats. The mask guarantees form , not truth : if port must be an integer, only digits survive, but an ambiguous prompt can still produce 5432 instead of 5439. And the guarantee only holds if generation finishes inside the token budget. Cut it off early and you get a valid prefix of JSON, not valid JSON. Before Episode 2 wires this call into a tool-calling loop, here are the four rules this run supports: 📺 Watch the full episode: What Happens in 1 LLM API Call? 494x Speed Gap https://youtu.be/qv8PDE7JpXs In Episode 2 of How AI Agents Actually Work, we take this constrained JSON output and wire it into a Python dispatcher to build tool calling from scratch. What Happens in 1 LLM API Call? Measured on GPU With Qwen3–8B https://pub.towardsai.net/what-happens-in-1-llm-api-call-measured-on-gpu-with-qwen3-8b-3654ffb0e290 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.