By Arun Prabhakaran · @LatentAILabs · Interactive Hub · Medium · Hugging Face · GitHub · LinkedIn
Every AI agent framework, whether it is LangGraph, the OpenAI Agents SDK, Claude Code or a custom MCP host, is a loop around one stateless request to an LLM. Before building the loop, it pays to know what happens inside that one request on the GPU.
Many agent failures trace back to four facts about that single call:
This is Episode 1 of How AI Agents Actually Work. Every number below comes from one setup: Qwen/Qwen3–8B in bfloat16 on an NVIDIA A100-SXM4–40GB (PyTorch 2.11.0+cu130), Hugging Face transformers with eager attention, batch size 1, no CUDA graphs or torch.compile. This is a simple research setup, not a tuned inference server, so absolute latencies would be lower on vLLM or SGLang. The code and raw results are on GitHub.
A chat UI feels like a session that remembers you. The model itself does not. The 15.3 GiB of weights in Qwen/Qwen3-8B are read-only during inference, and nothing about your request is written back into them. Anything that looks like memory is either text your application re-sends or a server-side cache of that text.
Here is the demonstration on our GPU :
Your application, not the model, keeps the conversation and re-sends the whole transcript on every turn.
I timed a 10-turn DevOps conversation in which each turn adds a server log and asks for a one-line summary. By Turn 10 the prompt is 963 tokens long. Two modes:
Each number is the median of 5 timed runs of one forward pass plus picking the first token (prefill time to first token, which excludes tokenization, scheduling and network):
Over all 10 turns, re-prefilling processes 5,549 tokens to read a 963-token conversation, 5.55x the 999 tokens the cached mode computes (868 ms vs 642 ms summed over the 10 turns, 1.35x). The token saving is large, but in this setup the time saving only shows once the prompt passes roughly 500 tokens: below that, every forward pass costs about the same ~65 ms, whatever its length (more on that floor in Section 3).
One trap we hit. Qwen3’s chat template ends every prompt with an empty reasoning block, <|im_start|>assistant\n<think>\n\n</think>\n\n. When that turn becomes history on the next request, the template writes
<|im_start|>assistant\n<reply> instead, without the think block. So the previous prompt and the new prompt stop matching 4 tokens before the reply, and a prefix cache can only reuse the tokens before that point. Our first version of this benchmark missed it. The numbers above come from a re-run that checks the shared prefix token by token, and the cached and full re-prefill runs pick the same next token on every turn.
Why is reuse safe at all? Causal attention only lets token i look at tokens 0..i, so appending new tokens never changes the Keys and Values of earlier ones. If the first k token IDs are identical, their cached K/V tensors are what a full re-prefill would compute. In bfloat16, different kernel shapes give small numerical differences: in our run the last-token logits differed by at most 1.19 (on logits around 50–60), and the chosen next token was the same on every turn. This small PyTorch check shows it:
import torch def verify_causal_prefix_kv_invariance(seq_len_turn1=16, seq_len_delta=8, d_model=64): """Single attention layer: attention outputs for the new tokens are the same whether the prefix is recomputed or its K/V are reused from a cache.""" torch.manual_seed(42) total = seq_len_turn1 + seq_len_delta wq = torch.nn.Linear(d_model, d_model, bias=False) wk = torch.nn.Linear(d_model, d_model, bias=False) wv = torch.nn.Linear(d_model, d_model, bias=False) x_prefix = torch.randn(1, seq_len_turn1, d_model) x_delta = torch.randn(1, seq_len_delta, d_model) x_full = torch.cat([x_prefix, x_delta], dim=1) # Mode 1: recompute everything q_full, k_full, v_full = wq(x_full), wk(x_full), wv(x_full) scores_full = (q_full @ k_full.transpose(-2, -1)) / (d_model ** 0.5) causal_mask = torch.triu(torch.full((total, total), float("-inf")), diagonal=1) attn_full = torch.softmax(scores_full + causal_mask, dim=-1) @ v_full # Mode 2: reuse cached K/V for the prefix, compute only the new tokens k_cached = torch.cat([wk(x_prefix), wk(x_delta)], dim=1) v_cached = torch.cat([wv(x_prefix), wv(x_delta)], dim=1) q_delta = wq(x_delta) scores_delta = (q_delta @ k_cached.transpose(-2, -1)) / (d_model ** 0.5) delta_mask = torch.triu(torch.full((seq_len_delta, total), float("-inf")), diagonal=seq_len_turn1 + 1) attn_delta_cached = torch.softmax(scores_delta + delta_mask, dim=-1) @ v_cached max_diff = (attn_full[:, seq_len_turn1:, :] - attn_delta_cached).abs().max().item() print(f"Max absolute difference, full re-prefill vs cached prefix: {max_diff:.2e}") assert max_diff < 1e-5 verify_causal_prefix_kv_invariance()
Rule 1**:** keep the start of the prompt fixed. A timestamp or request ID at the top of the system prompt changes an early token, and everything after it misses the cache, so the server re-prefills the whole history on every agent step. Put the system prompt and tool definitions first and unchanged, and append anything dynamic at the end. Check your model’s chat template too: as we saw with Qwen3, the template itself can rewrite earlier turns.
client.chat.completions.create(messages=[...]) takes a list of role/content dictionaries. The Transformer never sees those dictionaries. Before the first layer runs, the server calls tokenizer.apply_chat_template(), which turns the list into one flat sequence of integer token IDs with special control tokens marking each turn.
With enable_thinking=False, a system message "Answer with the 3-digit HTTP status code only." and the user question "What is the HTTP status code for Not Found?" become exactly 38 tokens:
I asked five short questions with known answers (HTTP Not Found → 404, HTTPS port → 443, 17 × 6 → 102, symbol for gold → Au, bits in a byte → 8), each formatted three ways, greedy decoding, at most 32 new tokens. A reply counts as “clean” only if it is the short answer and then stops.
Where does the model look? I recorded the attention weights (output_attentions=True) in all 36 layers for the 38-token prompt above, at the last prompt position (37), just before the model writes 404. Averaged over all heads and layers, 59.1% of that position's attention lands on the 7 positions that hold the four special-token IDs (151644, 151645, 151667, 151668). By layer it reaches 73.8–77.3% in layers 7–9 and peaks at 91.4% in layer 30 (88.7% in layer 29, 87.9% in layer 32). Those 7 positions are 18.4% of the prompt, so this is far above an even spread.
Two caveats. Position 0 is <|im_start|>, and the first token of a sequence is a well-known attention sink that soaks up attention in most LLMs (Xiao et al., arXiv:2309.17453), so part of this mass is sink behaviour. And attention weight is not causal influence. Read the result as "the model attends heavily to the template's scaffolding" on this one prompt, not as proof that those tokens steer the answer.
Rule 2: always format requests with the model’s own chat template, apply_chat_template() or the server's chat endpoint. Never hand-build prompt strings or reuse another model family's tags.
Once the prompt is a sequence of token IDs, the GPU runs the call in two phases. Prefill processes all input tokens in one forward pass and produces the first output token. Decode then produces one token per forward pass, because each new token depends on the previous one.
For each prompt length N in {64, 256, 512, 1,024, 2,048}, the script prefills N tokens and then decodes 64 tokens (m3_prefill_vs_decode_two_phases):
Two things stand out. Prefilling 512 input tokens took 78.0 ms, while one output token took 63.2 ms. And at N = 1,024, input went through at 7,672 tokens/s and output at 15.5 tokens/s, 494x slower per token.
Why? The textbook answer is the shape of the math. In prefill, every weight matrix multiplies a block of N token vectors at once, so each weight read from memory is reused N times. In decode there is a single new token vector, so each step has to read all 15.3 GiB of weights to do very little arithmetic. Decode is bound by memory bandwidth.
Our numbers add a second, setup-specific reason. If decode were limited only by reading the weights, the A100’s 1,555 GB/s of memory bandwidth (NVIDIA datasheet) would allow about 10 ms per token. We measured about 64 ms, an effective rate of only 237–242 GiB/s, about 255–260 GB/s (weights divided by time per token, not a profiled memory rate), roughly 16% of the spec. Decode time also barely moved with context length (64.3 ms at N = 64, 63.3 ms at N = 2,048), and prefilling 64 or 256 tokens cost about the same as one decode step. So in this Hugging Face eager, batch-1 setup, most of each forward pass is a fixed overhead from launching many small GPU kernels, on top of the weight reads. Serving engines such as vLLM and SGLang cut that overhead with CUDA graphs and batch many users per step, so the exact ratio depends on the stack. The direction does not: per token, decode costs far more than prefill.
For a whole call, decode dominates. With a 64-token reply, decode was 96.9% of the call at N = 1,024 and 92.8% at N = 2,048. A 500-token reply at 64 ms per token is about 32 seconds in this setup.
Rule 3: budget latency around output length. Keep structured outputs compact, don’t ask for prose fields you don’t need, and turn off long reasoning (enable_thinking=False on Qwen3) for simple extraction steps.
At the end of each decode step, the final hidden state is multiplied by the output matrix (lm_head, shape [4096, 151936]) to give 151,936 logits, one score per vocabulary entry. The sampler divides them by the temperature T and applies softmax. T = 0 means always take the top token. Higher T flattens the distribution so lower-ranked tokens get picked more often.
In a chat window, a different word now and then is harmless. In an agent, the reply often goes straight into json.loads() or a Pydantic model, and one stray character breaks the pipeline.
The task: extract "service" (string), "port" (integer) and "healthy" (boolean) from 8 server log lines. Each mode ran 16 calls at each of T ∈ {0.0, 0.3, 0.7, 1.0, 1.3} (the 8 cases twice, and top_p = 0.95 when T > 0), so 240 local generate() calls in total. Pass means strict json.loads() on the raw reply, the exact three keys, and the right types. Field values were not scored.
What does the prompt change? The right panel of Figure 4 shows the raw step-0 logits for the strict prompt. The top candidates are '{\n' (51.0, 97.1%), '{"' (47.5, 2.93%) and '{' (39.8). At rank 4 sits the Markdown fence '```' with a logit of 36.5. That is 14.5 points below the top token, so its probability is tiny, about 5 in 10 million, but it is not zero. With harder schemas, long inputs or adversarial text in the input, such low-probability tokens can still be sampled. A prompt only shifts the scores. Every token in the 151,936-entry vocabulary stays possible.
How do OpenAI Structured Outputs, vLLM guided decoding, Outlines and XGrammar make invalid JSON impossible instead of unlikely? They turn the JSON schema into a grammar (a finite-state machine, or a pushdown automaton for nested schemas) and plug a logits processor into the decoding loop, just before softmax.
At each step, the grammar looks at the text generated so far and works out which tokens can legally come next. Every other token’s logit is set to -inf. Because exp(-inf) = 0, those tokens have probability zero at any temperature.
Our grammar is deliberately simple and hand-written: compact single-line JSON of the form {"service": "<name>", "port": <int>, "healthy": <bool>}, where <name> must be one of the 8 service names in the test set. Here is the step trace from the run (constrained_step_traces) while it generated {"service": "auth-api", "port": 8081, "healthy": false}:
A minimal logits processor that does the same thing, and a check on the real step-0 logits from our run:
import torchfrom transformers import LogitsProcessor class PrefixGrammarLogitsProcessor(LogitsProcessor): """Masks every token that cannot extend the text generated so far into one of allowed_strings. Toy version: scans the whole vocab each step.""" def __init__(self, prompt_len: int, id_to_piece: dict, allowed_strings: list): self.prompt_len = prompt_len self.id_to_piece = id_to_piece self.allowed_strings = allowed_strings def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor: gen_ids = input_ids[0, self.prompt_len:].tolist() prefix = "".join(self.id_to_piece[i] for i in gen_ids) mask = torch.full_like(scores, float("-inf")) for tok_id, piece in self.id_to_piece.items(): if any(target.startswith(prefix + piece) for target in self.allowed_strings): mask[0, tok_id] = 0.0 return scores + mask # Step-0 top-5 logits from our A100 runvocab = {0: "{\n", 1: '{"', 2: "{", 3: "With the mask, **80 of 80** calls produced valid JSON at every temperature from 0.0 to 1.3 (median 2,424–2,472 ms per call). The extra time over the strict prompt comes from our slow Python scanner, which checks the whole vocabulary at every step, and from different output lengths. Production engines such as XGrammar precompute token masks per grammar state, which makes the per-step cost far smaller (Dong et al., arXiv:2411.15100).
Two caveats. The mask guarantees *form*, not *truth*: if port must be an integer, only digits survive, but an ambiguous prompt can still produce 5432 instead of 5439. And the guarantee only holds if generation finishes inside the token budget. Cut it off early and you get a valid prefix of JSON, not valid JSON.
Before Episode 2 wires this call into a tool-calling loop, here are the four rules this run supports:
📺 Watch the full episode: [What Happens in 1 LLM API Call? (494x Speed Gap)](https://youtu.be/qv8PDE7JpXs)
*In Episode 2 of How AI Agents Actually Work, we take this constrained JSON output and wire it into a Python dispatcher to build tool calling from scratch.*
[What Happens in 1 LLM API Call? Measured on GPU With Qwen3–8B](https://pub.towardsai.net/what-happens-in-1-llm-api-call-measured-on-gpu-with-qwen3-8b-3654ffb0e290) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.