{"slug": "how-llm-serves-a-request", "title": "How LLM Serves a Request", "summary": "Manas Pathak's primer explains that an LLM server processes a request in two phases—prefill, which reads the entire prompt in one forward pass, and decode, which generates each output token in a separate forward pass that streams all 8.6 GB of weight matrices from GPU memory (HBM) through the compute cores, making decode the dominant cost. The primer, part of a series on 'The Inference Wall,' uses Qwen3.5-4B and emphasizes that the speed of generating tokens is limited by memory bandwidth, not arithmetic.", "body_md": "# A primer: how an LLM actually serves a request\n\n*A primer for “The Inference Wall”. Read this before\nPart 1 if the words KV cache, prefill, decode,\nor batch are fuzzy. It explains the machine the five posts go on to break; it deliberately\nstops before any of their findings.*\n\n*Manas Pathak · August 21, 2026*\n\nThe five posts in this series each take a working LLM server, turn one knob until something\nbreaks, and read why. To follow *why* each break happens, you need a mechanical picture of\nwhat the server is doing between the moment a request arrives and the moment its answer\nfinishes streaming. That picture is small, it is not math-heavy, and once you have it every\npost is a variation on it. This primer builds it once. No benchmarks here, no surprises, just\nthe machine.\n\n## The model is a pile of weight matrices in memory\n\nA language model is, physically, a large collection of **weight matrices**, fixed numbers\nlearned during training. For the model this series uses (Qwen3.5-4B), that collection is\n**8.6 GB**. When you start the server, those 8.6 GB are loaded once into the GPU’s memory and\nthey stay there, unchanged, for the life of the server.\n\nThe GPU has two relevant parts. There is its **memory** (called HBM), which is large, holds\nthose 8.6 GB comfortably, but is relatively slow to read from. And there are its **compute\ncores**, which do the actual multiplying, are extremely fast, but have almost no storage of\ntheir own. This split is the single most important fact in the whole series, so hold onto it:\n**the weights live in the slow, roomy memory; the fast cores that use them cannot keep the\nweights parked next to themselves.**\n\n## Producing one token = streaming all the weights through the cores\n\n“Running the model on a token” means taking that token, represented as a vector of numbers,\nand multiplying it through every weight matrix in turn, layer by layer (this model has 32\nlayers), until numbers come out the other end that tell you the next token. That single sweep\nthrough all the matrices is called a **forward pass**.\n\nBecause the cores cannot hold 8.6 GB, doing a forward pass means **streaming** all 8.6 GB of\nweights out of HBM and through the cores. The multiplying itself is quick; the *moving* of\nthose bytes is the slow part. You will see the series lean on this again and again: the cost\nof producing a token is dominated by how many bytes of weights have to be streamed to produce\nit, not by the arithmetic done with them.\n\n## Two phases: prefill reads the prompt, decode writes the answer\n\nEvery request runs in two distinct phases, and they behave very differently.\n\n**Prefill** is the first phase: the model reads your whole prompt. Crucially, all the prompt\ntokens already exist (you typed them), so they can all be pushed through the forward pass\n*together*, in one sweep. A 100-token prompt is one forward pass over 100 tokens. Prefill is\nwhere the model does a lot of arithmetic at once, because every prompt token interacts with\nevery other (a prompt of length N does roughly N-by-N work as each token looks at all the\nothers).\n\n**Decode** is the second phase: generating the answer, one token at a time. Here is the\nconstraint that shapes everything downstream: to produce output token 2, the model needs\noutput token 1 as input, because a language model predicts each token from the ones before\nit. So the tokens of an answer *cannot* be produced together, the way a prompt’s tokens can.\nEach output token is its **own** forward pass, over just one new token, and each such pass\nstreams all 8.6 GB of weights again. Prefill amortizes one weight-stream over the whole\nprompt; decode is stuck paying one weight-stream per output token. That asymmetry is why\ndecode, not prefill, is the phase this series spends most of its time on.\n\n## The KV cache: why decode does not reread the whole conversation\n\nIf each output token needs “the tokens before it,” you might think every decode step reprocesses\nthe entire conversation so far. It does not, and the thing that saves it is the **KV cache**.\n\nWhen the model processes a token, part of its work produces two vectors for that token, a\n**key** and a **value** (K and V), which together are how later tokens will “look back” at this\none. The KV cache simply *stores* those K and V vectors for every token the model has already\nseen. So when the model generates the next token, it does not recompute the past, it looks up\nthe cached K and V of every earlier token and attends to them.\n\nTwo things to keep straight, because they trip people up:\n\n- The KV cache stores\n**per-token data (K and V vectors)**, not weight matrices. The 8.6 GB of weights are one thing; the KV cache is a separate, much smaller pile that grows as the conversation grows. On this model a cached token is around 130 KB, so a hundred cached tokens is barely ten megabytes, tiny next to 8.6 GB. - Because of the cache, a decode step feeds the model only each request’s\n**single most recent token**, not its whole history. The history is already in the cache; only the newest token is new.\n\nThe KV cache lives in the GPU’s memory alongside the weights, and unlike the weights it grows with every token of every active request. That makes it the part of memory that can fill up under load, which is why one of the posts is entirely about starving it.\n\n## Serving many requests at once: one weight-stream, a whole batch\n\nA real server is not answering one request; it is answering many at once. Here is how, and it is the mechanism the whole series turns on.\n\nThe server runs a loop. Each turn of the loop is one **step**: one forward pass, shared by\nevery request currently being worked on. That set of requests is the **running batch**. A\nsingle step does this:\n\n- Take each active request’s most recent token, one vector per request, and stack them into one taller matrix. If ten requests are active, that is ten vectors stacked together.\n- Stream the weight matrices from HBM\n**once**, and multiply them against that whole stack at the same time. A weight matrix multiplied by ten stacked tokens costs the same*streaming*as multiplying it by one, because it is the same weight matrix read once; only the arithmetic grows, and the arithmetic was the cheap part. - Out comes one new token for\n**every** request in the batch, all produced by that single forward pass.\n\nSo the expensive thing, streaming 8.6 GB of weights, is **shared across the entire batch in a\nsingle step**. Ten requests get their next token for the price of one weight-stream. This is\ncalled **batching**, and it is the single most important reason one GPU can serve many users\nat once. Each request also does a little of its own private work in the step (attending to its\n*own* KV cache, which is different from everyone else’s), but the big shared cost is the one\nweight-stream.\n\nThen the loop repeats. The next step feeds in the tokens just produced, streams the weights again, and advances every request by one more token. To generate a hundred-token answer takes about a hundred steps, a hundred weight-streams, each one shared across whatever batch is running.\n\nTwo numbers describe this batch that the posts will refer to. The **arrival rate** is how fast\nrequests come in (say, 10 per second); it is not the batch size, because each request lives in\nthe server for a while, so many are in flight at once. And `max_num_seqs`\n\nis a configured\n**ceiling** on the batch, the most requests the server will run together; the actual running\nbatch is whatever the load produces, up to that ceiling.\n\n## The knobs the posts will turn\n\nThat is the whole machine: weights streamed from HBM per step, a KV cache that lets decode feed just the latest token, and a batch that shares each weight-stream. Everything the series does is push on one part of it. Three levers show up by name, so here they are in one line each:\n\n: the cap on how many requests run in a batch at once.`max_num_seqs`\n\n**chunked prefill**: instead of letting a long prompt’s prefill occupy whole steps by itself, slice it and interleave the pieces into steps alongside the decode work.**quantization**: store the weights in fewer bits (4 instead of 16), so there are fewer bytes to stream per step.\n\nYou do not need to know how any of these work internally yet; the posts introduce each where\nit matters. What you need is the picture above: **a serving step is one shared stream of the\nweights that advances a whole batch of requests by one token, and the KV cache is what lets\neach request contribute just its newest token.** With that in hand, Part 1 can ask the\nquestion the series is really about, which is what happens to this machine when you push it\nuntil it breaks.\n\n## About the author\n\nI am **Manas Pathak**. Questions, corrections, or a number of your own that disagrees with\nmine are all welcome: email me at [mapathak@gmail.com](mailto:mapathak@gmail.com) or find me on\n[LinkedIn](https://www.linkedin.com/in/manas-pathak-806b002a/).\n\n**Next:** [Part 1 — An 8.6 GB model that serves only 7 requests a second](/inference-wall/articles/part-1/)\n\n*Disclaimer: This blog is written and published in my personal capacity. The opinions,\nfindings, and conclusions expressed herein are solely my own and do not necessarily\nrepresent the views, policies, or endorsements of my current or past employers.*", "url": "https://wpnews.pro/news/how-llm-serves-a-request", "canonical_source": "https://mapathak-commits.github.io/inference-wall/articles/primer/", "published_at": "2026-08-22 21:42:26+00:00", "updated_at": "2026-08-22 22:13:43.548841+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure"], "entities": ["Manas Pathak", "Qwen3.5-4B"], "alternates": {"html": "https://wpnews.pro/news/how-llm-serves-a-request", "markdown": "https://wpnews.pro/news/how-llm-serves-a-request.md", "text": "https://wpnews.pro/news/how-llm-serves-a-request.txt", "jsonld": "https://wpnews.pro/news/how-llm-serves-a-request.jsonld"}}