{"slug": "inside-vllm-following-one-request-from-the-api-to-gpu-execution", "title": "Inside vLLM: Following One Request from the API to GPU Execution", "summary": "A developer's source-code walkthrough of vLLM 0.22's V1 execution path traces a single offline inference request from the LLM.generate() API through inter-process communication, scheduling, GPU execution, and paged KV-cache access. The article explains how EngineCore runs in a child process with a continuous busy loop, separating scheduling from execution, and details the three core stages of EngineCore.step(): scheduling, model execution, and state update.", "body_md": "Article 1 of 3 · vLLM Internals\n\nThis English edition is adapted from the [published Chinese article on Zhihu](https://zhuanlan.zhihu.com/p/2060033883849204419). It preserves the source-code references, experimental boundaries, and reproducible artifacts while adapting the structure for an international engineering audience.\n\n**Series:** Part 1 · Request lifecycle · [Part 2 · CUDA kernels and paged attention](https://jacklei0901.github.io/articles/inside-vllm-cuda-kernels/) · [Part 3 · FlashAttention from PyTorch to Triton](https://jacklei0901.github.io/articles/building-flashattention-pytorch-triton/)\n\nThis article follows one offline inference request through vLLM V1: from\n\n`LLM.generate()` and inter-process communication to scheduling, input\n\nflattening, GPU model execution, paged KV-cache access, sampling, and resource\n\nreclamation. The goal is to answer one concrete question: what happens behind\n\nthe call to `llm.generate()` before the completed result reaches the caller?\n\nThe discussion assumes familiarity with Transformer inference, including\n\nprefill, decode, KV caching, and autoregressive generation. It focuses on how\n\nthose concepts appear in vLLM source code rather than reteaching the model\n\narchitecture.\n\n**Version scope.** The source references were verified against vLLM 0.22.0;\n\nthis edition was checked on September 3, 2026. vLLM evolves quickly, so some\n\nfilenames and call boundaries will move. The\n\nlong-lived ideas—continuous batching, token budgets, paged KV allocation, and\n\nthe separation between scheduling and execution—are the real subject of the\n\narticle.\n\nMany introductions stop at the useful analogy that PagedAttention manages the\n\nKV cache much like virtual memory manages pages. The analogy does not tell us\n\nhow a request is admitted, how variable-length requests become a flat token\n\nbatch, or what the page table looks like at the kernel boundary.\n\nThe answers are in the source. This article follows vLLM 0.22's V1 execution\n\npath from the public entry point to the CUDA boundary. It is a source-code\n\nwalkthrough, not an API tutorial.\n\nStart with the process boundary and the engine loop. In vLLM V1, EngineCore\n\nruns continuously in a child process. Requests enter and leave the active set\n\nbetween steps; the caller does not drive GPU execution one token at a time.\n\nOne detail shapes the entire path: in this configuration, vLLM 0.22 separates\n\nthe caller-facing engine from EngineCore with a process boundary.\n\n``` php\nREQUEST SUBMISSION · main process -> EngineCore child\n\nCaller\n  └─ LLM.generate(prompts)\n       └─ LLMEngine.add_request()\n            └─ EngineCoreClient\n                 └─ IPC input queue ──────────────▶ scheduler.add_request()\n\nCONTINUOUS ENGINE LOOP · EngineCore child\n\nrun_busy_loop()\n  ├─ receive newly submitted requests\n  └─ repeat EngineCore.step()\n       1. scheduler.schedule()\n          └─ spend token / KV-cache budgets -> SchedulerOutput\n       2. model_executor.execute_model()\n          └─ GPU forward pass and sampling -> sampled tokens\n       3. scheduler.update_from_output()\n          ├─ update request state and release finished KV blocks\n          └─ EngineCoreOutputs ───────────────▶ IPC output queue\n\nRESULT CONSUMPTION · EngineCore child -> main process\n\nIPC output queue\n  └─ LLMEngine.step() / get_output()\n       └─ detokenize and assemble RequestOutput\n            └─ return completed results to LLM.generate() and the caller\n```\n\nThis map establishes three boundaries used throughout the article.\n\nFirst, the main process consumes results; EngineCore produces them. Calling\n\n`LLMEngine.step()` does not tell the GPU to perform one step. It retrieves\n\nalready-produced output through `get_output()`, then detokenizes and assembles\n\nthe user-facing result. Scheduling and model execution advance independently\n\ninside EngineCore's busy loop.\n\nSecond, `EngineCore.step()` has three core stages: scheduling, execution, and\n\nstate update. The rest of the article expands those stages one at a time.\n\nThird, this loop implements continuous batching. The request set changes from\n\none step to the next: stage 3 removes completed requests and releases their\n\nresources, while the next call to `schedule()` can admit waiting work. The GPU\n\ntherefore does not have to wait for one fixed batch to finish before admitting\n\nanother request.\n\nThe rest of the article expands that loop into six parts:\n\n`LLM` and `LLMEngine` submit work to EngineCore.`schedule()` spends token and KV-cache budgets.\nEach part keeps only the code needed to connect one boundary to the next.\n\n`generate()`\nStart with the public interface:\n\n``` python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(model=\"facebook/opt-125m\")\noutputs = llm.generate([\"Once upon a time\"], SamplingParams(max_tokens=50))\n```\n\nThose two lines initialize the entire engine stack. Start with the `LLM` class.\n\nIn `entrypoints/llm.py`, `LLM` delegates engine construction to one factory\n\ncall:\n\n```\n# entrypoints/llm.py\nself.llm_engine = LLMEngine.from_engine_args(\n    engine_args=engine_args,\n    usage_context=UsageContext.LLM_CLASS,\n)\n```\n\n`LLM` packages model, dtype, and other configuration into `EngineArgs`, then\n\nuses the factory method to create `LLMEngine`. It is the public facade;\n\n`LLMEngine` owns the request-processing path followed below.\n\n`generate()` returns a `list[RequestOutput]`, one complete result per request.\n\nThe generated text is available as `output.outputs[0].text`.\n\nThe distinction is useful throughout the article: `LLM` is the facade, while\n\n`LLMEngine` connects that facade to EngineCore.\n\nThe internal logic of `generate`, after stripping away the details, is as follows:\n\n```\n# Conceptual skeleton\nfor prompt in prompts:\n    self.llm_engine.add_request(prompt, ...)   # 1. Register the request with the engine\n\noutputs = []\nwhile self.llm_engine.has_unfinished_requests():  # 2. While any request is unfinished\n    step_outputs = self.llm_engine.step()         #    advance the engine by one step\n    for out in step_outputs:\n        if out.finished:\n            outputs.append(out)\nreturn outputs                                     # 3. Return after collecting every result\n```\n\nThis `while ... step()` loop continues until every submitted request is\n\nfinished. Unlike a hand-written `for _ in range(max_tokens)` loop over one\n\nsequence, it tracks a set of requests whose lengths and completion times differ.\n\n`step()` Does Not Submit the Request\nThe name `step()` can suggest a synchronous \"submit, execute, return\" call.\n\nThe implementation does something different:\n\n``` python\n# v1/engine/llm_engine.py (simplified)\ndef step(self):\n    # 1) Fetch outputs already produced by EngineCore\n    outputs = self.engine_core.get_output()\n\n    # 2) Decode tokens and evaluate stopping conditions\n    processed = self.output_processor.process_outputs(outputs.outputs, ...)\n\n    # 3) Handle requests terminated by a stop string\n    self.engine_core.abort_requests(processed.reqs_to_abort)\n\n    # 4) Record statistics\n    ...\n\n    return processed.request_outputs\n```\n\n`step()` never submits a request or directly drives model execution. It calls\n\n`get_output()` to consume results. Submission already happened in\n\n`add_request()`; the actual computation runs independently elsewhere.\n\nThis description assumes the default multiprocess V1 configuration. If\n\n`VLLM_ENABLE_V1_MULTIPROCESSING=0` is set for debugging, the in-process client\n\ndrives EngineCore from `get_output()` and the child-process producer/consumer\n\nboundary collapses. The scheduling, execution, and update stages discussed\n\nbelow still apply, but they no longer run in a separate EngineCore process.\n\nThis producer-consumer split is asynchronous across the process boundary. In\n\nconfigurations that enable asynchronous scheduling, the startup log also states:\n\n```\nINFO ... Asynchronous scheduling is enabled.\n```\n\nThat \"elsewhere\" is a separate child process. A running server prints a prefix\n\nsuch as `(EngineCore pid=8884)` in its logs: this is the `EngineCore` process\n\nthat performs scheduling and model execution.\n\nThe main process (`LLMEngine`) is the consumer: `get_output()` receives results,\n\ndecodes them, and returns them to the caller. The child process (`EngineCore`)\n\nis the producer: its busy loop schedules requests, runs the model, produces\n\ntokens, and writes results to an IPC queue.\n\nThis lets host-side output processing overlap EngineCore execution. In an online\n\nserver, request tokenization and admission can also proceed independently of an\n\nalready-running GPU step. The IPC queues separate those execution timelines.\n\n`generate` is the `while has_unfinished_requests(): step()` loop.`LLMEngine.step()` consumes and decodes output; the separate `EngineCore`\nprocess performs scheduling and execution asynchronously.\nNext we cross the IPC boundary and enter `EngineCore`, whose busy loop has three\n\ncore stages: schedule, execute, and update.\n\n*Figure: ownership among vLLM's core classes. `LLM` owns `LLMEngine`, which\ncommunicates with `EngineCore`; `EngineCore` in turn owns the scheduler,\nmodel executor, and KV-cache manager. The horizontal line is the process\nboundary: main process/consumer above, child process/producer below.*\n\nThe previous section stopped at the child-process boundary. Each iteration of\n\nthe `EngineCore` busy loop runs the three stages of `step()`; this section\n\nstarts with `scheduler.schedule()`.\n\nContinuous batching is not a standalone module. It emerges from the decisions\n\nmade on every call to `schedule()`. One useful mental model is an operating-system\n\nscheduler whose scarce resources are token-compute budget and GPU KV-cache\n\npages.\n\nThe scheduler lives in `v1/core/sched/scheduler.py`; its central method is\n\n`schedule()` (`scheduler.py:329`). The file is large, but this path only needs\n\nthree pieces: the main loop, preemption, and the final packaged output.\n\nThe V1 scheduler can be summarized in one sentence: during each engine step,\n\nadvance as many requests as possible without exceeding the token budget or the\n\navailable KV blocks.\n\nThese concepts map naturally to familiar operating-system terms:\n\n`token_budget`: the maximum number of tokens processed in one step\n(`max_num_batched_tokens=8192` in the example startup log), analogous to a\nbounded compute time slice.`waiting`: requests waiting to begin or continue prefill, analogous to a ready\nqueue.`running`: admitted requests eligible to advance in the current step.\nThe comment at the start of `schedule()` is the best guide to its model:\n\n```\n# v1/core/sched/scheduler.py:330\n# There's no \"decoding phase\" nor \"prefill phase\" in the scheduler.\n# Each request just has the num_computed_tokens and num_tokens_with_spec.\n# At each step, the scheduler tries to assign tokens to the requests\n# so that each request's num_computed_tokens can catch up its\n# num_tokens_with_spec.\n```\n\nThe V1 scheduler does not maintain separate prefill and decode phases. Instead,\n\nit compares each request's `num_computed_tokens` with\n\n`num_tokens_with_spec` and decides how many tokens to advance in the current\n\nstep. A decode request commonly advances by one token; a prefill request may\n\nadvance by a larger chunk. The same progress model also accommodates chunked\n\nprefill and prefix-cache hits.\n\nThe following sketch is a useful map of `schedule()`:\n\n```\nOne schedule() step:\n  pass 1  RUNNING first (decode) ──── no free block → preempt from the tail\n  pass 2  fill from WAITING (prefill) ── over budget → clamp or defer\n  pack    SchedulerOutput (the execution plan)\n```\n\n*Figure: one `schedule()` step. The scheduler considers running requests,\nadmits waiting requests with any remaining budget, handles allocation failure\nthrough preemption, and packages the result as `SchedulerOutput`.*\n\nThe first substantive loop in `schedule()` traverses `running`\n\n(`scheduler.py:364`):\n\n```\n# v1/core/sched/scheduler.py:364\n# First, schedule the RUNNING requests.\nreq_index = 0\nwhile req_index < len(self.running) and token_budget > 0:\n    request = self.running[req_index]\n    num_new_tokens = (request.num_tokens_with_spec\n                      + request.num_output_placeholders\n                      - request.num_computed_tokens)\n    num_new_tokens = min(num_new_tokens, token_budget)\n    ...\n```\n\nThe source comments make the order explicit: running requests are considered\n\nbefore waiting requests. This ordering tends to protect inter-token latency for\n\nrequests that are already generating, although the observed latency trade-off\n\nstill depends on the configured scheduling policy and workload. It also lets\n\nrequests that already hold KV blocks continue making progress toward completion\n\nand eventual reclamation. These are useful consequences of the order, not a\n\nclaim that every running request always outranks every waiting request under all\n\npolicies.\n\nThe `waiting` queue is an admission queue, not a tokenization cache. Requests in\n\nit are waiting for scheduler resources; prompt preprocessing has already\n\nhappened before this point.\n\nAfter considering running requests, the scheduler uses any remaining budget to\n\nadmit work from `waiting` (`scheduler.py:544`). This is where the chunked-prefill\n\nbudget clamp appears.\n\nA waiting request may have 5,000 prompt tokens while only 2,000 tokens remain\n\nin the current step's budget. The relevant control flow is:\n\n```\n# Budget clamp inside the waiting loop\nif (not self.scheduler_config.enable_chunked_prefill\n        and num_new_tokens > token_budget):\n    # If chunked_prefill is disabled, we can stop the scheduling here.\n    break\n\nnum_new_tokens = min(num_new_tokens, token_budget)\nassert num_new_tokens > 0\n```\n\nTwo details matter here.\n\nFirst, `break` does not drop the request. It exits the waiting-loop because the\n\nremaining budget cannot admit another request in this step. The request stays\n\nin `waiting` and is reconsidered the next time `schedule()` runs.\n\nWith chunked prefill enabled, `min()` admits the portion that fits and leaves\n\nthe rest for a later step. With it disabled, an over-budget request stays in\n\n`waiting` and is retried as a whole. Neither path loses the request; removal\n\nhappens only after normal completion or an explicit abort.\n\nA useful rule when reading schedulers is to ask where an item goes after a\n\n`break` or `continue`. Here it remains in the same queue for a later step, just\n\nas an OS process that misses one time slice remains runnable.\n\nSecond, chunked prefill reduces to a budget clamp:\n\n`num_new_tokens = min(num_new_tokens, token_budget)`. A long prompt contributes\n\nonly the tokens that fit in the current step; the remainder is scheduled later.\n\nThe full policy considers more state, but this clamp is what divides a long\n\nprefill across scheduler steps.\n\n(The running-request loop has the same clamp at `scheduler.py:392`. Because V1\n\ndoes not maintain separate prefill and decode phases, the budget logic applies\n\nuniformly to both.)\n\nAnother branch appears when a running request needs more KV blocks and\n\n`allocate_slots` cannot satisfy the request. The allocation-retry loop beginning\n\nat `scheduler.py:443` can then preempt a victim:\n\n```\n# v1/core/sched/scheduler.py:443 (simplified)\nwhile True:\n    new_blocks = self.kv_cache_manager.allocate_slots(request, num_new_tokens, ...)\n    if new_blocks is not None:\n        break                              # Allocation succeeded; schedule normally\n    # Allocation failed; choose a victim to preempt\n    preempted_req = self.running.pop()      # Default policy: pop from the tail (LIFO)\n    self._preempt_request(preempted_req, scheduled_timestamp)\n    if preempted_req == request:\n        break                              # No victim remains; defer this request\n```\n\nUnder the default policy, `self.running.pop()` selects a victim from the tail\n\n(LIFO). This usually victimizes a more recently admitted request with less\n\naccumulated progress, reducing the work discarded, although queue position is\n\nnot a universal measure of recomputation cost. With `PRIORITY`\n\nscheduling, victim selection also considers priority (`scheduler.py:456`).\n\n`_preempt_request` (`scheduler.py:929`) defines what happens to the preempted\n\nrequest's KV-cache state:\n\n``` python\n# v1/core/sched/scheduler.py:929\ndef _preempt_request(self, request, timestamp):\n    self.kv_cache_manager.free(request)          # ← release its KV blocks immediately\n    self.encoder_cache_manager.free(request)\n    request.status = RequestStatus.PREEMPTED\n    request.num_computed_tokens = 0              # ← reset computed progress\n    ...\n    self.waiting.prepend_request(request)        # ← prepend it to the waiting queue\n```\n\nTwo details matter here. First, `num_computed_tokens = 0` records recomputation\n\nsemantics: the scheduler no longer treats the request's previous progress as\n\nresident KV state. When rescheduled, it must rebuild the required state, subject\n\nto any cache reuse the current configuration can legitimately recover. This is\n\ndifferent from swapping the victim's KV tensors to host memory and restoring\n\nthem later.\n\nThe trade-off is workload-dependent. Swapping consumes host-device bandwidth\n\nand host memory; recomputation consumes additional GPU work. vLLM V1 chooses\n\nthe latter for this path, avoiding a KV transfer over PCIe at the cost of\n\nre-running model computation.\n\n*Figure: two preemption strategies. Swapping preserves KV state through a host-device transfer; recomputation releases the blocks and later rebuilds\nthe required state.*\n\nThe second detail is `prepend_request`, an anti-starvation measure. A preempted\n\nrequest returns to the head of `waiting` rather than the tail. Appending it\n\nbehind every newly arrived request could cause repeated preemption and\n\nindefinite delay; prepending gives the sacrificed request priority on the next\n\nadmission pass.\n\n`SchedulerOutput`\n`schedule()` packages its decisions into `SchedulerOutput` (defined in\n\n`v1/core/sched/output.py:181`) and hands that execution plan to the model\n\nrunner. The execution layer consumes the decision; it does not reschedule the\n\nrequests.\n\nMany fields support optional features such as speculative decoding,\n\nmultimodality, structured output, and connectors. Four core fields are enough\n\nto enter the execution layer:\n\n```\n# v1/core/sched/output.py:181 (selected core fields)\n@dataclass\nclass SchedulerOutput:\n    scheduled_new_reqs: list[NewRequestData]      # First scheduling: send complete request data\n    scheduled_cached_reqs: CachedRequestData      # Previously scheduled: send only the delta\n    num_scheduled_tokens: dict[str, int]          # req_id -> tokens processed in this step\n    total_num_scheduled_tokens: int\n    ...\n    finished_req_ids: set[str]                     # Completed requests; tell the worker to reclaim state\n    ...\n    new_block_ids_to_zero: list[int] | None = None # Newly allocated blocks that the worker must zero\n```\n\nTheir roles become clearer at the execution boundary.\n\n`scheduled_new_reqs` carries the state needed when a request is first scheduled;\n\n`scheduled_cached_reqs` carries updates for requests already known to the model\n\nrunner. This avoids rebuilding and transmitting the complete request state on\n\nevery step—a state-cache plus incremental-update pattern.\n\n`num_scheduled_tokens` implements the unified progress model: it records how\n\nmany tokens each request will process in this step. The execution layer can use\n\nthose counts to pack prefill and decode work into one flat token batch.\n\n`finished_req_ids` tells the worker which completed requests can release their\n\ncached state, analogous to reclaiming OS resources after a process exits.\n\n`new_block_ids_to_zero` identifies freshly allocated cache blocks that must be\n\ncleared before the forward pass when KV-cache zeroing is enabled. Zeroing keeps\n\nstale data or NaNs from an earlier owner from corrupting attention or\n\nstate-space-model computation. The field is conditional; not every\n\nconfiguration needs this step.\n\nSeveral scheduler concepts have useful operating-system analogies:\n\n| Operating system concepts | vLLM Scheduler correspondence | \n|---|---|\n| Ready Queue | `waiting` | \n| Run Queue | `running` | \n| Physical page frame allocator | `kv_cache_manager` | \n| Page fault/out of memory | `allocate_slots` returns None | \n| Process swap | Preemption by releasing KV state and recomputing | \n| time slice | token budget | \n| Priority compensation after preemption | `prepend_request` returns the victim to the head of`waiting` | \n| Output of scheduling decision | `SchedulerOutput` | \n\nContinuous batching follows from rebuilding the execution batch at every\n\n`schedule()` call. Completed requests leave, waiting requests enter, long\n\nprompts may be split into chunks, and allocation pressure may trigger\n\npreemption. The engine can therefore reuse newly available capacity without\n\nwaiting for an earlier fixed batch to finish in full.\n\nNext, follow this `SchedulerOutput` plan into the execution layer and see how\n\nvariable-length requests become flat tensors for the model.\n\nThe `SchedulerOutput` from the previous part reaches `GPUModelRunner` in the\n\nworker. Suppose it assigns four tokens to request A and one token each to\n\nrequests B and C. How does the runner turn those unequal slices into tensors for\n\nthe GPU?\n\nA conventional implementation might pad every request to the same length and\n\nstack a `[batch, seq_len]` tensor. This path instead packs the scheduled tokens.\n\nThe path enters `GPUModelRunner.execute_model`\n\n(`v1/worker/gpu_model_runner.py:3913`) and calls `_prepare_inputs`\n\n(`gpu_model_runner.py:1839`). `InputBatch`\n\n(`v1/worker/gpu/input_batch.py:36`) holds the prepared state:\n\n```\n# v1/worker/gpu/input_batch.py:36 (excerpt)\nclass InputBatch:\n    req_ids: list[str]\n    num_reqs: int\n\n    # [num_reqs] tokens processed for each request in this step\n    num_scheduled_tokens: np.ndarray\n    # sum(num_scheduled_tokens): total tokens in this step\n    num_tokens: int\n\n    # [num_reqs + 1] request boundaries in the flattened sequence (prefix sum)\n    query_start_loc: torch.Tensor\n    # [num_reqs] current total length of each request, including history\n    seq_lens: torch.Tensor\n\n    # [num_tokens] ← one-dimensional, not [batch, seq_len]\n    input_ids: torch.Tensor\n    # [num_tokens]\n    positions: torch.Tensor\n\n    # [total_num_logits] positions that actually require logits\n    logits_indices: torch.Tensor\n```\n\nThe key shape is `input_ids: [num_tokens]`: the scheduled tokens form one\n\npacked dimension rather than a padded batch matrix.\n\nV1 concatenates the tokens selected for the current step into one packed\n\nsequence, regardless of which request they belong to or whether they represent\n\nprefill or decode work.\n\nIn the example, A contributes a four-token prefill chunk, while B and C each\n\ncontribute one decode token. If their current positions are 50 and 30, the\n\npacked tensors look like this:\n\n```\ninput_ids   = [A0, A1, A2, A3, B0, C0]   # six values, one-dimensional\npositions   = [ 0,  1,  2,  3, 50, 30]   # each token's position within its own request\n```\n\nA starts at position 0; B and C retain positions from their own sequences rather\n\nthan being renumbered within the packed batch. Here\n\n`num_scheduled_tokens = [4, 1, 1]` and\n\n`query_start_loc = [0, 4, 5, 6]`. The latter is a prefix sum that marks each\n\nrequest's range. Attention combines these boundaries with `seq_lens` to keep\n\nrequests separate.\n\nThis design has two advantages.\n\nFirst, packing avoids computation on padding positions. Every scheduled position\n\ncorresponds to actual request work, although kernels may still pad the packed\n\ntoken count internally for CUDA graphs or alignment.\n\nSecond, packing carries the scheduler's unified token-count abstraction into\n\nexecution. The runner uses `num_scheduled_tokens` to place each request's slice\n\nin the flat sequence. Prefill and decode work can therefore coexist in one\n\nstep, while metadata preserves their separate boundaries.\n\n*Figure: three variable-length requests—a prefill chunk and two decode\nrequests—packed into one-dimensional `input_ids`, `positions`, and\n`slot_mapping` arrays.*\n\n`slot_mapping` and `logits_indices`\nTwo additional fields connect this packed representation to later stages.\n\n`slot_mapping` is one-dimensional with shape `[num_tokens]`, just like\n\n`input_ids`. For every token in the current step, it records the destination\n\nslot for the newly computed K and V tensors. Part 5 follows this value into the\n\nKV-cache write kernel.\n\n`logits_indices` has shape `[total_num_logits]`, which can be much smaller than\n\n`num_tokens`. To generate the next token, vLLM usually needs only the final\n\nhidden state selected for each request. If a prefill chunk processes four\n\ntokens, the first three positions do not need logits. `logits_indices` selects\n\nonly the positions that feed the sampler; Part 6 follows that path.\n\nThe model runner therefore produces a set of aligned flat views: `input_ids`\n\nand `positions` feed the forward pass; `query_start_loc` and `seq_lens` delimit\n\nrequests for attention; `slot_mapping` selects KV-cache destinations; and\n\n`logits_indices` selects the positions that require logits.\n\n`input_ids: [num_tokens]` rather than a padded `[batch, seq]` matrix.`query_start_loc` marks request boundaries in the packed query, while\n`seq_lens` supplies each request's visible KV length.`num_scheduled_tokens` connects scheduler decisions to the packed execution\nbatch, allowing prefill and decode work to coexist in one step.\nNext, the flat token sequence enters the model forward pass. The remaining\n\nquestion is how request boundaries and KV-block mappings reach attention\n\noperators buried inside many model layers.\n\nThe packed sequence now enters the model forward pass. Attention still needs\n\nstep-specific metadata that does not appear in an ordinary model signature.\n\nThat metadata includes `block_table`, `seq_lens`, and `query_start_loc`. The\n\nrequest set and lengths change each step, yet every attention layer needs access\n\nto the corresponding state. How does it reach operators buried inside the\n\nTransformer stack?\n\nA direct design would pass the metadata through every layer:\n\n`forward(input_ids, attn_metadata)`, then `layer(x, attn_metadata)`, then\n\n`attention(x, attn_metadata)`. In the OPT model definition, however, the\n\nforward signature contains `input_ids` and `positions` but no\n\n`attn_metadata`. The metadata enters through a different route.\n\n`forward_context`\nThe answer is in `execute_model`. The model's forward is wrapped in a context manager and runs:\n\n```\n# Conceptual skeleton corresponding to execute_model in v1/worker\nwith set_forward_context(attn_metadata, self.vllm_config, ...):\n    hidden_states = self.model(input_ids=input_ids, positions=positions)\n```\n\n`set_forward_context` (`vllm/forward_context.py:250`) installs\n\n`attn_metadata`, `slot_mapping`, and related state in a scoped\n\n`forward_context` while the model executes, then restores the previous context:\n\n``` python\n# vllm/forward_context.py:250 (structural excerpt)\ndef set_forward_context(attn_metadata, vllm_config, ..., slot_mapping=None, ...):\n    forward_context = create_forward_context(\n        attn_metadata, vllm_config, ..., slot_mapping, ...\n    )\n    try:\n        with override_forward_context(forward_context):   # ← install the context globally\n            yield                                          # ← model forward executes inside this scope\n    finally:\n        ...                                                # restore the previous context afterwards\n```\n\nThe metadata therefore travels through scoped context rather than through every\n\nmodel-layer signature. The ordinary forward path can keep tensor arguments such\n\nas `input_ids` and `positions` while attention retrieves its runtime metadata\n\nseparately.\n\nDeep inside the model, the attention layer retrieves that context. Consider\n\n`unified_attention_with_output`\n\n(`model_executor/layers/attention/attention.py:734`):\n\n```\n# model_executor/layers/attention/attention.py:734 (excerpt)\ndef unified_attention_with_output(query, key, value, output, layer_name, ...):\n    layer_name = _resolve_layer_name(layer_name)\n    # ↓ attn_metadata is retrieved rather than passed as an argument\n    attn_metadata, self, kv_cache, _ = get_attention_context(layer_name)\n    self.impl.forward(self, query, key, value, kv_cache, attn_metadata, output=output, ...)\n```\n\nThe function receives `layer_name`, not `attn_metadata` directly.\n\n`get_attention_context(layer_name)` resolves the layer and retrieves the\n\nruntime state from `forward_context` (`attention.py:648`):\n\n```\n# model_executor/layers/attention/attention.py:648 (excerpt)\ndef get_attention_context(layer_name):\n    forward_context = get_forward_context()          # ← retrieve the global context\n    attn_metadata = forward_context.attn_metadata    # retrieve this layer's metadata by layer_name\n    attn_layer = forward_context.no_compile_layers[layer_name]\n    kv_cache = attn_layer.kv_cache\n    slot_mapping = forward_context.slot_mapping.get(layer_name)\n    return attn_metadata, attn_layer, kv_cache, slot_mapping\n```\n\nThe resulting path is: `execute_model` installs the context, the model runs with\n\nits normal tensor inputs, and each attention operator uses `layer_name` to\n\nretrieve `attn_metadata`, `kv_cache`, and `slot_mapping` before dispatching to\n\n`self.impl.forward`.\n\n*Figure: `execute_model` installs `attn_metadata` in `forward_context`; each\nattention layer retrieves its state by `layer_name` before calling the backend.*\n\n`torch.compile`\nPassing data through context is less explicit than ordinary arguments, but it\n\nserves the compiled execution path. `torch.compile` benefits from stable graph\n\ninputs and shapes; threading a changing Python metadata object through every\n\nlayer can introduce graph breaks, guards, or recompilation. Keeping that object\n\noutside the model signature lets the compiled graph consume the tensor state it\n\nneeds through a controlled side channel. The exact compilation behavior still\n\ndepends on the selected backend and configuration, so this should be read as\n\nthe design goal rather than a guarantee that recompilation can never occur.\n\n`unified_attention_with_output` also accepts an apparently unused parameter,\n\n`kv_cache_dummy_dep`. It is intentionally present and is not discarded in the\n\nfunction body:\n\n```\n# attention.py:744\n# kv_cache_dummy_dep is not used but accepting it creates a data dependency\n# that ensures torch.compile preserves ordering between KV cache update and\n# attention forward.\n```\n\nKV-cache update must precede the attention read, but mutation alone may not\n\nexpose that ordering to the compiler. The dummy tensor makes the dependency\n\nexplicit in the graph: the update produces a value consumed by attention, which\n\nprevents the two operations from being reordered across that edge.\n\n`attn_metadata` stays out of the ordinary model-forward\nsignature and is installed in a scoped `layer_name` before entering the backend.\nThe next part follows `slot_mapping`, `block_table`, and the KV cache into the\n\nwrite and read paths.\n\nPagedAttention is closely associated with vLLM, but the virtual-memory analogy\n\nalone does not identify the actual address structures. This part follows the\n\nwrite table and the read table in code.\n\nThe KV cache is a pool of fixed-size blocks rather than one contiguous region\n\nper request. Each block stores K and V for `block_size` tokens (use\n\n`block_size=16` as a concrete example). A request's logical token sequence may\n\nspan physically scattered blocks. This paging model removes the need to reserve\n\none large contiguous allocation for every request and substantially reduces\n\nfragmentation from variable request lengths.\n\nHere, **block** means a KV-cache allocation unit. Part 2 will also use CUDA\n\nthread blocks and matrix tiles; those are separate execution and computation\n\nconcepts.\n\nThe two paths use different address structures: newly computed K/V is written\n\nby token, while attention reads a request's history by logical cache block.\n\nThe previous part showed the attention layer retrieving `slot_mapping` from\n\n`forward_context`. We can now follow it into the write kernel.\n\nEach cache block contains `block_size` token slots. Flattening physical block\n\nand in-block coordinates gives a cache-wide slot index:\n\n`physical_block_id × block_size + in_block_offset`. `slot_mapping`, with shape\n\n`[num_tokens]`, gives that destination for every token processed in the step.\n\nUnder the FlashAttention and FlashInfer paths, KV writing finally falls to `reshape_and_cache_flash` (bound as `ops.reshape_and_cache_flash` in `vllm/v1/attention/backends/fa_utils.py`, defined in `vllm/_custom_ops.py:2744`):\n\n``` python\n# vllm/_custom_ops.py:2744\ndef reshape_and_cache_flash(\n    key: torch.Tensor,          # K produced in this step\n    value: torch.Tensor,        # V produced in this step\n    key_cache: torch.Tensor,    # K block pool\n    value_cache: torch.Tensor,  # V block pool\n    slot_mapping: torch.Tensor, # ← destination slot for each token\n    kv_cache_dtype: str,\n    k_scale: torch.Tensor,\n    v_scale: torch.Tensor,\n) -> None:\n```\n\nNote: vLLM provides both `reshape_and_cache` and\n\n`reshape_and_cache_flash`, paired with different KV-cache layouts and read\n\nkernels. In V1, KV update is represented separately from the attention read;\n\nthe dependency described in Part 4 preserves their required order under\n\ncompilation.\n\nFor token `i`, the kernel reads the corresponding K and V vectors and writes\n\nthem to `slot_mapping[i]`. Decode commonly contributes one new position per\n\nrequest; prefill contributes a chunk. The mapping is a per-token destination\n\ntable, not the data itself.\n\n`block_table` per Request\nReading introduces logical-to-physical indirection. A query attends to visible\n\nhistorical K/V positions that may be scattered across physical cache blocks;\n\n`block_table` locates those blocks.\n\n`block_table` is part of `attn_metadata`. For each request, entry `j` stores the\n\nphysical cache block backing logical block `j`. With `block_size=16`, logical\n\ntokens 0–15 use the physical block named by `block_table[0]`, tokens 16–31 use\n\n`block_table[1]`, and so on. This logical-to-physical array is the concrete\n\nstructure behind the page-table analogy.\n\nOn the GPU, the actual reading occurs in `forward` in the FlashAttention backend, which hands `block_table` directly to the flash kernel (`vllm/v1/attention/backends/flash_attn.py:796`):\n\n```\n# vllm/v1/attention/backends/flash_attn.py:758 (excerpt)\ncu_seqlens_q = attn_metadata.query_start_loc\nseqused_k    = attn_metadata.seq_lens\nblock_table  = attn_metadata.block_table\n...\nflash_attn_varlen_func(\n    q=query[:num_actual_tokens],\n    k=key_cache,\n    v=value_cache,\n    out=output[:num_actual_tokens],\n    cu_seqlens_q=cu_seqlens_q,   # request boundaries in the flattened sequence (query_start_loc)\n    seqused_k=seqused_k,         # history length of each request (seq_lens)\n    block_table=block_table,     # ← page table: logical block → physical block\n    ...\n)\n```\n\nThe GPU kernel follows `block_table` while loading the K/V history used by the\n\ncurrent query. The equivalent address translation is easier to see in the CPU\n\nbackend. The following excerpt is simplified from\n\n`csrc/cpu/cpu_attn_impl.hpp`; the FlashAttention backend performs the same kind\n\nof logical-to-physical lookup within its own tiled implementation:\n\n```\n// Simplified from the attention read loop in csrc/cpu/cpu_attn_impl.hpp\nfor (block_idx = start_block_idx; block_idx < end_block_idx; ++block_idx) {\n    int physical_block_idx = block_table[block_idx];        // ← logical block number → physical block number\n    kv_cache_t* k_cache_block_ptr =\n        k_head_cache_ptr + physical_block_idx * kv_cache_num_blocks_stride;\n    // Compute Q@K using K from this block ...\n}\n```\n\n`physical_block_idx = block_table[block_idx]` is the essential page-table\n\nlookup: traverse logical blocks in sequence, resolve each physical block, then\n\nload K and V from that location.\n\nIn PagedAttention, two sets of addressing with different granularity are used for writing and reading.\n\nWrites use `slot_mapping` at token granularity: each newly computed token has\n\none destination slot in the cache.\n\nReads use `block_table` at block granularity. Each request owns a sequence of\n\nlogical blocks whose entries identify the physical blocks containing its KV\n\nhistory.\n\nKeeping the two addressing schemes separate is what makes paging practical.\n\nIn the OS analogy, `block_table` is the page table, the block pool is physical\n\nmemory, and `slot_mapping` gives the physical destination for a write. It is\n\nnot a literal implementation of virtual memory, but the indirection is closely\n\nrelated.\n\n*Figure: `slot_mapping` sends each newly computed token to one physical cache\nslot, while `block_table` maps a request's logical history to a sequence of\nphysical blocks for reading.*\n\n`reshape_and_cache_flash` uses `slot_mapping: [num_tokens]` to place\neach newly computed K/V pair in its destination slot.`physical = block_table[logical]` while\ngathering the request's KV history.\nAfter attention consumes the updated cache, the model produces hidden states.\n\nThe final part follows selected hidden states through logits and sampling, then\n\nshows how completed requests release their blocks.\n\nThe model forward returns a packed set of hidden states. Only selected positions\n\nneed logits for next-token sampling.\n\nThe `logits_indices` prepared in Part 3 now determines which hidden states are\n\nprojected into logits.\n\nThe model computes a hidden state for every scheduled token, but ordinary\n\nnext-token generation needs logits only at selected positions—typically the\n\nlast scheduled position for each request. For a four-token prefill chunk, the\n\nfirst three hidden states therefore need not pass through `lm_head` for that\n\nsampling step.\n\n`logits_indices` selects those hidden states before the vocabulary projection.\n\nBecause `lm_head` projects into a vocabulary with tens or hundreds of thousands\n\nof entries, avoiding unused positions saves substantial matrix-multiplication\n\nwork.\n\n``` php\nhidden_states[num_tokens]  --index_select(logits_indices)-->  keep only the required final position for each request\n                           --lm_head-->  logits[num_reqs, vocab_size]\n```\n\n`Sampler` converts each `vocab_size`-dimensional logits vector into a token ID\n\naccording to temperature, top-p, top-k, or greedy selection.\n\nThe sampling results are packed into `ModelRunnerOutput`—whose central field is\n\n`sampled_token_ids`—and sent back to `EngineCore` across the IPC boundary. This\n\ncompletes the worker-side work for the scheduler step.\n\nWhen `sampled_token_ids` returns to EngineCore,\n\n`scheduler.update_from_output` (`v1/core/sched/scheduler.py:1283`) performs the\n\nthird stage of `EngineCore.step()`: applying model output to request state.\n\nFirst, take out the new token:\n\n```\n# scheduler.py:1363 (excerpt)\nreq_index = model_runner_output.req_id_to_index[req_id]\ngenerated_token_ids = sampled_token_ids[req_index] if sampled_token_ids else []\n```\n\nSecond, `_update_request_with_output` appends the new token and evaluates stop\n\nconditions such as EOS, `max_tokens`, and configured stop strings:\n\n```\n# scheduler.py:1407 (excerpt)\nif new_token_ids:\n    new_token_ids, stopped = self._update_request_with_output(request, new_token_ids)\n```\n\nThird, when a request stops, its KV blocks are returned to the pool:\n\n```\n# scheduler.py:1474 (excerpt)\nif stopped:\n    finish_reason = request.get_finished_reason()\n    finished = self._handle_stopped_request(request)\n    if finished:\n        kv_transfer_params = self._free_request(request)   # ← release its KV blocks\n    ...\n    stopped_running_reqs.add(request)\n```\n\nAfter the cycle ends, remove these stopped requests from the `running` queue:\n\n```\n# scheduler.py:1528 (excerpt)\nif stopped_running_reqs:\n    self.running = remove_all(self.running, stopped_running_reqs)\n```\n\nThis closes the continuous-batching loop. When a request finishes,\n\n`_free_request` returns its blocks to the pool and removes one entry from\n\n`running`. On the next step, `schedule()` can spend those blocks on requests in\n\n`waiting`. Resource reclamation is therefore part of admission control, not an\n\nunrelated cleanup detail.\n\nAn unfinished request remains eligible in `running`; a later scheduling step can\n\nadvance it again.\n\nFinally, EngineCore packages the results as `EngineCoreOutputs` and sends them\n\nto the main process over IPC. `LLMEngine.step()` consumes them through\n\n`get_output()`, detokenizes token IDs, assembles `RequestOutput`, and returns it\n\nto the caller. That completes the path from an API request to generated text.\n\n*Figure: `logits_indices` selects the required hidden states, `lm_head`\nproduces logits, and sampling yields new token IDs. Finished requests release\ntheir KV blocks; unfinished requests remain active.*\n\n`lm_head` work for unused intermediate positions.`Sampler` produces token IDs according to each request's sampling parameters\nand returns them in `ModelRunnerOutput`.` update_from_output` appends new tokens, evaluates stopping conditions, and\ncalls `_free_request` for completed requests.\nThe six parts reduce vLLM's request path to a scheduler, a paged memory manager,\n\nand a GPU execution pipeline. Four design choices are especially reusable:\n\nFirst, the scheduler expresses both prefill and decode as per-request token\n\nprogress. The execution layer then packs those scheduled tokens into one\n\ndimension. Chunked prefill and continuous batching build on the same contract\n\ninstead of requiring separate batch formats.\n\nSecond, V1 preempts by recomputing rather than swapping. Under KV-cache\n\npressure, it frees the victim's blocks, resets `num_computed_tokens`, and\n\nrequeues the request for prefill instead of moving KV state to CPU memory and\n\nback over PCIe. This trades additional computation for less host-device data\n\nmovement.\n\nThird, `forward_context` keeps step-specific `attn_metadata` out of the forward\n\nsignature so `torch.compile` can reuse a stable graph. A dummy tensor dependency\n\nthen constrains operator ordering. The indirection is less obvious to read, but\n\nit serves compilation and execution correctness.\n\nFourth, writes are addressed per token while reads are resolved per block.\n\n`slot_mapping` selects each token's write slot; `block_table` maps a request's\n\nlogical KV blocks to physical blocks. At its core, the page-table lookup is\n\n`physical = block_table[logical]`.\n\nThe same discipline appears throughout the path: identify whether the scarce\n\nresource is compute, bandwidth, KV capacity, TTFT, or inter-token latency, then\n\nmake the trade-off explicit. vLLM's throughput does not come from one isolated\n\ntrick. It comes from small mechanisms that compose: a budget clamp expressed by\n\n`min`, one level of `block_table` indirection, and a deliberate\n\n`num_computed_tokens = 0` on preemption.\n\nTo keep the path traceable, this article stops at the Python/CUDA boundary. It\n\ndoes not cover:\n\n`kv_cache_manager`, including reference\ncounting and prefix-cache hash reuse. The article treats it only as a block\nallocator.\n*This walkthrough targets the vLLM 0.22.0 V1 architecture. Source locations are included so readers can compare the excerpts with their installed version. The\neditable Excalidraw sources for Figures 2–8 are published with this site.*", "url": "https://wpnews.pro/news/inside-vllm-following-one-request-from-the-api-to-gpu-execution", "canonical_source": "https://dev.to/yuan_lei_e631e36865ed370b/inside-vllm-following-one-request-from-the-api-to-gpu-execution-1ja6", "published_at": "2026-09-07 12:52:14+00:00", "updated_at": "2026-09-07 12:57:31.260157+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["vLLM", "EngineCore", "PagedAttention", "CUDA", "FlashAttention", "Triton", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/inside-vllm-following-one-request-from-the-api-to-gpu-execution", "markdown": "https://wpnews.pro/news/inside-vllm-following-one-request-from-the-api-to-gpu-execution.md", "text": "https://wpnews.pro/news/inside-vllm-following-one-request-from-the-api-to-gpu-execution.txt", "jsonld": "https://wpnews.pro/news/inside-vllm-following-one-request-from-the-api-to-gpu-execution.jsonld"}}