Inside vLLM: Following One Request from the API to GPU Execution

A developer's source-code walkthrough of vLLM 0.22's V1 execution path traces a single offline inference request from the LLM.generate() API through inter-process communication, scheduling, GPU execution, and paged KV-cache access. The article explains how EngineCore runs in a child process with a continuous busy loop, separating scheduling from execution, and details the three core stages of EngineCore.step(): scheduling, model execution, and state update.

Article 1 of 3 · vLLM Internals This English edition is adapted from the published Chinese article on Zhihu https://zhuanlan.zhihu.com/p/2060033883849204419 . It preserves the source-code references, experimental boundaries, and reproducible artifacts while adapting the structure for an international engineering audience. Series: Part 1 · Request lifecycle · Part 2 · CUDA kernels and paged attention https://jacklei0901.github.io/articles/inside-vllm-cuda-kernels/ · Part 3 · FlashAttention from PyTorch to Triton https://jacklei0901.github.io/articles/building-flashattention-pytorch-triton/ This article follows one offline inference request through vLLM V1: from LLM.generate and inter-process communication to scheduling, input flattening, GPU model execution, paged KV-cache access, sampling, and resource reclamation. The goal is to answer one concrete question: what happens behind the call to llm.generate before the completed result reaches the caller? The discussion assumes familiarity with Transformer inference, including prefill, decode, KV caching, and autoregressive generation. It focuses on how those concepts appear in vLLM source code rather than reteaching the model architecture. Version scope. The source references were verified against vLLM 0.22.0; this edition was checked on September 3, 2026. vLLM evolves quickly, so some filenames and call boundaries will move. The long-lived ideas—continuous batching, token budgets, paged KV allocation, and the separation between scheduling and execution—are the real subject of the article. Many introductions stop at the useful analogy that PagedAttention manages the KV cache much like virtual memory manages pages. The analogy does not tell us how a request is admitted, how variable-length requests become a flat token batch, or what the page table looks like at the kernel boundary. The answers are in the source. This article follows vLLM 0.22's V1 execution path from the public entry point to the CUDA boundary. It is a source-code walkthrough, not an API tutorial. Start with the process boundary and the engine loop. In vLLM V1, EngineCore runs continuously in a child process. Requests enter and leave the active set between steps; the caller does not drive GPU execution one token at a time. One detail shapes the entire path: in this configuration, vLLM 0.22 separates the caller-facing engine from EngineCore with a process boundary. php REQUEST SUBMISSION · main process - EngineCore child Caller └─ LLM.generate prompts └─ LLMEngine.add request └─ EngineCoreClient └─ IPC input queue ──────────────▶ scheduler.add request CONTINUOUS ENGINE LOOP · EngineCore child run busy loop ├─ receive newly submitted requests └─ repeat EngineCore.step 1. scheduler.schedule └─ spend token / KV-cache budgets - SchedulerOutput 2. model executor.execute model └─ GPU forward pass and sampling - sampled tokens 3. scheduler.update from output ├─ update request state and release finished KV blocks └─ EngineCoreOutputs ───────────────▶ IPC output queue RESULT CONSUMPTION · EngineCore child - main process IPC output queue └─ LLMEngine.step / get output └─ detokenize and assemble RequestOutput └─ return completed results to LLM.generate and the caller This map establishes three boundaries used throughout the article. First, the main process consumes results; EngineCore produces them. Calling LLMEngine.step does not tell the GPU to perform one step. It retrieves already-produced output through get output , then detokenizes and assembles the user-facing result. Scheduling and model execution advance independently inside EngineCore's busy loop. Second, EngineCore.step has three core stages: scheduling, execution, and state update. The rest of the article expands those stages one at a time. Third, this loop implements continuous batching. The request set changes from one step to the next: stage 3 removes completed requests and releases their resources, while the next call to schedule can admit waiting work. The GPU therefore does not have to wait for one fixed batch to finish before admitting another request. The rest of the article expands that loop into six parts: LLM and LLMEngine submit work to EngineCore. schedule spends token and KV-cache budgets. Each part keeps only the code needed to connect one boundary to the next. generate Start with the public interface: python from vllm import LLM, SamplingParams llm = LLM model="facebook/opt-125m" outputs = llm.generate "Once upon a time" , SamplingParams max tokens=50 Those two lines initialize the entire engine stack. Start with the LLM class. In entrypoints/llm.py , LLM delegates engine construction to one factory call: entrypoints/llm.py self.llm engine = LLMEngine.from engine args engine args=engine args, usage context=UsageContext.LLM CLASS, LLM packages model, dtype, and other configuration into EngineArgs , then uses the factory method to create LLMEngine . It is the public facade; LLMEngine owns the request-processing path followed below. generate returns a list RequestOutput , one complete result per request. The generated text is available as output.outputs 0 .text . The distinction is useful throughout the article: LLM is the facade, while LLMEngine connects that facade to EngineCore. The internal logic of generate , after stripping away the details, is as follows: Conceptual skeleton for prompt in prompts: self.llm engine.add request prompt, ... 1. Register the request with the engine outputs = while self.llm engine.has unfinished requests : 2. While any request is unfinished step outputs = self.llm engine.step advance the engine by one step for out in step outputs: if out.finished: outputs.append out return outputs 3. Return after collecting every result This while ... step loop continues until every submitted request is finished. Unlike a hand-written for in range max tokens loop over one sequence, it tracks a set of requests whose lengths and completion times differ. step Does Not Submit the Request The name step can suggest a synchronous "submit, execute, return" call. The implementation does something different: python v1/engine/llm engine.py simplified def step self : 1 Fetch outputs already produced by EngineCore outputs = self.engine core.get output 2 Decode tokens and evaluate stopping conditions processed = self.output processor.process outputs outputs.outputs, ... 3 Handle requests terminated by a stop string self.engine core.abort requests processed.reqs to abort 4 Record statistics ... return processed.request outputs step never submits a request or directly drives model execution. It calls get output to consume results. Submission already happened in add request ; the actual computation runs independently elsewhere. This description assumes the default multiprocess V1 configuration. If VLLM ENABLE V1 MULTIPROCESSING=0 is set for debugging, the in-process client drives EngineCore from get output and the child-process producer/consumer boundary collapses. The scheduling, execution, and update stages discussed below still apply, but they no longer run in a separate EngineCore process. This producer-consumer split is asynchronous across the process boundary. In configurations that enable asynchronous scheduling, the startup log also states: INFO ... Asynchronous scheduling is enabled. That "elsewhere" is a separate child process. A running server prints a prefix such as EngineCore pid=8884 in its logs: this is the EngineCore process that performs scheduling and model execution. The main process LLMEngine is the consumer: get output receives results, decodes them, and returns them to the caller. The child process EngineCore is the producer: its busy loop schedules requests, runs the model, produces tokens, and writes results to an IPC queue. This lets host-side output processing overlap EngineCore execution. In an online server, request tokenization and admission can also proceed independently of an already-running GPU step. The IPC queues separate those execution timelines. generate is the while has unfinished requests : step loop. LLMEngine.step consumes and decodes output; the separate EngineCore process performs scheduling and execution asynchronously. Next we cross the IPC boundary and enter EngineCore , whose busy loop has three core stages: schedule, execute, and update. Figure: ownership among vLLM's core classes. LLM owns LLMEngine , which communicates with EngineCore ; EngineCore in turn owns the scheduler, model executor, and KV-cache manager. The horizontal line is the process boundary: main process/consumer above, child process/producer below. The previous section stopped at the child-process boundary. Each iteration of the EngineCore busy loop runs the three stages of step ; this section starts with scheduler.schedule . Continuous batching is not a standalone module. It emerges from the decisions made on every call to schedule . One useful mental model is an operating-system scheduler whose scarce resources are token-compute budget and GPU KV-cache pages. The scheduler lives in v1/core/sched/scheduler.py ; its central method is schedule scheduler.py:329 . The file is large, but this path only needs three pieces: the main loop, preemption, and the final packaged output. The V1 scheduler can be summarized in one sentence: during each engine step, advance as many requests as possible without exceeding the token budget or the available KV blocks. These concepts map naturally to familiar operating-system terms: token budget : the maximum number of tokens processed in one step max num batched tokens=8192 in the example startup log , analogous to a bounded compute time slice. waiting : requests waiting to begin or continue prefill, analogous to a ready queue. running : admitted requests eligible to advance in the current step. The comment at the start of schedule is the best guide to its model: v1/core/sched/scheduler.py:330 There's no "decoding phase" nor "prefill phase" in the scheduler. Each request just has the num computed tokens and num tokens with spec. At each step, the scheduler tries to assign tokens to the requests so that each request's num computed tokens can catch up its num tokens with spec. The V1 scheduler does not maintain separate prefill and decode phases. Instead, it compares each request's num computed tokens with num tokens with spec and decides how many tokens to advance in the current step. A decode request commonly advances by one token; a prefill request may advance by a larger chunk. The same progress model also accommodates chunked prefill and prefix-cache hits. The following sketch is a useful map of schedule : One schedule step: pass 1 RUNNING first decode ──── no free block → preempt from the tail pass 2 fill from WAITING prefill ── over budget → clamp or defer pack SchedulerOutput the execution plan Figure: one schedule step. The scheduler considers running requests, admits waiting requests with any remaining budget, handles allocation failure through preemption, and packages the result as SchedulerOutput . The first substantive loop in schedule traverses running scheduler.py:364 : v1/core/sched/scheduler.py:364 First, schedule the RUNNING requests. req index = 0 while req index < len self.running and token budget 0: request = self.running req index num new tokens = request.num tokens with spec + request.num output placeholders - request.num computed tokens num new tokens = min num new tokens, token budget ... The source comments make the order explicit: running requests are considered before waiting requests. This ordering tends to protect inter-token latency for requests that are already generating, although the observed latency trade-off still depends on the configured scheduling policy and workload. It also lets requests that already hold KV blocks continue making progress toward completion and eventual reclamation. These are useful consequences of the order, not a claim that every running request always outranks every waiting request under all policies. The waiting queue is an admission queue, not a tokenization cache. Requests in it are waiting for scheduler resources; prompt preprocessing has already happened before this point. After considering running requests, the scheduler uses any remaining budget to admit work from waiting scheduler.py:544 . This is where the chunked-prefill budget clamp appears. A waiting request may have 5,000 prompt tokens while only 2,000 tokens remain in the current step's budget. The relevant control flow is: Budget clamp inside the waiting loop if not self.scheduler config.enable chunked prefill and num new tokens token budget : If chunked prefill is disabled, we can stop the scheduling here. break num new tokens = min num new tokens, token budget assert num new tokens 0 Two details matter here. First, break does not drop the request. It exits the waiting-loop because the remaining budget cannot admit another request in this step. The request stays in waiting and is reconsidered the next time schedule runs. With chunked prefill enabled, min admits the portion that fits and leaves the rest for a later step. With it disabled, an over-budget request stays in waiting and is retried as a whole. Neither path loses the request; removal happens only after normal completion or an explicit abort. A useful rule when reading schedulers is to ask where an item goes after a break or continue . Here it remains in the same queue for a later step, just as an OS process that misses one time slice remains runnable. Second, chunked prefill reduces to a budget clamp: num new tokens = min num new tokens, token budget . A long prompt contributes only the tokens that fit in the current step; the remainder is scheduled later. The full policy considers more state, but this clamp is what divides a long prefill across scheduler steps. The running-request loop has the same clamp at scheduler.py:392 . Because V1 does not maintain separate prefill and decode phases, the budget logic applies uniformly to both. Another branch appears when a running request needs more KV blocks and allocate slots cannot satisfy the request. The allocation-retry loop beginning at scheduler.py:443 can then preempt a victim: v1/core/sched/scheduler.py:443 simplified while True: new blocks = self.kv cache manager.allocate slots request, num new tokens, ... if new blocks is not None: break Allocation succeeded; schedule normally Allocation failed; choose a victim to preempt preempted req = self.running.pop Default policy: pop from the tail LIFO self. preempt request preempted req, scheduled timestamp if preempted req == request: break No victim remains; defer this request Under the default policy, self.running.pop selects a victim from the tail LIFO . This usually victimizes a more recently admitted request with less accumulated progress, reducing the work discarded, although queue position is not a universal measure of recomputation cost. With PRIORITY scheduling, victim selection also considers priority scheduler.py:456 . preempt request scheduler.py:929 defines what happens to the preempted request's KV-cache state: python v1/core/sched/scheduler.py:929 def preempt request self, request, timestamp : self.kv cache manager.free request ← release its KV blocks immediately self.encoder cache manager.free request request.status = RequestStatus.PREEMPTED request.num computed tokens = 0 ← reset computed progress ... self.waiting.prepend request request ← prepend it to the waiting queue Two details matter here. First, num computed tokens = 0 records recomputation semantics: the scheduler no longer treats the request's previous progress as resident KV state. When rescheduled, it must rebuild the required state, subject to any cache reuse the current configuration can legitimately recover. This is different from swapping the victim's KV tensors to host memory and restoring them later. The trade-off is workload-dependent. Swapping consumes host-device bandwidth and host memory; recomputation consumes additional GPU work. vLLM V1 chooses the latter for this path, avoiding a KV transfer over PCIe at the cost of re-running model computation. Figure: two preemption strategies. Swapping preserves KV state through a host-device transfer; recomputation releases the blocks and later rebuilds the required state. The second detail is prepend request , an anti-starvation measure. A preempted request returns to the head of waiting rather than the tail. Appending it behind every newly arrived request could cause repeated preemption and indefinite delay; prepending gives the sacrificed request priority on the next admission pass. SchedulerOutput schedule packages its decisions into SchedulerOutput defined in v1/core/sched/output.py:181 and hands that execution plan to the model runner. The execution layer consumes the decision; it does not reschedule the requests. Many fields support optional features such as speculative decoding, multimodality, structured output, and connectors. Four core fields are enough to enter the execution layer: v1/core/sched/output.py:181 selected core fields @dataclass class SchedulerOutput: scheduled new reqs: list NewRequestData First scheduling: send complete request data scheduled cached reqs: CachedRequestData Previously scheduled: send only the delta num scheduled tokens: dict str, int req id - tokens processed in this step total num scheduled tokens: int ... finished req ids: set str Completed requests; tell the worker to reclaim state ... new block ids to zero: list int | None = None Newly allocated blocks that the worker must zero Their roles become clearer at the execution boundary. scheduled new reqs carries the state needed when a request is first scheduled; scheduled cached reqs carries updates for requests already known to the model runner. This avoids rebuilding and transmitting the complete request state on every step—a state-cache plus incremental-update pattern. num scheduled tokens implements the unified progress model: it records how many tokens each request will process in this step. The execution layer can use those counts to pack prefill and decode work into one flat token batch. finished req ids tells the worker which completed requests can release their cached state, analogous to reclaiming OS resources after a process exits. new block ids to zero identifies freshly allocated cache blocks that must be cleared before the forward pass when KV-cache zeroing is enabled. Zeroing keeps stale data or NaNs from an earlier owner from corrupting attention or state-space-model computation. The field is conditional; not every configuration needs this step. Several scheduler concepts have useful operating-system analogies: | Operating system concepts | vLLM Scheduler correspondence | |---|---| | Ready Queue | waiting | | Run Queue | running | | Physical page frame allocator | kv cache manager | | Page fault/out of memory | allocate slots returns None | | Process swap | Preemption by releasing KV state and recomputing | | time slice | token budget | | Priority compensation after preemption | prepend request returns the victim to the head of waiting | | Output of scheduling decision | SchedulerOutput | Continuous batching follows from rebuilding the execution batch at every schedule call. Completed requests leave, waiting requests enter, long prompts may be split into chunks, and allocation pressure may trigger preemption. The engine can therefore reuse newly available capacity without waiting for an earlier fixed batch to finish in full. Next, follow this SchedulerOutput plan into the execution layer and see how variable-length requests become flat tensors for the model. The SchedulerOutput from the previous part reaches GPUModelRunner in the worker. Suppose it assigns four tokens to request A and one token each to requests B and C. How does the runner turn those unequal slices into tensors for the GPU? A conventional implementation might pad every request to the same length and stack a batch, seq len tensor. This path instead packs the scheduled tokens. The path enters GPUModelRunner.execute model v1/worker/gpu model runner.py:3913 and calls prepare inputs gpu model runner.py:1839 . InputBatch v1/worker/gpu/input batch.py:36 holds the prepared state: v1/worker/gpu/input batch.py:36 excerpt class InputBatch: req ids: list str num reqs: int num reqs tokens processed for each request in this step num scheduled tokens: np.ndarray sum num scheduled tokens : total tokens in this step num tokens: int num reqs + 1 request boundaries in the flattened sequence prefix sum query start loc: torch.Tensor num reqs current total length of each request, including history seq lens: torch.Tensor num tokens ← one-dimensional, not batch, seq len input ids: torch.Tensor num tokens positions: torch.Tensor total num logits positions that actually require logits logits indices: torch.Tensor The key shape is input ids: num tokens : the scheduled tokens form one packed dimension rather than a padded batch matrix. V1 concatenates the tokens selected for the current step into one packed sequence, regardless of which request they belong to or whether they represent prefill or decode work. In the example, A contributes a four-token prefill chunk, while B and C each contribute one decode token. If their current positions are 50 and 30, the packed tensors look like this: input ids = A0, A1, A2, A3, B0, C0 six values, one-dimensional positions = 0, 1, 2, 3, 50, 30 each token's position within its own request A starts at position 0; B and C retain positions from their own sequences rather than being renumbered within the packed batch. Here num scheduled tokens = 4, 1, 1 and query start loc = 0, 4, 5, 6 . The latter is a prefix sum that marks each request's range. Attention combines these boundaries with seq lens to keep requests separate. This design has two advantages. First, packing avoids computation on padding positions. Every scheduled position corresponds to actual request work, although kernels may still pad the packed token count internally for CUDA graphs or alignment. Second, packing carries the scheduler's unified token-count abstraction into execution. The runner uses num scheduled tokens to place each request's slice in the flat sequence. Prefill and decode work can therefore coexist in one step, while metadata preserves their separate boundaries. Figure: three variable-length requests—a prefill chunk and two decode requests—packed into one-dimensional input ids , positions , and slot mapping arrays. slot mapping and logits indices Two additional fields connect this packed representation to later stages. slot mapping is one-dimensional with shape num tokens , just like input ids . For every token in the current step, it records the destination slot for the newly computed K and V tensors. Part 5 follows this value into the KV-cache write kernel. logits indices has shape total num logits , which can be much smaller than num tokens . To generate the next token, vLLM usually needs only the final hidden state selected for each request. If a prefill chunk processes four tokens, the first three positions do not need logits. logits indices selects only the positions that feed the sampler; Part 6 follows that path. The model runner therefore produces a set of aligned flat views: input ids and positions feed the forward pass; query start loc and seq lens delimit requests for attention; slot mapping selects KV-cache destinations; and logits indices selects the positions that require logits. input ids: num tokens rather than a padded batch, seq matrix. query start loc marks request boundaries in the packed query, while seq lens supplies each request's visible KV length. num scheduled tokens connects scheduler decisions to the packed execution batch, allowing prefill and decode work to coexist in one step. Next, the flat token sequence enters the model forward pass. The remaining question is how request boundaries and KV-block mappings reach attention operators buried inside many model layers. The packed sequence now enters the model forward pass. Attention still needs step-specific metadata that does not appear in an ordinary model signature. That metadata includes block table , seq lens , and query start loc . The request set and lengths change each step, yet every attention layer needs access to the corresponding state. How does it reach operators buried inside the Transformer stack? A direct design would pass the metadata through every layer: forward input ids, attn metadata , then layer x, attn metadata , then attention x, attn metadata . In the OPT model definition, however, the forward signature contains input ids and positions but no attn metadata . The metadata enters through a different route. forward context The answer is in execute model . The model's forward is wrapped in a context manager and runs: Conceptual skeleton corresponding to execute model in v1/worker with set forward context attn metadata, self.vllm config, ... : hidden states = self.model input ids=input ids, positions=positions set forward context vllm/forward context.py:250 installs attn metadata , slot mapping , and related state in a scoped forward context while the model executes, then restores the previous context: python vllm/forward context.py:250 structural excerpt def set forward context attn metadata, vllm config, ..., slot mapping=None, ... : forward context = create forward context attn metadata, vllm config, ..., slot mapping, ... try: with override forward context forward context : ← install the context globally yield ← model forward executes inside this scope finally: ... restore the previous context afterwards The metadata therefore travels through scoped context rather than through every model-layer signature. The ordinary forward path can keep tensor arguments such as input ids and positions while attention retrieves its runtime metadata separately. Deep inside the model, the attention layer retrieves that context. Consider unified attention with output model executor/layers/attention/attention.py:734 : model executor/layers/attention/attention.py:734 excerpt def unified attention with output query, key, value, output, layer name, ... : layer name = resolve layer name layer name ↓ attn metadata is retrieved rather than passed as an argument attn metadata, self, kv cache, = get attention context layer name self.impl.forward self, query, key, value, kv cache, attn metadata, output=output, ... The function receives layer name , not attn metadata directly. get attention context layer name resolves the layer and retrieves the runtime state from forward context attention.py:648 : model executor/layers/attention/attention.py:648 excerpt def get attention context layer name : forward context = get forward context ← retrieve the global context attn metadata = forward context.attn metadata retrieve this layer's metadata by layer name attn layer = forward context.no compile layers layer name kv cache = attn layer.kv cache slot mapping = forward context.slot mapping.get layer name return attn metadata, attn layer, kv cache, slot mapping The resulting path is: execute model installs the context, the model runs with its normal tensor inputs, and each attention operator uses layer name to retrieve attn metadata , kv cache , and slot mapping before dispatching to self.impl.forward . Figure: execute model installs attn metadata in forward context ; each attention layer retrieves its state by layer name before calling the backend. torch.compile Passing data through context is less explicit than ordinary arguments, but it serves the compiled execution path. torch.compile benefits from stable graph inputs and shapes; threading a changing Python metadata object through every layer can introduce graph breaks, guards, or recompilation. Keeping that object outside the model signature lets the compiled graph consume the tensor state it needs through a controlled side channel. The exact compilation behavior still depends on the selected backend and configuration, so this should be read as the design goal rather than a guarantee that recompilation can never occur. unified attention with output also accepts an apparently unused parameter, kv cache dummy dep . It is intentionally present and is not discarded in the function body: attention.py:744 kv cache dummy dep is not used but accepting it creates a data dependency that ensures torch.compile preserves ordering between KV cache update and attention forward. KV-cache update must precede the attention read, but mutation alone may not expose that ordering to the compiler. The dummy tensor makes the dependency explicit in the graph: the update produces a value consumed by attention, which prevents the two operations from being reordered across that edge. attn metadata stays out of the ordinary model-forward signature and is installed in a scoped layer name before entering the backend. The next part follows slot mapping , block table , and the KV cache into the write and read paths. PagedAttention is closely associated with vLLM, but the virtual-memory analogy alone does not identify the actual address structures. This part follows the write table and the read table in code. The KV cache is a pool of fixed-size blocks rather than one contiguous region per request. Each block stores K and V for block size tokens use block size=16 as a concrete example . A request's logical token sequence may span physically scattered blocks. This paging model removes the need to reserve one large contiguous allocation for every request and substantially reduces fragmentation from variable request lengths. Here, block means a KV-cache allocation unit. Part 2 will also use CUDA thread blocks and matrix tiles; those are separate execution and computation concepts. The two paths use different address structures: newly computed K/V is written by token, while attention reads a request's history by logical cache block. The previous part showed the attention layer retrieving slot mapping from forward context . We can now follow it into the write kernel. Each cache block contains block size token slots. Flattening physical block and in-block coordinates gives a cache-wide slot index: physical block id × block size + in block offset . slot mapping , with shape num tokens , gives that destination for every token processed in the step. Under the FlashAttention and FlashInfer paths, KV writing finally falls to reshape and cache flash bound as ops.reshape and cache flash in vllm/v1/attention/backends/fa utils.py , defined in vllm/ custom ops.py:2744 : python vllm/ custom ops.py:2744 def reshape and cache flash key: torch.Tensor, K produced in this step value: torch.Tensor, V produced in this step key cache: torch.Tensor, K block pool value cache: torch.Tensor, V block pool slot mapping: torch.Tensor, ← destination slot for each token kv cache dtype: str, k scale: torch.Tensor, v scale: torch.Tensor, - None: Note: vLLM provides both reshape and cache and reshape and cache flash , paired with different KV-cache layouts and read kernels. In V1, KV update is represented separately from the attention read; the dependency described in Part 4 preserves their required order under compilation. For token i , the kernel reads the corresponding K and V vectors and writes them to slot mapping i . Decode commonly contributes one new position per request; prefill contributes a chunk. The mapping is a per-token destination table, not the data itself. block table per Request Reading introduces logical-to-physical indirection. A query attends to visible historical K/V positions that may be scattered across physical cache blocks; block table locates those blocks. block table is part of attn metadata . For each request, entry j stores the physical cache block backing logical block j . With block size=16 , logical tokens 0–15 use the physical block named by block table 0 , tokens 16–31 use block table 1 , and so on. This logical-to-physical array is the concrete structure behind the page-table analogy. On the GPU, the actual reading occurs in forward in the FlashAttention backend, which hands block table directly to the flash kernel vllm/v1/attention/backends/flash attn.py:796 : vllm/v1/attention/backends/flash attn.py:758 excerpt cu seqlens q = attn metadata.query start loc seqused k = attn metadata.seq lens block table = attn metadata.block table ... flash attn varlen func q=query :num actual tokens , k=key cache, v=value cache, out=output :num actual tokens , cu seqlens q=cu seqlens q, request boundaries in the flattened sequence query start loc seqused k=seqused k, history length of each request seq lens block table=block table, ← page table: logical block → physical block ... The GPU kernel follows block table while loading the K/V history used by the current query. The equivalent address translation is easier to see in the CPU backend. The following excerpt is simplified from csrc/cpu/cpu attn impl.hpp ; the FlashAttention backend performs the same kind of logical-to-physical lookup within its own tiled implementation: // Simplified from the attention read loop in csrc/cpu/cpu attn impl.hpp for block idx = start block idx; block idx < end block idx; ++block idx { int physical block idx = block table block idx ; // ← logical block number → physical block number kv cache t k cache block ptr = k head cache ptr + physical block idx kv cache num blocks stride; // Compute Q@K using K from this block ... } physical block idx = block table block idx is the essential page-table lookup: traverse logical blocks in sequence, resolve each physical block, then load K and V from that location. In PagedAttention, two sets of addressing with different granularity are used for writing and reading. Writes use slot mapping at token granularity: each newly computed token has one destination slot in the cache. Reads use block table at block granularity. Each request owns a sequence of logical blocks whose entries identify the physical blocks containing its KV history. Keeping the two addressing schemes separate is what makes paging practical. In the OS analogy, block table is the page table, the block pool is physical memory, and slot mapping gives the physical destination for a write. It is not a literal implementation of virtual memory, but the indirection is closely related. Figure: slot mapping sends each newly computed token to one physical cache slot, while block table maps a request's logical history to a sequence of physical blocks for reading. reshape and cache flash uses slot mapping: num tokens to place each newly computed K/V pair in its destination slot. physical = block table logical while gathering the request's KV history. After attention consumes the updated cache, the model produces hidden states. The final part follows selected hidden states through logits and sampling, then shows how completed requests release their blocks. The model forward returns a packed set of hidden states. Only selected positions need logits for next-token sampling. The logits indices prepared in Part 3 now determines which hidden states are projected into logits. The model computes a hidden state for every scheduled token, but ordinary next-token generation needs logits only at selected positions—typically the last scheduled position for each request. For a four-token prefill chunk, the first three hidden states therefore need not pass through lm head for that sampling step. logits indices selects those hidden states before the vocabulary projection. Because lm head projects into a vocabulary with tens or hundreds of thousands of entries, avoiding unused positions saves substantial matrix-multiplication work. php hidden states num tokens --index select logits indices -- keep only the required final position for each request --lm head-- logits num reqs, vocab size Sampler converts each vocab size -dimensional logits vector into a token ID according to temperature, top-p, top-k, or greedy selection. The sampling results are packed into ModelRunnerOutput —whose central field is sampled token ids —and sent back to EngineCore across the IPC boundary. This completes the worker-side work for the scheduler step. When sampled token ids returns to EngineCore, scheduler.update from output v1/core/sched/scheduler.py:1283 performs the third stage of EngineCore.step : applying model output to request state. First, take out the new token: scheduler.py:1363 excerpt req index = model runner output.req id to index req id generated token ids = sampled token ids req index if sampled token ids else Second, update request with output appends the new token and evaluates stop conditions such as EOS, max tokens , and configured stop strings: scheduler.py:1407 excerpt if new token ids: new token ids, stopped = self. update request with output request, new token ids Third, when a request stops, its KV blocks are returned to the pool: scheduler.py:1474 excerpt if stopped: finish reason = request.get finished reason finished = self. handle stopped request request if finished: kv transfer params = self. free request request ← release its KV blocks ... stopped running reqs.add request After the cycle ends, remove these stopped requests from the running queue: scheduler.py:1528 excerpt if stopped running reqs: self.running = remove all self.running, stopped running reqs This closes the continuous-batching loop. When a request finishes, free request returns its blocks to the pool and removes one entry from running . On the next step, schedule can spend those blocks on requests in waiting . Resource reclamation is therefore part of admission control, not an unrelated cleanup detail. An unfinished request remains eligible in running ; a later scheduling step can advance it again. Finally, EngineCore packages the results as EngineCoreOutputs and sends them to the main process over IPC. LLMEngine.step consumes them through get output , detokenizes token IDs, assembles RequestOutput , and returns it to the caller. That completes the path from an API request to generated text. Figure: logits indices selects the required hidden states, lm head produces logits, and sampling yields new token IDs. Finished requests release their KV blocks; unfinished requests remain active. lm head work for unused intermediate positions. Sampler produces token IDs according to each request's sampling parameters and returns them in ModelRunnerOutput . update from output appends new tokens, evaluates stopping conditions, and calls free request for completed requests. The six parts reduce vLLM's request path to a scheduler, a paged memory manager, and a GPU execution pipeline. Four design choices are especially reusable: First, the scheduler expresses both prefill and decode as per-request token progress. The execution layer then packs those scheduled tokens into one dimension. Chunked prefill and continuous batching build on the same contract instead of requiring separate batch formats. Second, V1 preempts by recomputing rather than swapping. Under KV-cache pressure, it frees the victim's blocks, resets num computed tokens , and requeues the request for prefill instead of moving KV state to CPU memory and back over PCIe. This trades additional computation for less host-device data movement. Third, forward context keeps step-specific attn metadata out of the forward signature so torch.compile can reuse a stable graph. A dummy tensor dependency then constrains operator ordering. The indirection is less obvious to read, but it serves compilation and execution correctness. Fourth, writes are addressed per token while reads are resolved per block. slot mapping selects each token's write slot; block table maps a request's logical KV blocks to physical blocks. At its core, the page-table lookup is physical = block table logical . The same discipline appears throughout the path: identify whether the scarce resource is compute, bandwidth, KV capacity, TTFT, or inter-token latency, then make the trade-off explicit. vLLM's throughput does not come from one isolated trick. It comes from small mechanisms that compose: a budget clamp expressed by min , one level of block table indirection, and a deliberate num computed tokens = 0 on preemption. To keep the path traceable, this article stops at the Python/CUDA boundary. It does not cover: kv cache manager , including reference counting and prefix-cache hash reuse. The article treats it only as a block allocator. This walkthrough targets the vLLM 0.22.0 V1 architecture. Source locations are included so readers can compare the excerpts with their installed version. The editable Excalidraw sources for Figures 2–8 are published with this site.