A Request's Life in vLLM V1

A technical analysis of vLLM 0.32.0 (commit 5df63a7) finds that the engine's per-token overhead outside the forward pass — msgpack decoding, detokenization, stop-string checks, JSON encoding and socket writes — is paid on a single asyncio event loop per API server process, and that one Python API server against a mock engine ran out of CPU at about 37,000 streamed tokens per second. Past that point the engine's batch stopped growing and the cost surfaced as time to first token, a number most people read as an engine problem. The analysis also documents that vLLM's documented process formula of A + DP + N (plus one when DP > 1) disagrees with the launch code in three places: `-tp 1` starts 2 processes rather than 3, `-dp 2` forces the `mp` backend even at TP=1 because `ParallelConfig` checks `world_size_across_dp > 1` first, and a supervisor process from `run_multi_api_server` is omitted from the docs entirely.

A Request's Life in vLLM V1 Part 2 of a ten-part series on serving engines from the inside. Pinned to vLLM 0.32.0 5df63a7 https://github.com/vllm-project/vllm/tree/5df63a7203cdb9328d824a27b682e77c7b9928be . Part 1 https://hiraditya.github.io/posts/anatomy-of-an-llm-serving-engine/ named the five layers and where vLLM and SGLang cut them into processes. This part follows one streamed chat request through vLLM’s cut. vLLM V1 keeps its busy loop away from HTTP. API server processes own the request from the socket to the last SSE event, and they do the tokenizing. One engine-core process per data-parallel rank runs the scheduler and owns the KV ledger. One worker process per GPU runs the model. The consequence is easy to miss: every per-token cost that isn’t the forward pass is paid in the API server, on a single asyncio event loop per process. That covers msgpack decoding, detokenization, stop-string checks, JSON encoding and the socket write. And nothing in the path pushes back on the engine when that loop falls behind. Against a mock engine, one Python API server ran out of CPU at about 37,000 streamed tokens per second. Past that point the engine’s batch stopped growing, and the cost landed in time to first token, a number most people read as an engine problem. The process tree, four ways Before tracing a request, count the processes it can cross. vLLM’s architecture overview gives a formula: A + DP + N processes, plus one when DP 1, where A is the API server count, DP the data-parallel size and N the GPU count.<sup 1 fn:arch </sup The code at the pinned commit disagrees with it in two places, and Part 1 covered a third. | Launch flags | Processes vLLM starts | Docs formula | Process titles | |---|---|---|---| | -tp 1 | 2 | 3 | vllm serve API server , VLLM::EngineCore worker inside | | -tp 2 | 4 | 4 | vllm serve , VLLM::EngineCore , VLLM::Worker TP0 , VLLM::Worker TP1 | | -tp 1 -dp 2 | 8 | 7 | vllm serve supervisor , VLLM::APIServer 0 and 1 , VLLM::DPCoordinator , VLLM::EngineCore DP0 and DP1 , VLLM::Worker DP0 and DP1 | | -tp 2 -dp 2 | 10 | 9 | supervisor, two API servers, coordinator, two engine cores, VLLM::Worker DP0 TP0 through VLLM::Worker DP1 TP1 | The titles come from set process title , which prefixes VLLM:: and appends the parallel ranks that are larger than one.<sup 2 fn:titles </sup The counts are what the launch code creates. A ps listing can also show helper processes that libraries start on their own, so match processes by these titles rather than by count. The first disagreement is the -tp 1 row, which Part 1 traced to the uni executor running the worker inside the engine core. The second is new: -dp 2 takes that arrangement away even at TP=1. ParallelConfig picks the mp backend whenever world size across dp 1 , and that check runs before the world size == 1 check that would pick uni .<sup 3 fn:backend </sup So the in-process worker exists only when the whole deployment has one GPU. The third is the supervisor. With internal load balancing, --api-server-count defaults to the data-parallel size, so -dp 2 means two API servers.<sup 4 fn:serve-default </sup Any count above one sends vllm serve down run multi api server . That process binds the HTTP socket and every API server’s ZMQ listeners, launches the engine cores and the DP coordinator, and spawns the API servers with the spawn start method. Then it waits. It serves no HTTP itself, and the docs formula leaves it out.<sup 5 fn:multi </sup The coordinator appears whenever DP 1, unless an external load balancer fronts a dense model. 6 fn:coord One streamed chat request Take the simplest real case: vllm serve at TP=2 with one API server, and a client that posts /v1/chat/completions with "stream": true . The request crosses the API server’s event loop, a renderer thread, three threads in the engine core and the GPU workers. sequenceDiagram autonumber participant C as Client participant A as API server event loop participant R as Renderer thread participant I as EngineCore input thread participant B as EngineCore busy loop participant W as GPU workers participant O as EngineCore output thread C- A: POST /v1/chat/completions, stream true A- R: chat template and tokenization R-- A: prompt token IDs and arrival time A- A: OutputProcessor registers request state A- I: EngineCoreRequest, msgpack over ZMQ I- I: decode, build Request, hash prompt blocks I- B: input queue loop every engine step B- W: SchedulerOutput through shared-memory MessageQueue W-- B: ModelRunnerOutput from the output rank B- O: EngineCoreOutputs keyed by client index O- A: msgpack over ZMQ PUSH A- A: decode, detokenize, check stop strings A- C: SSE data event end A- C: final DONE event Into the API server. uvicorn runs on uvloop , and the chat route lands in OpenAIServingChat.create chat completion . Rendering comes first. render chat async stamps arrival time , then runs the chat template and the tokenizer on the renderer’s thread pool so they don’t block the event loop.<sup 7 fn:render </sup That pool has one worker by default --renderer-num-workers , so every request’s template and tokenization in one API server goes through a single thread.<sup 8 fn:renderer-workers </sup The thread keeps that work off the event loop’s call stack. It does not take it off the GIL, because a Jinja chat template is Python. Across to the engine. AsyncLLM.generate calls add request to build the EngineCoreRequest and register the request with OutputProcessor . That happens before add request reaches its first await , so every other task on the loop sees the request at once. Then add request async encodes the request with msgspec and calls send multipart on the client’s ZMQ socket, with the engine’s identity as the first frame.<sup 9 fn:add </sup The API server’s job for this request is now to wait. Inside the engine core. Part 1 described the two I/O threads. The input thread does more than decode. It calls preprocess add request , which turns the wire struct into the scheduler’s Request and computes the prompt’s block hashes for prefix caching. The docstring says why: so “request initialization runs in parallel with Model forward”.<sup 10 fn:input </sup Aborts take a fast path. The input thread puts each abort on both the ordinary input queue and a separate aborts queue , and the busy loop drains aborts queue right after each forward pass, before it processes that pass’s output.<sup 11 fn:aborts </sup With async scheduling, which is on unless the configuration rules it out, the busy loop runs step with batch queue and keeps up to two batches in flight.<sup 12 fn:batchq </sup Part 3 takes that loop apart. Down to the GPUs and back. MultiprocExecutor.execute model enqueues the step on the shared-memory MessageQueue . Each worker’s worker busy loop dequeues it and runs it, and only the output rank answers: TP rank 0 of the last pipeline stage. 13 fn:exec update from output then returns a dict of EngineCoreOutputs keyed by client index . That key is the API server the request came from. The output thread encodes each entry into a reused buffer and sends it on that client’s PUSH socket. 14 fn:output-thread Back out through the API server. This half is where the per-token work lives. A task named EngineCoreOutputQueueTask reads the PULL socket, decodes the frames and puts the batch on outputs queue . A second task, the output handler, takes batches off that queue and turns engine outputs into RequestOutput s: 15 fn:handler 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Condensed from AsyncLLM. run output handler, vllm/v1/engine/async llm.py @ 5df63a7. Runs as one asyncio task on the API server's event loop. async def output handler : while True: 1. One EngineCoreOutputs per engine step, already msgpack-decoded by the socket task. outputs queue is an unbounded asyncio.Queue. outputs = await engine core.get output async 2. Process at most 128 requests' outputs at a time VLLM V1 OUTPUT PROC CHUNK SIZE . Each one gets an incremental detokenizer step and a stop-string check, and its RequestOutput is put into that request's collector. for start in range 0, len outputs.outputs , chunk size : processed = output processor.process outputs outputs.outputs start:start + chunk size , outputs.timestamp, ... 3. Yield the event loop between chunks so HTTP handlers and the socket task get a turn during a large batch. if start + chunk size < len outputs.outputs : await asyncio.sleep 0 4. A stop string matched in text. Only now does the engine hear about it, through an ABORT on the input socket. if processed.reqs to abort: await engine core.abort requests async processed.reqs to abort 5. Prometheus bookkeeping runs on the same loop, once per engine step. logger manager.record ... Each request’s generate coroutine waits on its collector and yields the RequestOutput to chat completion stream generator . That generator builds a ChatCompletionStreamResponse , calls model dump json , and yields data: {...}\n\n for uvicorn to write. After the last token comes data: DONE . 16 fn:sse Count what ran on the event loop for one token of one request: a share of a msgpack decode, a detokenizer step, a stop-string scan, a Pydantic object, a JSON dump and a socket write. None of that depends on the model. All of it scales with output tokens per second across every stream the process holds. Nothing pushes back Follow a token batch out of the engine and look for the place where a slow API server could make the engine wait. There isn’t one. - The ZMQ sockets. make zmq socket sets SNDHWM and RCVHWM to 0, which means no high-water mark. A PUSH socket never blocks its sender, so the engine’s output thread never stalls on a slow reader.<sup 17 fn:hwm </sup - outputs queue . An unbounded asyncio.Queue . The socket task can decode faster than the output handler processes, and the backlog grows in the API server’s memory.<sup 18 fn:outq </sup - The per-request collector. RequestOutputCollector holds one output. When a new one arrives before generate has taken the last one, it merges the two instead of queueing:<sup 19 fn:collector </sup 1 2 3 4 5 6 7 8 9 10 11 Condensed from RequestOutputCollector.put, vllm/v1/engine/output processor.py @ 5df63a7. def put self, output : if self.output is None: 1. The consumer is keeping up: park this output and wake generate . self.output = output self.ready.set else: 2. The consumer is behind: fold the new token delta into the one still waiting. Streaming clients get fewer, larger chunks, never a longer queue. self.output.add output, aggregate=self.aggregate I think this is the right design for a machine where the GPU costs more than the CPU feeding it. A frontend process that falls behind should never stall a batch on the GPU. But it turns frontend saturation into a quiet failure. The engine’s step rate stays the same and its own metrics look healthy. What changes is on the client side, and it takes one of two shapes. A backlog that spans several engine steps comes out as clumps of several tokens in one SSE event. A loop that is simply out of CPU keeps streaming to the requests it already has and makes new ones wait, so the engine’s batch stops growing. A latency dashboard that reports time per output token will miss both. The lag also feeds itself through stop strings. A stop string is matched in the output handler, so the ABORT leaves the API server only after every batch queued ahead of it has been processed. If outputs queue is k steps deep, the engine has already decoded k more steps for a request the client is finished with. Those tokens come back over ZMQ and add to the backlog. Stop tokens don’t have this problem, because the engine sees them as soon as it samples them. There is a cheap way to see the clumping from outside. Send "return token ids": true with a streaming chat request and each SSE event carries the token IDs in that delta.<sup 20 fn:token-ids </sup Without speculative decoding, the engine produces one token per request per step. So an event with more than one token ID means the collector merged outputs. Either the API server’s event loop was behind the engine at that moment, or that client’s connection was slow to drain and held up its own generate loop. Events from many connections at once point at the loop. With speculative decoding, several tokens per step is normal, so compare against the accepted-tokens-per-step rate instead. Two API servers, one socket --api-server-count is the knob for a frontend that can’t keep up. What it actually balances is worth reading before you turn it. The supervisor creates one listening socket, with SO REUSEPORT set, and passes the same socket object to every API server it spawns.<sup 5 fn:multi </sup Each server runs a NoSignalServer , a uvicorn subclass that replaces uvicorn’s accept loop when there are several workers. Every worker publishes its number of open connections into a shared array. When the socket becomes readable, a worker that isn’t the least loaded waits up to five times 1 ms for the count to change, then tries accept anyway, and moves on if another worker won. 21 fn:accept 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Condensed from NoSignalServer. accept, vllm/entrypoints/launchers/launcher.py @ 5df63a7. async def accept self, sock : while True: await readable loop, sock 1. Publish this worker's open-connection count, idle keep-alive connections included. self. publish load 2. Defer to a less loaded peer for up to ACCEPT DEFER CHECKS 5 ms. for in range ACCEPT DEFER CHECKS : if self. least loaded : break await asyncio.sleep 0.001 3. Race for the connection. A peer may already have taken it. try: conn, = sock.accept except BlockingIOError, InterruptedError, ConnectionAbortedError : continue 4. From here the connection, and every request on it, belongs to this API server. self. track self. connect loop, conn This is a trap for benchmarks and for proxies. The unit of balance is a TCP connection, decided once at accept time. Every request on a keep-alive connection goes to the API server that accepted it, however many tokens it streams. Open connections count as load even when they are idle. A load generator that reuses a small connection pool, or an L7 proxy with a few long-lived upstream connections, can leave one API server doing most of the per-token work while the other sits mostly idle. Check the per-process CPU, not the request count, before deciding that a second API server didn’t help. Each API server is also its own engine client. The supervisor gives each one its own pair of ZMQ listeners and a client index . Every EngineCoreRequest carries that index, and the engine’s output thread routes each request’s tokens back to the API server holding its HTTP connection.<sup 22 fn:multi-client </sup Prometheus switches to multiprocess mode so the vllm: metrics aggregate across servers. With data parallelism, every API server can send to every engine core. DPLBAsyncMPClient picks an engine for each new request by score. For each engine, the score takes the coordinator’s latest waiting + running count, or this client’s own in-flight count to that engine multiplied by the number of clients, whichever is larger. Waiting requests add a penalty that grows from zero at 50% KV-cache usage to three times the waiting count at 100%.<sup 23 fn:dplb </sup The coordinator publishes those counts no more often than once every 100 ms.<sup 24 fn:coord-interval </sup Nothing in that score knows which engine already holds a request’s prefix in its cache. Prefix-aware routing is the cluster layer’s job, and it is why llm-d subscribes to the KV events from Part 1. The Rust frontend changes this picture. With VLLM USE RUST FRONTEND , vllm serve ignores --api-server-count and starts one frontend process, because, in the code’s own comment, it is “multithreaded”.<sup 4 fn:serve-default </sup Part 1 described where that frontend sits. The per-token work is the same. The difference is that it is no longer pinned to one event loop per process. Where the frontend runs out The trace above gives a prediction that a GPU isn’t needed to test. The API server’s cost splits into two parts: | Paid once per request | Paid once per output token | |---|---| | HTTP parsing, chat template, tokenization on the renderer thread | Share of a msgpack decode on the socket task | | EngineCoreRequest encode and ZMQ send | Incremental detokenizer step and stop-string scan | | Collector, request state and generate coroutine setup | RequestOutput , chunk object, model dump json , SSE write | An engine’s output rate grows with batch size. The per-token column doesn’t shrink as the batch grows. So each API server process has an output rate in tokens per second above which its event loop can’t keep up. That ceiling depends on the CPU, not the GPU. My prediction was that tokens per SSE event would rise above one at the same point. vLLM’s repository already has most of what it takes to measure the ceiling. rust/src/mock-engine is “a small engine-side process for frontend stress testing”. It joins the frontend’s startup handshake, treats prefill as instant and returns random tokens until each request reaches max tokens , and its README documents running it behind the Python frontend.<sup 25 fn:mock </sup As shipped, it steps as fast as it can, which no GPU does. A 19-line patch adds a fixed step clock.<sup 26 fn:patch </sup At 10 ms per step, every running request gets exactly 100 tokens per second, and the engine’s output rate is its batch size times 100. vllm-bench , the Rust load generator in the same tree, drives the frontend in a closed loop. 27 fn:bench 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1. Python frontend only: --data-parallel-size-local 0 starts no engine and waits for one on the handshake port. No weights load, so the model directory needs only Qwen3-0.6B's config and tokenizer files. vllm serve ./Qwen3-0.6B --served-model-name qwen3 \ --data-parallel-address 127.0.0.1 --data-parallel-rpc-port 29550 \ --data-parallel-size 1 --data-parallel-size-local 0 \ --api-server-count 1 2. From vllm/rust with the patch applied: one engine step every 10 ms. cargo run --release -p vllm-mock-engine -- \ --handshake-address tcp://127.0.0.1:29550 --vocab-size 32000 \ --step-interval-us 10000 3. Closed loop at 512 clients, 256 prompt and 256 output tokens per streamed chat request. --request-rate only spreads out the first connections; the concurrency cap does the rest. cargo run --release -p vllm-bench -- --backend openai-chat \ --base-url http://127.0.0.1:8000 --model qwen3 --tokenizer ./Qwen3-0.6B \ --dataset-name random --random-input-len 256 --random-output-len 256 \ --ignore-eos --max-concurrency 512 --num-prompts 3072 --request-rate 400 The runs used one Apple M4 four performance cores, six efficiency cores, 32 GB with vLLM built for CPU from the pinned commit on Python 3.13. The frontend, the mock engine and the load generator shared the machine. Each point sends six times its concurrency in requests, and throughput is counted over the window where at least 95% of the target concurrency was in flight. The driver and the probe are published next to the patch. 28 fn:sweep Two properties of a laptop shaped the setup. macOS caps the listen backlog at 128 connections kern.ipc.somaxconn , so a closed loop that opens a thousand connections in the same instant is refused by the kernel before any frontend sees most of them. Arrivals are spread over about two seconds instead. And CPU time on Apple silicon isn’t a fixed unit of work, because the scheduler moves threads between core types and changes clocks with load. So the per-token costs below are quoted only where the API server’s thread was saturated. A first pass that ran while the machine idled toward sleep put the ceiling at less than half the figure below. Hold the machine awake with caffeinate before you trust a laptop benchmark. Here is one Python API server: | Clients | Client tokens/s | Engine batch | TTFT p50 | Mean TPOT | Tokens per SSE event | API server CPU | |---|---|---|---|---|---|---| | 128 | 12,784 | 128 | 11 ms | 10.00 ms | 1.0 | 0.53 cores | | 256 | 25,545 | 255 | 11 ms | 10.00 ms | 1.0 | 0.57 cores | | 384 | 36,502 | 365 | 165 ms | 10.01 ms | 1.0 | 0.90 cores | | 512 | 37,346 | 373 | 827 ms | 10.36 ms | 1.0 | 1.01 cores | | 1,024 | 34,746 | 347 | 4,431 ms | 11.78 ms | 1.0 | 1.03 cores | The engine batch is the client’s token rate divided by 100. Every token the engine makes reaches the client eventually, so in steady state the two rates match. At 1,024 clients, 19% of requests failed to connect, because the saturated loop accepted too slowly and the 128-entry backlog overflowed. Two more passes over the same points landed within 2% of these throughputs. One Python API server tops out at about 37,000 streamed tokens per second on this machine, or 27 µs of API server CPU time per token at saturation. An M4 performance core is fast per thread, so treat that as an optimistic figure for a server CPU. Below the ceiling the frontend is invisible: an 11 ms TTFT, a token every 10 ms, one token per event. The prediction held for throughput and failed for the symptom. Tokens per event never left 1.0, and the return token ids probe running alongside the load agreed. The saturated server kept streaming to the requests it already had at almost exactly the engine’s pace. New requests paid instead. TTFT went from 11 ms to 827 ms at 512 clients and 4.4 s at 1,024. The engine’s batch stopped growing at about 370, however many clients were waiting. vLLM’s own TTFT histogram saw almost all of that wait. In a pass that scraped /metrics , its mean at 512 clients was 658 ms against the client’s 678 ms. That metric runs from the arrival time stamp at the top of render chat async to the moment the output handler starts on the batch carrying the first token.<sup 29 fn:ttft-metric </sup So the wait happened after the request was accepted and parsed and before its first token was processed. The first place I’d look is the single renderer thread, which needs the GIL that the busy event loop holds. I haven’t instrumented it further. This is the failure I’d worry about on a GPU. The engine isn’t stalled. It’s starved. A batch capped at 370 by the frontend is throughput the GPU could have had, and the evidence lands in TTFT, the metric everyone reads as prefill and scheduling. The engine’s waiting queue won’t show it, because the waiting requests haven’t reached the engine. The ceiling is a token rate, so the batch it allows scales with step time: about 370 running requests at 10 ms per step, about 1,100 at 30 ms. Small models with fast steps hit it first. Clumps need something else: a backlog that spans several small steps. With the step clock off, the mock engine finishes requests almost as fast as it admits them. Every probe request then got its whole response as one SSE event carrying all 256 token IDs, with a median TTFT of 15 ms at 64 clients. The cause is in asyncio. Queue.get returns at once when the queue has items, without yielding to the event loop.<sup 30 fn:queue-get </sup The output handler yields only between 128-request chunks of one step and when the queue runs empty. So when the queued steps are small, the handler drains them back to back, and each request’s collector merges them before its generate gets a turn. A real engine produces the same shape whenever the API server’s loop stalls for several steps while batches are small. More frontend fixes the ceiling. Here are the three frontends at 1,024 clients: | Frontend | Client tokens/s | Engine batch | TTFT p50 | Failed to connect | Frontend CPU | |---|---|---|---|---|---| | One Python API server | 34,746 | 347 | 4,431 ms | 19% | 1.03 cores | | Two Python API servers | 62,960 | 630 | 1,308 ms | 6% | 1.91 cores | | Rust frontend | 102,211 | 1,022 | 10 ms | 0% | 1.53 cores | Two API servers kept up through 512 clients, at 51,000 tokens per second with an 18 ms median TTFT, and reached 63,000 at 1,024. That’s 1.7 times one server’s ceiling, on a machine with four performance cores that was also running the load generator and the mock engine. vllm-bench holds one connection per in-flight request, which is the easy case for connection-level balancing. From 32 clients up, the two servers’ CPU time stayed within 1.5% of each other. The Rust frontend never reached a ceiling. It held a 10 ms median TTFT and one token per event up to 2,048 clients and 204,000 tokens per second. It used 2.4 cores spread across its threads, while the load generator used 1.6. That’s about 12 µs of CPU per token, against 27 µs for the Python server. The sweep stopped at 2,048 clients, so its ceiling on this machine is somewhere above that. So count API servers the way you count GPUs, from a ceiling measured on your own CPU with the mock engine, and watch TTFT rather than inter-token latency for the symptom. Or move to the Rust frontend, if your flags allow it. At the pinned commit it refuses 57 vllm serve flags, among them --tokenizer , --middleware , --otlp-traces-endpoint and the external and hybrid DP load-balancing modes. It accepts and ignores nine more. 31 fn:rs-unsupported References Disclaimer: Researched and drafted with AI assistance Claude Opus 5.5, Gemini 4 Argon . Direction, technical judgment, and final edits are mine; every claim is traceable to the sources cited above. 1. vLLM docs, “Architecture Overview,” process count summary. The A + DP + N formula and its worked examples. arch overview.md L103 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/docs/design/arch overview.md L103 ↩︎ fnref:arch 2. vLLM process titles. set process title and its callers for API servers, engine cores and workers. system utils.py L180 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/system utils.py L180 , v1/utils.py L524 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/utils.py L524 , core.py L1408 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1408 , multiproc executor.py L1088 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/executor/multiproc executor.py L1088 ↩︎ fnref:titles 3. vLLM executor backend selection, ParallelConfig. post init . The mp default when world size across dp 1 , and the later uni default when world size == 1 . parallel.py L1035 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L1035 , parallel.py L1079 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py L1079 ↩︎ fnref:backend 4. vLLM ServeSubcommand.cmd . The --api-server-count defaults per load-balancing mode, the Rust frontend override, and the dispatch to run server , run multi api server or run headless . serve.py L107 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/cli/serve.py L107 ↩︎ fnref:serve-default ↩︎<sup 2</sup fnref:serve-default:1 5. vLLM run multi api server and APIServerProcessManager . The supervisor path, the shared listening socket with reuse port , and the spawned API server processes. serve.py L269 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/cli/serve.py L269 , v1/utils.py L167 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/utils.py L167 ↩︎ fnref:multi ↩︎<sup 2</sup fnref:multi:1 6. vLLM DP coordinator launch. VllmConfig.needs dp coordinator and the run coordinator condition in launch core engines . vllm.py L843 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/vllm.py L843 , engine/utils.py L1227 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/utils.py L1227 ↩︎ fnref:coord 7. vLLM chat rendering. render chat async stamps arrival time and offloads template rendering and tokenization to the renderer’s executor. renderers/base.py L1327 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/renderers/base.py L1327 , renderers/base.py L142 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/renderers/base.py L142 ↩︎ fnref:render 8. vLLM ModelConfig.renderer num workers . Defaults to 1: “Number of worker threads in the renderer thread pool.” model.py L408 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/model.py L408 ↩︎ fnref:renderer-workers 9. vLLM AsyncLLM.generate and add request , and the client’s ZMQ send. Registration with OutputProcessor before the first await , then send multipart with the engine identity frame. async llm.py L698 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/async llm.py L698 , async llm.py L572 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/async llm.py L572 , core client.py L1250 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core client.py L1250 ↩︎ fnref:add 10. vLLM engine-core input thread. process input sockets decodes each request and calls preprocess add request , which builds the Request with the block hasher. core.py L1831 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1831 , core.py L1089 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1089 ↩︎ fnref:input 11. vLLM abort fast path. “Aborts are added to both queues, allows us to eagerly process aborts while also ensuring ordering in the input queue.” core.py L1928 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1928 ↩︎ fnref:aborts 12. vLLM batch queue and async scheduling. step with batch queue , max concurrent batches returning 2 under async scheduling, and the default that turns async scheduling on unless a listed feature rules it out. core.py L698 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L698 , vllm.py L577 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/vllm.py L577 , vllm.py L1585 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/vllm.py L1585 ↩︎ fnref:batchq 13. vLLM MultiprocExecutor . worker busy loop and get output rank “the first TP worker of the last PP stage” . multiproc executor.py L1037 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/executor/multiproc executor.py L1037 , multiproc executor.py L548 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/executor/multiproc executor.py L548 ↩︎ fnref:exec 14. vLLM engine-core output thread. process output sockets encodes each client index, outputs pair and sends it on that client’s socket. core.py L1933 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1933 ↩︎ fnref:output-thread 15. vLLM AsyncLLM. run output handler and the output chunk size. async llm.py L825 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/async llm.py L825 , envs.py L1495 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/envs.py L1495 ↩︎ fnref:handler 16. vLLM chat completion stream generator . Chunk construction, model dump json , and the final data: DONE . serving.py L466 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/chat completion/serving.py L466 , serving.py L933 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/chat completion/serving.py L933 ↩︎ fnref:sse 17. vLLM make zmq socket . RCVHWM and SNDHWM set to 0 on every receiving and sending socket type. network utils.py L372 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/network utils.py L372 ↩︎ fnref:hwm 18. vLLM AsyncMPClient output path. The unbounded outputs queue and the process outputs socket task. core client.py L1131 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core client.py L1131 , core client.py L1171 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core client.py L1171 ↩︎ fnref:outq 19. vLLM RequestOutputCollector . “When streaming deltas, RequestOutputs are merged if the producer gets ahead of the consumer.” output processor.py L53 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/output processor.py L53 ↩︎ fnref:collector 20. vLLM chat request return token ids . “In streaming mode … token ids contains the delta tokens for each chunk.” protocol.py L423 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/chat completion/protocol.py L423 ↩︎ fnref:token-ids 21. vLLM NoSignalServer . Least-loaded accept across API servers on one shared socket. launcher.py L45 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/launchers/launcher.py L45 , launcher.py L101 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/launchers/launcher.py L101 , v1/utils.py L218 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/utils.py L218 ↩︎ fnref:accept 22. vLLM per-client routing. client index in each API server’s client config, and the engine’s per-client output sockets. v1/utils.py L167 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/utils.py L167 , core.py L1933 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py L1933 ↩︎ fnref:multi-client 23. vLLM DPLBAsyncMPClient.get core engine for request . The load score, the KV-pressure penalty, and the local count bump between coordinator updates. core client.py L1642 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core client.py L1642 ↩︎ fnref:dplb 24. vLLM DPCoordinator . min stats update interval ms defaults to 100. coordinator.py L154 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/coordinator.py L154 ↩︎ fnref:coord-interval 25. vLLM mock engine, rust/src/mock-engine/README.md . Purpose, flags, and the Python-frontend setup with --data-parallel-size-local 0 . README https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/rust/src/mock-engine/README.md ↩︎ fnref:mock 26. Step-clock patch for the mock engine. Adds --step-interval-us , a tokio interval with MissedTickBehavior::Delay in place of the yield now between steps. Applies to vLLM at 5df63a7 . mock-engine-step-interval.patch https://github.com/hiraditya/hiraditya.github.io/blob/main/assets/code/mock-engine-step-interval.patch ↩︎ fnref:patch 27. vllm-bench , rust/src/bench/README.md . Closed-loop and rate-controlled load, per-request ITL arrays with --save-detailed , and the steady-state window. README https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/rust/src/bench/README.md ↩︎ fnref:bench 28. Sweep driver and SSE probe for this post. The driver starts a frontend and the mock engine, runs one vllm-bench point per concurrency, and records process CPU, vLLM’s TTFT histogram and plateau throughput. The probe counts token ids per SSE event. frontend ceiling sweep.py https://github.com/hiraditya/hiraditya.github.io/blob/main/assets/code/frontend ceiling sweep.py , sse token probe.py https://github.com/hiraditya/hiraditya.github.io/blob/main/assets/code/sse token probe.py ↩︎ fnref:sweep 29. vLLM TTFT metric. first token latency is iteration timestamp - arrival time . arrival time is stamped at the top of render chat async , and iteration timestamp when the output handler creates the batch’s IterationStats . renderers/base.py L1336 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/renderers/base.py L1336 , stats.py L478 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/metrics/stats.py L478 , stats.py L521 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/metrics/stats.py L521 ↩︎ fnref:ttft-metric 30. CPython asyncio.Queue.get . It waits only while self.empty , and otherwise returns get nowait without suspending. queues.py L172 https://github.com/python/cpython/blob/v3.13.7/Lib/asyncio/queues.py L172 ↩︎ fnref:queue-get 31. vLLM Rust frontend UnsupportedArgs . 57 Unsupported fields that stop vllm-rs serve with an error, and nine Noop fields it accepts with a warning, api server count among them. unsupported.rs L91 https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/rust/src/cmd/src/cli/unsupported.rs L91 ↩︎ fnref:rs-unsupported CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ by the author.