# Latency Optimization Levers for Open-Weight LLM inference:Part-2 > Source: > Published: 2026-09-13 12:31:01+00:00 In production, serving a large language model is a latency problem that goes beyond any single request. Once a model answers fast enough for one user, the questions that decide whether it survives real traffic are different. How many requests per second can a single endpoint absorb before latency falls apart? And where does the time actually go in a request, beyond the token-by-token decode loop: in the first token, in per-step overhead, in prefill work that could have been skipped? This article, the second in a series on latency optimization levers for open-weight LLM inference (part-1 [here](https://medium.com/towards-artificial-intelligence/latency-optimization-levers-for-open-weight-llm-inference-on-sagemaker-1cdcc1a01f62), which covered quantization, speculative-decoding and choosing hardware), works through four techniques that target exactly those questions. Each is measured on a concrete deployment and reported against the request rate or latency percentile it actually moves: The setup is held fixed across every configuration: Qwen3–8B (Apache-2.0) served on vLLM through SageMaker’s DJL-LMI container, benchmarked with a fixed ShareGPT-derived harness. Two caveats. Model quality is out of scope, so every number here is a latency/throughput artifact, not a claim about what the model outputs. And hardware availability shaped the work: the goal was to run these levers on a frontier open model (Kimi K2, ~1 T params), but H200 (p5e/p5en) capacity was unavailable in every region we tried, and even 4-GPU A10G/L40S capacity was intermittent. So the passes here run on the GPUs we could get, mostly g5 (A10G), with the load-testing baseline on g6e (L40S). Absolute numbers therefore reflect that hardware; the mechanisms and the direction of each result do not depend on it. Part 1 noted that prefill is compute-bound and decode is memory-bandwidth-bound: two phases with opposite hardware profiles. In standard (“colocated”) serving they nonetheless run on the same GPUs and contend for them. A big prefill for one request stalls the decode steps of everyone else in the batch, which is one reason TTFT (Time To First Token) and inter-token latency degrade together under load. **Disaggregation** separates them: prefill runs on one set of GPUs (or instances), decode on another, and the KV cache computed during prefill is handed across to the decode workers. The promises are (1) prefill spikes stop blocking decode, so TTFT and tail ITL (Inter-Token Latency) can be tuned independently, and (2) the two pools scale independently, so you add decode workers without touching prefill. **The transfer is the crux, and** **NixlConnector is what makes it practical.** For a decoder to continue a request that some other GPU prefilled, it needs that request's KV cache, the per-layer key/value tensors the prompt produced. Moving those between GPUs (or between hosts) fast enough that the hand-off doesn't erase the benefit is the hard part of disaggregation. vLLM abstracts it behind a pluggable **KV connector**, and we used the V1 **NixlConnector**, built on NVIDIA's **NIXL** (NVIDIA Inference Xfer Library). NIXL presents one transfer API over whatever fabric is available and picks the fastest path automatically: CUDA IPC over NVLink or PCIe within a node, and RDMA or plain TCP across nodes. Mechanically, the prefiller registers its computed KV blocks with NIXL as a kv_producer; the decoder, a kv_consumer, pulls exactly the blocks a given request needs; and a small proxy orchestrates the pair, calling prefill with max_tokens=1 to force the prompt's KV, then handing the request to decode, which pulls that KV over NIXL and streams the answer. Without a transfer layer this fast and this fabric-agnostic, disaggregation is impractical. NIXL is what turns "separate the phases" into something with an acceptable hand-off cost. **Why it can be a net latency win even though it adds a transfer hop.** Inserting a KV transfer between prefill and decode obviously adds work, so it is worth being explicit about why the total request need not get slower. There are two reasons. First, the KV that moves is small relative to the compute that produced it. For an 8B model a ~1.5k-token prompt is only a few megabytes of KV, which crosses even a plain TCP link in a handful of milliseconds, while the prefill that generated it took far longer. The hand-off is cheap next to the work it lets you relocate. Second, and more important, disaggregation removes *contention*. In colocated serving a long prefill occupies the GPU and stalls every in-flight decode, because they share it, spiking inter-token latency and the tail. Move prefill to its own GPUs and decode runs uninterrupted. The transfer cost is paid once per request, whereas the contention it eliminates was being paid on every decode step of every concurrent request. That asymmetry is why the felt latency, meaning TTFT stability and tail ITL under load, improves even though a hop was added. We measured two topologies, which also happen to bracket that transfer link from fastest to slowest. **Single-node (intra-node KV transfer).** Both engines in one container on one box, KV moving over NVLink or PCIe. It works, giving coherent generation and a warm single-stream TTFT of ~125 ms. But on a small box, dedicating GPUs to each phase *starves* each phase. A 1-prefill/1-decode split puts decode on a single A10G, and a 256-token generation there takes ~6.5 s, so the endpoint saturates at about 1 rps. Splitting the GPUs did not add decode capacity; it subtracted it. **Two-node (cross-node KV transfer, no RDMA).** Prefiller and decoder on separate g5.2xlarge instances in one subnet, KV moving over NIXL/TCP (g5 has no EFA/RDMA). This is the topology disaggregation is actually for. It works across instances, and the cross-node transfer overhead is small: sub-saturation TTFT P50 91 ms / P99 103 ms (server-side), ITL ~22 ms. But throughput still saturates at about 0.17 rps, the same wall as single-node, because decode still runs on one A10G. **The link scales with the hardware.** Our two nodes were g5.2xlarge, which have no EFA/RDMA, so cross-node KV moved over ordinary TCP, the slowest link disaggregation would ever run on, and even then the added TTFT was only tens of milliseconds. On the instances a frontier model actually needs (p5/p5en, with NVLink inside the node and EFA or GPUDirect RDMA between nodes) the same KV crosses one to two orders of magnitude faster, at tens to hundreds of GB/s, so the transfer term effectively disappears next to prefill and decode. The link was the weakest part of our setup and was still cheap; on the hardware where disaggregation is actually worth doing, meaning large models with a scaled decode fleet, the hand-off is nowhere near the bottleneck. What does not change with the fabric is the point below: the win is contention removal and independent scaling, not the transfer. The honest verdict, which matches vLLM’s own documentation (“disaggregated prefill does not improve throughput”): Disaggregation is a **TTFT/ITL-shaping and independent-scaling** lever, **not a throughput lever by itself**. On a fixed, single-decode-GPU setup it changes nothing about the throughput knee: that knee is set by the decode GPU regardless of where prefill runs. The throughput win only materializes when you use the independence to **scale the decode fleet** (one prefiller feeding many decoders, “xPyD”), which is inherently a multi-node deployment. **Automatic Prefix Caching (APC)** is a lever on the prefill and first-token path. Prefill’s cost is roughly linear in prompt length, because every prompt token’s KV has to be computed before the first output token appears. But many real requests share a long, identical **prefix**, such as a system prompt, a few-shot exemplar block, RAG boilerplate, or the accumulated history of a multi-turn chat, and differ only in a short suffix. Recomputing that shared prefix’s KV on every request is pure waste, and APC removes it. The mechanism is exact and automatic. vLLM already stores the KV cache in fixed-size **blocks** of tokens; APC hashes the token contents of each block, so when a new request’s leading blocks hash identically to blocks already in the cache, their KV is reused directly and prefill starts from the first token that actually differs. There is no prompt templating or manual cache key to manage, since identical leading tokens are detected automatically, and because prefill is what sets TTFT, a cache hit is a direct first-token speedup. It is a single serving flag: OPTION_ENABLE_PREFIX_CACHING=true. To isolate it cleanly on one APC-on endpoint (Qwen3–8B AWQ INT4, g5.2xlarge), we compared two query patterns against a ~1,455-token shared preamble: a shared-prefix set where every request reuses that preamble (cache hits after warmup), and a unique-prefix set where each request carries a distinct long preamble (always a miss, which is exactly what APC-off would do): **APC cut TTFT by about 37%** on hits (0.651 s down to 0.409 s) by not re-prefilling the shared preamble. It is orthogonal to and stackable with every Part 1 lever, since it touches the prefill path while decode ITL and tok/s are unchanged, and the win *grows* with prefix length and model size. Prefill cost is roughly linear in prefix tokens, so a longer system prompt or a bigger model on a heavily prefix-sharing workload benefits far more than this modest 8B, 1.5k-token example. It is close to free for any application with structured, repeated prompts. (Caveat: this measures single-stream TTFT; APC also raises prefill throughput and effective capacity under load, which we did not separately quantify.) Decode is memory-bandwidth-bound, but that is not the whole story at **low batch**. Each decode step launches hundreds to thousands of tiny GPU kernels, and at concurrency 1 the GPU’s per-kernel work is so small that the **CPU cost of launching each kernel** (Python dispatch, framework overhead, the CUDA launch API) becomes a real part of the step time, with the GPU idling between kernels waiting for the CPU. This is a compute-orchestration bottleneck that sits alongside the bandwidth one; the bandwidth model simply doesn’t account for it. A CUDA graph is a recorded graph of GPU operations, meaning the kernels of a computation and their dependencies, that can be replayed with a single launch call instead of the CPU re-issuing every kernel one by one. vLLM exploits this because a decode step runs the same kernel sequence every time. It captures that sequence once (for each of a set of batch sizes) during warmup, then replays the graph on every subsequent step, so per-step cost becomes a single launch instead of one CPU round-trip per kernel. It is on by default. Setting enforce_eager (LMI: OPTION_ENFORCE_EAGER=true) disables it, and several features, notably some speculative-decoding modes, force eager mode and give this up. Two otherwise-identical FP8 Qwen3–8B endpoints on g5.2xlarge, differing only in that flag, at concurrency 1: graphs cut ITL by about 22% and lifted decode throughput by about 28% at low batch, exactly the launch-overhead regime the roofline model does not cover. The same regime shows up in TTFT, which dropped from 130 ms to 102 ms (about 21%). TTFT is not pure prefill. It also includes sampling the first token, a decode step that replays the captured graph, and at these prompt lengths prefill is launch-bound rather than compute-bound, so it benefits from the same overhead removal. Expect the TTFT win to shrink for long prompts, where prefill is compute-bound. The whole effect shrinks as concurrency rises and GPU compute comes to dominate, so this is a single-stream, low-QPS win. This carries a useful implication for the speculative-decoding results. Speculative decoding required enforce_eager in that vLLM build, because it breaks the two things CUDA graphs require: static shapes and data-independent control flow. The number of draft tokens accepted per step varies with the acceptance test, which makes the next step’s shapes data-dependent, and rejection sampling reads results back to the CPU to decide how many tokens to keep, a data-dependent branch mid-step that a blindly-replayed graph cannot express. So every EAGLE3, n-gram, and draft-model pass ran with graphs off and was silently paying the roughly 22% decode penalty. Those spec-decode wins are real, but they were measured against a graphs-disabled floor. Whenever a feature forces eager mode, that trade is worth pricing. A newer vLLM that lets speculative decoding run with graphs (padding proposals to fixed shapes and handling variable acceptance inside the captured graph) would raise that floor. Part 1 reported a concurrency sweep of 1, 2, 4, 8, and 16 in-flight requests. That is a **closed-loop** measurement: a fixed number of clients each wait for their response before sending the next, so the offered load is self-limited by how fast the server answers. It can tell you latency at a concurrency, but it structurally cannot overload the endpoint, so it never reveals the request rate at which the server stops keeping up. Real traffic does not wait. An **open-loop** driver fixes the *arrival rate* instead. It fires requests on a Poisson schedule at a target RPS regardless of how many are still in flight, and sweeps that rate upward. That is what exposes the **saturation knee**: the offered QPS beyond which a queue builds without bound, achieved throughput falls behind offered, and TTFT and tail latency diverge. Run against the [Part 1](https://medium.com/towards-artificial-intelligence/latency-optimization-levers-for-open-weight-llm-inference-on-sagemaker-1cdcc1a01f62) champion (INT4 + EAGLE3 on 4× L40S), the knee is sharp, and importantly it is **not where a naive throughput check would place it**: Up to about 5 rps the endpoint is healthy: latency stable, TTFT ~140 ms, zero errors. Between 5 and 10 rps it falls off a cliff, with TTFT P50 jumping to 11.9 s while requests pile into a growing queue. The subtle part is that **achieved throughput keeps tracking offered rps up through 20**, so a throughput-only monitor would call the endpoint fine long after it became unusable. The real, latency-defined usable capacity is about 5 rps. That gap, throughput holding while latency collapses, is exactly what closed-loop sweeps cannot show, and it is the honest baseline against which any “serve more traffic” architecture (like disaggregation, abiove) has to be judged. Part 1’s discipline was to name the axis each lever actually moves. Extending that table to the advanced levers: These four techniques act on the parts of serving that a fast single decode step doesn’t touch, and the discipline for using them is the same throughout: know which number each one moves, and measure it under conditions that can actually expose the failure. Two threads run through the whole study. The first is axis discipline: prefix caching and disaggregation act on the prefill and TTFT path, CUDA graphs on decode overhead, and none of them is a free throughput lever. Throughput ultimately comes from adding decode hardware, which is precisely what disaggregation exists to let you do cleanly. The second is measurement discipline: each technique moves a specific number, and you only learn whether it helped by measuring that number under conditions that can expose the failure. Know which axis a lever acts on, turn it only when your workload actually loads that axis, and validate it against an open-loop curve rather than a throughput monitor that stays green well past the point of collapse. [Latency Optimization Levers for Open-Weight LLM inference:Part-2](https://pub.towardsai.net/latency-optimization-levers-for-open-weight-llm-inference-part-2-9926ea5eb0d2) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.