Latency Optimization Levers for Open-Weight LLM Inference: Part-1

Serving an open-weight large language model in a user-facing product is primarily a latency problem, and open weights enable engineers to control inference speed through levers such as weight quantization, speculative decoding, tensor parallelism, and hardware selection. The article, based on Qwen3-8B served on AWS SageMaker, explains that prefill is compute-bound and determines time to first token (TTFT), while decode determines inter-token latency, and notes that model quality is out of scope.

Serving a large language model in a user-facing product is a latency problem before it is anything else. A model that answers correctly but takes ten seconds to do so is unusable in a chat box, a coding assistant, or a voice agent. And unlike a training run, inference latency is paid on every single request, forever. So once a model is good enough for the task, the engineering effort moves to a different question: how do we make it respond faster, on the hardware we can actually get, without changing what it says? That last clause matters. There are two axes worth tracking when serving an LLM, and being precise about which one a change actually moves is most of the point: Every one of these levers is unlocked by serving an open-weight model. When you consume a model through a closed, hosted API, latency is whatever the provider gives you: you cannot change the numeric precision of the weights, attach a speculative draft head, split the model across GPUs of your choosing, or move it to faster hardware. With an open-weight model you own the entire serving stack, so each of these becomes a knob you can actually turn. That is the premise of this article: open weights are what make inference latency an engineering problem you can solve, rather than a number you are handed. The rest of this article walks through the levers that reduce LLM inference latency, the mechanism behind each one, how to turn it on, and what it actually bought when measured. The levers are weight quantization, speculative decoding, tensor parallelism, and hardware selection. Each is described in a general way, but the numbers come from a concrete setup: Qwen3–8B , an open-weight Apache-2.0 model from Alibaba, served on AWS SageMaker, described in the setup section below. None of these techniques are specific to Qwen3–8B; they apply to any open-weight model, and the mechanisms and the direction of each result carry over unchanged to larger models. An 8B model is used here mainly for practical reasons: GPU capacity for the much larger instances that frontier open models like Kimi K2 or GLM-4.6 would require was hard to obtain on AWS, so a model that fits comfortably on available hardware makes the levers easy to isolate and measure. The absolute latencies would differ on a bigger model, but which lever moves which axis would not. Two scoping notes. First, model quality is out of scope for this document. Several of these levers, quantization especially, can in principle change what the model outputs, and a production adoption decision must measure that. Here we hold the question aside and treat each configuration purely as a latency artifact; measuring quality impact is its own exercise see Inference Engineering , §5.1.3, for how one would do it . Second, this article assumes familiarity with the basics of transformer inference and GPUs. For a refresher on the underlying concepts, such as GPU architecture, number formats, and inference engines, Philip Kiely’s Inference Engineering Baseten Books is an excellent reference 551c and is cited throughout by section. The Medium article LLM Inference Handbook 2026 https://pub.towardsai.net/llm-inference-handbook-2026-135c266b86e7 is another good primer on these fundamentals, a shorter and freely accessible companion. To reason about which lever helps, it helps to understand where the time actually goes when a model generates a response. An LLM answers a request in two distinct phases, and they have completely different performance characteristics. Prefill is the first phase. The model reads the entire prompt, every token of it, in a single forward pass, and produces the first output token. Because the whole prompt is available at once, this pass is highly parallel: the GPU has a large matrix of work to do and can keep its compute units busy. Prefill is compute-bound . Its cost scales with prompt length, and it determines the time to first token TTFT . Decode is the second phase, and it is where responses are actually generated. The model produces one token at a time, and each token requires its own forward pass that depends on all the tokens before it. Generating a 256-token answer means 256 sequential forward passes. Each of those passes must read the model’s entire set of weights out of GPU memory to produce a single token’s worth of computation. This is the key asymmetry: the arithmetic done per pass is tiny, but the bytes moved are enormous, namely the whole weight matrix plus the growing KV cache. Decode is therefore memory-bandwidth-bound , not compute-bound. The GPU spends most of each step waiting on memory, not computing. The formal version of this is the ops:byte ratio or arithmetic intensity : a GPU can do some number of floating-point operations per byte it reads from memory, and if a kernel does fewer operations per byte than that ratio, it is limited by memory bandwidth rather than compute Inference Engineering , §2.4 . Single-stream decode sits far below the ratio, so it is squarely memory-bound. This single fact, that decode is memory-bandwidth-bound , is the mental model the rest of the article hangs on. It predicts, before running anything, which levers should work: The measurements bear this out. On the baseline used in this work, Qwen3–8B in FP16 on four NVIDIA L4 GPUs, growing the prompt from 72 to 1,608 tokens added only about 0.35 seconds prefill is cheap and compute-bound , while growing the output from 256 to 1,024 tokens scaled latency almost linearly, roughly 4x. Output length, not input length, drives latency, exactly as the decode-bound model predicts. Because latency is not a single number, a few terms are used throughout Inference Engineering , §1.4 : Every configuration in this article is reported against the same set of these metrics, measured the same way, so the levers can be compared directly. The model. Every configuration serves Qwen3–8B , an 8-billion-parameter open-weight model released by Alibaba under the Apache-2.0 license. Because the weights are open, the same model can be loaded in different numeric precisions FP16, FP8, AWQ INT4 , paired with a speculative draft head, and moved between GPUs without any change to what it is, which is exactly what lets the levers below be measured against a common reference. The FP16 baseline uses the SageMaker JumpStart artifact; the INT4 passes use the pre-quantized Qwen/Qwen3-8B-AWQ checkpoint; the speculative-decoding passes attach the trained RedHatAI/Qwen3-8B-speculator.eagle3 EAGLE head. The verifier weights are the same 8B model in all three. The hardware. Three SageMaker instance types are used, all Ada-generation sm 89 NVIDIA GPUs so the number formats and kernels are comparable: The serving engine. The model is served with vLLM , through SageMaker’s DJL-LMI container. vLLM is a widely used open-source inference engine for serving LLMs on GPUs, designed for high throughput and supporting the optimizations this article relies on, including quantization and speculative decoding; SageMaker’s DJL-LMI container packages it behind a managed endpoint Inference Engineering , §4.3.1 . The inference engine is held fixed across every configuration so that it is never a confounding variable. The only things that change between passes are the levers under test. vLLM uses continuous batching by default, and it is worth being precise about what that does, because it moves both of the axes from the introduction. Static batching waits for a fixed-size batch to fill before it runs inference, which leaves the earliest requests sitting in a queue. Continuous batching also called in-flight, or token-level, batching instead operates one decode step at a time. It swaps a new request into the running batch as soon as a slot frees up, so a request begins on the next step instead of waiting for a batch to be assembled. Cutting that queue time means continuous batching lowers latency and raises throughput relative to static or dynamic batching; it is a strict improvement over them, not a tradeoff Inference Engineering , §7.2.1 . The tradeoff lives one level down, in batch size . As concurrency rises, more sequences ride along in each decode step. Because that step is memory-bandwidth-bound and must read the full weight matrix regardless of how many sequences share it, packing more of them in raises aggregate throughput. But they now contend for the same bandwidth, so each individual request’s P50 and tail latency grow. This is the latency-versus-throughput tension from the introduction, made concrete: “increasing the batch size will produce more throughput overall, but each user’s latency will get worse” Inference Engineering , §7.2.1 . It is why every configuration below is reported twice, once as single-request latency at concurrency 1 and once as a concurrency sweep, so the single-stream effect of each lever is never conflated with this batching effect. The workload. The query set is 180 real, filtered ShareGPT https://huggingface.co/collections/bunnycore/sharegpt-datasets prompts: self-contained tasks that force the model to read the input and generate a response. Prompts are grouped into buckets by input length, with output length fixed per bucket so that prefill cost and decode cost can be read separately: How each pass is measured. Every configuration is deployed to its own SageMaker endpoint, benchmarked with a fixed harness, and torn down. Single-request numbers are taken at concurrency 1; the concurrency sweep drives the real medium bucket at 1, 2, 4, 8, and 16 concurrent requests. All latencies are end-to-end request time in seconds unless noted, and TM95 is the mean after dropping the slowest 5% of samples. Reported deltas between passes only change one thing at a time, whether precision, draft head, GPU count, or instance type, so each measured difference can be attributed to the single lever it isolates. The first lever follows straight from the mental model. Decode reads the entire weight matrix out of GPU memory once per token, so the number of bytes in that matrix sets a hard floor on how fast a token can be produced. Shrink the weights and every decode step moves fewer bytes, so on a bandwidth-bound path it finishes sooner. That is the whole mechanism, and it is why quantizing the weights is the most direct latency lever available. Qwen3–8B has roughly 8 billion parameters. Stored in FP16 that is about 16 GB of weights that decode must stream every token. Two quantization formats reduce that: Both are measured on the same 4× L4 box TP=4 as the FP16 baseline, so the only thing changing is the precision of the weights. The results track the byte counts closely: FP8 halves the weight bytes and cuts median latency by about a third from 4.88 to 3.23 s while lifting decode throughput from about 52 to about 80 tokens/sec, roughly the 1.5x the byte ratio predicts. INT4 halves the bytes again relative to FP8 and takes another large step: real medium P50 falls to 2.21 s 32% below FP8, 55% below the FP16 baseline at about 116 tokens/sec. The saving shows up on every bucket, and it is largest on the long-output real decode bucket where decode dominates the request, exactly where a bandwidth lever should help most. There is one counter-current worth naming. FP8 wins first-token latency , even though INT4 wins everything after it: TTFT is 66 ms for FP8 versus 90 ms for INT4. Prefill is compute-bound, not bandwidth-bound, and the AWQ dequant step converting the stored 4-bit weights back to a higher precision the matmul units can actually multiply in adds a little compute to that first parallel pass. FP8 avoids this because Ada GPUs have native FP8 tensor-core support, so the 8-bit weights feed the matmul directly with no conversion; there is no equivalent native 4-bit matmul, so INT4 always pays the dequant. So FP8 starts printing marginally sooner, while INT4 finishes the whole response sooner. For a streaming UI where the first token is the felt moment this is a real, if small, consideration; for total latency INT4 is the clear winner. The ranking also holds under load. At 16 concurrent requests the throughput ordering is FP16 at 1.74, FP8 at 2.42, and INT4 at 3.25 requests/sec: the same half-the-bytes advantage that lowers single-request latency also lets more sequences share each bandwidth-bound decode step, so INT4 sustains about a third more throughput than FP8 with a tighter tail. Smaller weights help both axes at once. This is the one clean case in this article where a lever is nearly free. One caveat carried from the introduction: these are latency and throughput numbers only. FP8 and especially INT4 change the stored weights, and whether they change what the model says is a separate measurement Inference Engineering , §5.1.3 that a production adoption must run before trusting the speedup. The weight-quantization lever makes each decode step cheaper. Speculative decoding attacks the other term in the mental model: the number of memory-bound steps. If decode is slow because it takes one full weight-reading forward pass to produce one token, then producing several tokens per pass is the other way to go faster. The trick is to let something cheap guess the next few tokens, and then have the real model check them. A draft proposes, say, four tokens; the 8B target runs a single forward pass that verifies all four at once and keeps the longest correct prefix. When the guesses are good, one expensive target pass yields several tokens instead of one, and the effective decode rate climbs above what the weight-bandwidth ceiling would otherwise allow. When the guesses are bad, the rejected tokens are thrown away and you have paid a little extra for nothing. So the whole lever lives or dies on acceptance rate , how often the guesses are right, and that is what separates the three approaches below Inference Engineering , §5.2 . N-gram speculation draft-free . The simplest scheme has no draft model at all: it proposes continuations by matching the recent context against text already seen so far in the same request, meaning the prompt plus the tokens generated up to this point prompt-lookup , which is cheap and works well when the output repeats text from the input or from earlier in its own response, as in summarization, code edits, or extraction. On the FP8 4× L4 baseline, adding n-gram OPTION SPECULATIVE CONFIG={"method":"ngram","num speculative tokens":5,...} leaves the short and medium buckets roughly flat, but on the long-output real decode bucket it cuts P50 from 12.69 to about 10.1 s a 20% drop and pushes the effective decode rate to about 115 tokens/sec, above the roughly 80 the FP8 weights alone can sustain, which is the tell-tale sign that multiple tokens are landing per pass. The catch is twofold. Acceptance is bimodal, so latency variance widens; and it hurts under load , because the verification compute competes with continuous batching once the GPU is busy, so past about two concurrent requests both latency and throughput fall below plain FP8. N-gram is a low-QPS, repetitive-output single-stream win, to be turned off under concurrency Inference Engineering , §5.2.4 . A separate draft model, and a trap. The more general scheme uses a small real model as the drafter: Qwen3–0.6B proposes, the 8B verifies. Because a real model can propose any continuation, not just text echoed from the prompt, it was expected to help the buckets where n-gram was flat. Instead it produced a 2.7× regression on the 4× L4 box: real medium P50 of 8.82 s versus FP8’s 3.23, and decode throughput of about 30 tokens/sec versus about 80, worse on every bucket. The cause is not speculative decoding itself but how this vLLM build shards it: it forces the draft’s tensor-parallel degree to equal the target’s, so the 0.6B draft was run at TP=4. A 0.6B model gains nothing from being split across four GPUs, but it now pays a tensor-parallel all-reduce on every speculative step, and acceptance is not high enough to amortize that. The fix, running the draft at TP=1, is only possible on this build by making the target TP=1 too a single L4 , and there the same draft-model configuration flips to a large win 33% to 46% below FP8 on the same box , confirming that the regression was entirely the forced draft parallelism. The full single-GPU numbers belong to the tensor-parallelism section. The lesson here is that a draft model’s overhead is real and can swamp its benefit if the serving stack shards it badly. A trained draft head EAGLE3 , the one that wins. The best result comes from EAGLE3: a small draft “head” trained against the target and fused into it, rather than a separate model bolted on. EAGLE works by reusing the target’s own internal state: a small trained layer takes the hidden-state features the target already computes and predicts the next few tokens from them autoregressively, which the target then verifies in a single pass. Because that head is trained on the target’s own representations rather than being an independent model, its guesses land far more often, and EAGLE3 raises acceptance further by drawing on features from several layers of the target rather than just the last Inference Engineering , §5.2.3 . Because it is fused, it is not subject to the draft-TP guard that wrecked the standalone draft, and it runs cleanly at TP=4 on the same 4× L4 box. Stacked on FP8 SPEC={"method":"eagle3","model":"RedHatAI/Qwen3-8B-speculator.eagle3","num speculative tokens":3} , it is the fastest single-request configuration in the study up to this point: The min latency on the 256-output buckets drops to about 1.35 s, the signature of very high acceptance: the trained head proposes tokens the target keeps. It also beats INT4 weights on the decode bucket 6.30 vs 8.45 s, 160 vs 116 tok/s despite INT4 having the smaller weights, because EAGLE3 is producing multiple tokens per pass on top of a full-precision-speed pass. And unlike n-gram and the standalone draft, it degrades gracefully under load : at 16 concurrent requests it holds about 4.55 s P50 at 2.77 requests/sec, competitive rather than collapsing past concurrency 2. The one place spec decode still cedes ground is raw throughput under load: INT4 sustains 3.25 requests/sec at concurrency 16 versus EAGLE3’s 2.77, because the per-request token savings erode as batching fills the GPU. That points at the obvious move: EAGLE3 is a decode-step-count lever and weight quantization is a bytes-per-step lever, so they are orthogonal and should compose. The stacking section returns to exactly that. Every result so far has lived on the 4× L4 box, served tensor-parallel across all four GPUs. Tensor parallelism TP is itself one of the levers the mental model predicts: splitting the model across more GPUs adds memory bandwidth, and decode is bandwidth-bound, so more GPUs should mean faster decode. This section holds the precision fixed and changes the GPU count instead, comparing the same weights on one L4 TP=1 against four TP=4 , to see what the split is actually worth Inference Engineering , §5.4.1 . TP works by cutting each layer’s weight matrices into shards, one per GPU. Each GPU holds a quarter of the weights and streams a quarter of the bytes per decode step, so four of them bring roughly four times the aggregate bandwidth to bear on a single request. The cost is that the shards have to be recombined: every layer ends with an all-reduce, a communication step across the GPUs that TP=1 never pays. So the split adds bandwidth but also adds a fixed per-step overhead that grows with GPU count and does not shrink when the weights get smaller. Holding precision fixed, going from one L4 to four is a large latency win, just not the clean 4× the bandwidth count suggests: The gain is real but sub-linear, eaten partly by the per-layer all-reduce and partly by the fact that a single L4 under sustained decode is so bandwidth-starved that its tail becomes fragile the 1× L4 FP8 decode bucket had a P99 of 91 s against a 39 s median, with a couple of requests stalling badly . Four GPUs both lower the median and stabilize the tail. The more interesting effect is on the other levers. Compare the INT4-over-FP8 win at each scale: Quantization helps more on a single GPU. The reason is exactly the all-reduce: on one L4 decode is purely bandwidth-bound and INT4’s half-the-bytes advantage shows in full, but on four GPUs each step also pays a fixed communication cost that quantization cannot reduce, so the relative benefit of moving fewer weight bytes is diluted. Tensor parallelism and weight quantization both target bandwidth, and they partly overlap, so stacking them does not simply add their percentages. Tensor parallelism also decides which speculative-decoding scheme is even viable, which is the payoff for the numbers deferred earlier. The standalone 0.6B draft regressed 2.7× on 4× L4 because this vLLM build forced the draft to TP=4, making it pay an all-reduce on every speculative step for a model far too small to benefit from the split. Drop the target to a single L4 so the draft can run at TP=1, and the same configuration flips to a win: FP8 with the draft at TP=1 comes in at real medium P50 6.66 s versus the single-L4 FP8 baseline’s 9.89, a 33% improvement. A trained EAGLE3 head does better still on the same single GPU at 5.62 s 43% below FP8 , because it is fused into the target and never incurs a separate draft all-reduce at any TP. The ranking on one L4 runs from FP8 at 9.89 s, to draft-TP1 at 6.66 s, to EAGLE3 at 5.62 s, roughly tied with INT4 at 5.70 s. That mirrors the 4× L4 ranking, but only once the draft is freed from the tensor-parallel penalty. Finally, TP=1 is not merely the slow option; it is a different point on a different axis. A single L4 is one quarter of the GPUs, and INT4 on that one card sustains about 45 tokens/sec, close to the FP16 four-GPU baseline’s roughly 52 on a quarter of the hardware. TP=4 is the lever you pull when single-request latency is what matters; TP=1 with INT4 is the lever you pull when cost per token is. The choice between them is latency versus dollars, a distinct trade from the latency-versus-throughput one that runs through the rest of this article. Every lever so far changed the software: the precision, the decoding scheme, the number of GPUs. The last lever changes the GPU itself, and it is the most direct of all. Decode is bandwidth-bound, so a GPU with more memory bandwidth speeds up decode without touching the weights, the config, or anything the model does. Nothing about the deployment changes except the card underneath it Inference Engineering , §3.2 . The L40S is, like the L4, an Ada-generation sm 89 GPU, so the same number formats and kernels run unchanged, but it has roughly twice the memory bandwidth. To isolate the hardware effect, the fastest configuration built so far AWQ INT4 weights plus an EAGLE3 head, the stack the next section covers is moved from the 4× L4 box to a 4× L40S box ml.g6e.12xlarge , TP=4 on both, with nothing else changed: The move cuts real medium P50 by 39% and the decode bucket by 41%, where throughput climbs from 207 to 361 tokens/sec a 74% gain . That is most of the way to the roughly 2× the bandwidth ratio would suggest, but not all of it: not every part of a decode step is a memory read the all-reduce and the arithmetic do not get faster , and the input-heavy real long bucket, which carries more compute-bound prefill, scales less 35% than the decode-dominated buckets. The lever helps exactly where the model says it should: the more decode-bound the workload, the closer to the full 2× it gets. All three 256-output buckets drop below one second, TTFT falls to 64 ms the lowest measured anywhere in the study , and throughput under load is the best of any configuration. Measured against the FP8 4× L4 starting point this article opened from 3.23 s, about 80 tok/s , the same-hardware software levers plus this hardware move land at 0.95 s, a 71% latency reduction with roughly 4.5× the decode throughput. Two practical caveats keep this honest. First, faster memory is not free: L40S instances cost more per hour than L4, so the latency win has to be weighed against the higher hourly rate, a cost comparison this latency-only study does not make but a production decision must. Second, the faster cards are scarcer: the g6e capacity simply was not available in the first region tried, and the endpoint had to be brought up elsewhere. “The hardware we can actually get,” from the introduction, is a real constraint: the fastest GPU only helps if you can allocate it. The levers have been presented one at a time so that each could be measured in isolation, but the reason they were worth measuring separately is that they compose. Look back at what each one actually targets in the decode step: These are three independent terms in the same cost. Shrinking one does not use up the others, so stacking them multiplies rather than merely adds, which is why the fastest configurations are combinations, not any single lever. The first stack combines the two best single-stream software levers on the same 4× L4 box: AWQ INT4 weights cheapest passes with an EAGLE3 head fewest passes . Spec decode is orthogonal to weight precision, since the EAGLE head simply verifies against a cheaper target, so they slot together, and the result is the fastest configuration on L4: That is 24% below EAGLE3 on FP8 the INT4 weights speed up every target and verify pass and 29% below INT4 alone EAGLE3 adds multiple tokens per pass on top . Crucially, it does this without the load penalty that sank n-gram and the standalone draft: at 16 concurrent requests it holds about 3.24 requests/sec, essentially INT4’s own throughput, so the spec-decode layer buys the single-stream win with no throughput regression. It moves both axes the right way at once. Adding the hardware lever from the previous section is the final multiplication. The complete progression, each row changing exactly one thing from the row above, tells the whole story of the article: From the FP16 baseline to the champion, that is an 81% latency reduction 4.88 s down to 0.95 s and roughly 7× the decode throughput about 52 to about 361 tokens/sec , built from four independent changes that each contributed and none of which cancelled the others. Serving an open-weight model gives you a stack you control, and every latency lever in this article is something that control unlocks: the precision of the weights, the decoding scheme, the number of GPUs, and the GPU itself. The single idea that made them predictable is that decode is memory-bandwidth-bound. Once that is the mental model, each lever is just a different way of moving fewer bytes, taking fewer passes, or reading bytes faster, and the measurements followed the prediction almost every time. They did not all move the same axis, and knowing which is the actual skill. Weight quantization and faster hardware lowered single-request latency and raised throughput, nearly free wins. Speculative decoding was a single-stream latency lever whose throughput benefit eroded under load, and in the n-gram and standalone-draft forms it turned actively harmful under concurrency; only a fused EAGLE3 head both won single-stream and held up batched. Tensor parallelism traded GPUs for latency and interacted with the other levers, diluting quantization’s edge and dictating which draft schemes were even viable. And KV-cache precision, had it been measured here, would have been a memory lever rather than a latency one. A lever is only useful once you know which number it moves. Two closing cautions. The first is the one held aside from the start: these are latency and throughput artifacts, not quality results. Quantization and speculative decoding can in principle change what the model outputs, and no configuration here should be adopted before its answers are measured against the FP16 baseline on a representative set Inference Engineering , §5.1.3 . The second is that the fastest configuration is only as available as its hardware: capacity, not just latency, decides what you can actually ship. But with an open-weight model and a bandwidth-bound mental model, inference latency stops being a fixed cost and becomes a set of knobs. Stacked, those knobs took Qwen3–8B from a 4.88-second baseline response to under a second. Latency Optimization Levers for Open-Weight LLM Inference: Part-1 https://pub.towardsai.net/latency-optimization-levers-for-open-weight-llm-inference-on-sagemaker-1cdcc1a01f62 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.