A language model endpoint hides a resource allocator. Every accepted request asks for a share of the model weights, temporary activation space, a KV cache that grows one token at a time, and repeated access to accelerator compute and memory bandwidth. The inference engine decides when each request enters, where its state resides, which other requests share an iteration, and whether a new prompt may delay tokens already streaming.
This drawing follows one request and then adds contention. Prefill processes the supplied prompt in parallel. Decode produces one new token per sequence per iteration. Their shapes differ, so a schedule that raises total tokens per second can also make the first token or the gaps between later tokens worse. Memory is equally conditional: weights are mostly fixed, activations peak around kernel execution, and KV state scales with live sequence tokens.
The citable object is a worked memory-and-time ledger. Its 7B teaching case uses explicit binary units and keeps every category separate. The scheduler panel uses fixed synthetic arrivals and labeled assumptions; it is an algorithm demonstration, not a hardware benchmark. Product shootouts and vendor benchmark tables stay outside the permanent sheets because model revision, traffic shape, kernels, hardware, and latency targets change the answer.
Serving begins where one forward pass ends #
An offline model call can be described as tensor operations. A service must also admit, queue, cancel, stream, meter, isolate, and recover requests. It receives text or token IDs, applies the chat template and tokenizer, validates length, chooses a replica, waits for capacity, allocates KV pages, runs prefill, samples the first token, and repeats decode until an end condition. Only then can it release the request’s cache blocks.
Each boundary carries state. The router knows tenant and deadline. The scheduler knows arrivals, token budgets, and resident sequences. The model runner owns device buffers and kernels. The sampler owns decoding parameters and random state. The streamer owns backpressure and cancellation. A request can fail before any matrix multiplication, finish while its batch peers continue, or lose its client while device work remains queued.
The service clock starts before the GPU clock. Authentication, request parsing, tokenizer work, replica selection, and queue delay can dominate a short prompt. Streaming introduces another clock at the network boundary: a token may be computed but unavailable to the caller while serialization or a slow connection backs up. Measure timestamps at acceptance, scheduler admission, prefill start/end, each token-ready event, each token-write event, and release. Those points separate engine delay from transport delay without pretending that either is invisible to the user.
This makes serving a separate discipline from model architecture. The Transformer drawing’s KV-cache section explains why earlier keys and values can be reused. The inference engine turns that reuse into an allocation and scheduling problem across many unrelated sequences. The site’s interactive demos index supplies adjacent tokenization and retrieval experiments; the scheduler below stays local so its assumptions remain beside the ledger.
The weight file may fit while the complete runtime does not. Kernel workspaces, graph captures, allocator reserve, activations, and the request’s growing KV cache require additional bytes. “Weights fit” answers one ledger line, not the resident-memory total.
Prefill and decode ask the accelerator for different work #
Prefill consumes all prompt tokens. Within each layer, many prompt positions can be projected and processed in parallel. Attention still respects the causal mask, but the device receives large matrix operations with substantial arithmetic per launch. Prompt length drives time-to-first-token because decode cannot emit the first sampled token until the prompt’s final position has passed through the model.
Decode begins after prefill. Each active sequence contributes one current token to an iteration, reads its previous KV entries, produces logits, samples one token, and appends one K vector and one V vector per layer. The serial dependency is absolute: token t + 1 needs token t. Batching adds sequences to the iteration, yet no scheduler can parallelize future positions of one autoregressive sequence.
TTFT [ms/request] = queue delay [ms/request] + prefill service [ms/request] + first-step overhead [ms/request]
ITLi [ms/token] = available time(tokeni+1) [ms] − available time(tokeni) [ms]
The distinction predicts interference. A long prefill admitted whole may occupy the runner long enough to widen decode gaps for sequences already streaming. Chunked prefill divides that prompt into bounded token work so decode iterations can run between chunks. SARATHI [1] formalized this piggybacking approach; DistServe
later treated prefill and decode as separable phases that can be provisioned around different latency objectives.
[2]Token throughput needs two counters because input and output tokens represent different shapes. A prefill-heavy service may report a large total-token rate while generating few streamed tokens. A decode-heavy service may spend most of its time rereading weights and growing KV for small one-token batches. Report input tokens/s and output tokens/s separately, alongside request lengths. End-to-end latency also scales with requested output: approximately TTFT plus the sum of later ITLs, so comparing requests with different completion lengths without bucketing them mixes workload with system behavior.
Weights, activations, and KV state occupy different ledgers #
Weight memory is primarily fixed after a replica loads. Let P be parameters and b w stored bytes per parameter. Quantized formats add scales, zero points, packing, and sometimes a higher-precision subset, so metadata belongs on the same line rather than disappearing into a slogan.
Mweights [bytes] = P [parameters] × bw [bytes/parameter] + Mmetadata [bytes] Activation and runtime memory varies with kernels, batch token count, graph captures, temporary outputs, communication buffers, and allocator policy. It should be measured at the intended maximum batch shape and carried as M runtime,peak. Training-style “activations per token” estimates often mislead here because inference kernels may fuse operations and reuse storage.
KV state scales directly. With L layers, H kv KV heads per layer, head dimension d h elements per head, S sequence tokens per request, and b kv bytes per cached element, every token stores one key and one value at every layer.
MKV,request [bytes/request] = 2 [K,V] × L [layers] × Hkv [heads/layer] × dh [elements/head] × S [tokens/request] × bkv [bytes/element] The worked case assumes 7,000,000,000 parameters in BF16, no added weight metadata, 32 layers, 8 KV heads, head dimension 128, BF16 KV, 4,096 tokens per resident request, and 16 resident requests. Weights consume 14,000,000,000 bytes = 13.04 GiB. KV consumes 131,072 bytes/token = 128 KiB/token, then 536,870,912 bytes/request = 0.50 GiB/request, then 8.00 GiB for 16 requests. Add a separately budgeted 3.00 GiB runtime peak and a 4.00 GiB safety reserve on a 40 GiB device. The committed-plus-reserved total is 28.04 GiB.
Sequence length in that multiplication is each request’s current prompt plus generated tokens, not the configured maximum unless memory is actually reserved to that maximum. For a mixed batch, replace N × S with the sum of live sequence lengths: Σ r S r. Paging rounds each request to block boundaries, so physical allocation uses Σ r ceil(S r / B) × B tokens for block size B tokens/block. Prefix sharing can make physical occupancy smaller than the naïve sum when multiple requests reference the same immutable blocks; copy-on-write restores separate ownership when a branch diverges.
The safety line is deliberate capacity, not unidentified waste. It absorbs measurement error, allocator behavior, and transient shapes. It should be chosen from observed peaks and failure tolerance, then named in the workbook. If a runtime reserves memory without committing it, record both allocator-reserved and tensor-allocated bytes. Device telemetry alone cannot say which bytes are reclaimable or which request owns them.
Grouped-query attention lowers H kv below the query-head count, which is why the cache formula needs KV heads rather than total attention heads. The GQA paper [3] describes converting multi-head checkpoints to fewer KV heads. The model configuration, not its marketing size, supplies the ledger inputs.
The memory-only ceiling is floor((40 − 13.04 − 3.00 − 4.00) GiB ÷ 0.50 GiB/request) = 39 requests. That does not promise acceptable latency or 39 simultaneous compute slots. It says the declared memory categories leave room for 39 equal-size KV allocations under these assumptions.
Paging turns variable sequences into block tables #
A naïve allocator reserves one contiguous region for a request’s maximum sequence. If the output ends early, the unused tail becomes internal waste. If many differently sized regions are allocated and freed, total free memory can be large while no single contiguous hole satisfies the next request. Copying a cache to grow it adds traffic precisely when the engine is busy.
PagedAttention [4] applies an operating-system idea to KV storage. A request’s logical cache is divided into fixed-size token blocks. Its block table maps logical block numbers to non-contiguous physical blocks. The next token fills the current block or allocates another; completion returns those blocks to a shared pool. Only the last block can contain ordinary rounding waste. The vLLM paper also describes copy-on-write sharing for common prompt state and parallel sampling.
Block size creates a real trade. Smaller blocks reduce last-block waste and make short growth increments cheaper, but increase block-table entries and allocation operations. Larger blocks simplify metadata and can suit kernels, but amplify rounding loss across many short sequences. Prefix caching adds a second policy question: reusable blocks save prefill work only when hit rate and reuse lifetime repay the bytes and lookup cost.
Consider three requests with live lengths 17, 33, and 65 tokens under 16-token blocks. Their logical occupancy is 115 tokens. Physical occupancy is 2 + 3 + 5 = 10 blocks, or 160 token slots, leaving 45 slots unused across their last blocks. With four-token blocks, physical occupancy falls to 124 slots, while block-table entries rise from 10 to 31. This arithmetic describes rounding only; kernel layout, allocator metadata, and shared prefixes add separate terms.
Paging improves allocation efficiency; it does not reduce the logical KV bytes required by the formula. Under pressure, an engine may reject admission, preempt a sequence, swap state, or free it and recompute its prefix later. Each response has a latency cost. “No OOM” can therefore coexist with cache thrash and terrible tail latency.
Iteration-level scheduling connects utilization to latency #
Static batching fixes a group of requests for a larger span of work. Padding makes short prompts pay for a long prompt’s shape, and a short generation can leave its slot idle until longer peers finish. New arrivals wait for the batch boundary. This policy is simple and can work for homogeneous offline jobs, but interactive traffic rarely arrives or ends in lockstep. Continuous batching revisits membership at iteration boundaries. Finished sequences leave; admitted sequences join when token and memory budgets permit. Orca [5] called the underlying move
iteration-level scheduling and paired it with selective batching. Paged KV allocation makes changing membership practical because each request carries a block table rather than a fixed rectangular cache slab.
The scheduler still chooses whose latency to spend. First-come, first-served is predictable until long prompts block the line. Prioritizing prefills can lower new-request TTFT while stretching ITL for active decodes. Prioritizing decodes preserves streaming cadence while queues age. Chunking prefills caps interference. Tenant weights, deadlines, maximum batched tokens, preemption cost, and replica routing all change the trace.
The panel’s metrics use request acceptance as time zero. Increase prompt length to add prefill work, lower the KV budget to force admission pressure, or compare policies with the same fixed arrivals. Its synthetic tick costs are intentionally visible. A production comparison needs real prompt/output distributions, a declared SLO, warm-up, model and tokenizer revisions, kernel versions, accelerator topology, and separate input/output token accounting.
A small class of long prompts, burst arrivals, allocator retries, preempted sequences, or replica imbalance can occupy less than five percent of samples yet dominate the slowest one percent. Record the joint prompt/output distribution and correlate tail requests with queue age, prefill chunks, evictions, and replica assignment.
-
▸Weight, runtime, and KV formulas with the worked 7B ledger
-
▸Concurrency envelope and quantization decision tables
-
▸TTFT, ITL, percentile, and benchmark-input record
-
▸One-page OOM, queue-growth, and cache-fragmentation checklist
IO-aware attention and quantization attack different costs #
Standard attention materializes or repeatedly moves intermediates between high-bandwidth memory and faster on-chip memory. FlashAttention [6] tiles queries, keys, and values, computes attention blocks on chip, and maintains the softmax statistics needed to combine tiles exactly. The result changes memory traffic and temporary storage without approximating dense attention. IO-awareness matters because fewer floating-point operations do not guarantee less wall-clock time when data movement is the limiting resource.
Quantization changes representation. Lower-bit weights reduce stored bytes and memory traffic; lower-bit activations can enable different matrix kernels; lower-bit KV reduces the per-token cache line. These are three decisions with different calibration, kernel, and quality consequences. A nominal four-bit weight file may still dequantize into higher precision for unsupported operations, carry scales and outliers, or use kernels that underperform at small batches.
SmoothQuant [7] targets activation outliers that make W8A8 post-training quantization difficult. For a channel-wise scale
s, the transformation divides activation channel
X by
s and multiplies the matching weight channel by
s. Their matrix product is mathematically unchanged before quantization; the difficulty moves from activations toward weights, which are easier to calibrate under the method’s assumptions. The quantized computation can still introduce error.
(X diag(s)−1)(diag(** s**) W) = XW [same real-valued product before rounding] Quantization stops being “free” when quality loss crosses the task boundary, conversion or calibration becomes operationally fragile, the target hardware lacks the intended kernels, metadata erases expected memory savings, or latency moves the wrong way at the deployed batch shape. Measure perplexity only if it predicts the application. Otherwise use the task evaluation, prompt languages, structured-output validity, and tail cases the endpoint must preserve.
Keep memory claims aligned with the chosen target. Weight-only quantization can cut the fixed replica line while leaving BF16 KV unchanged, so long-context concurrency barely moves after weights cease to dominate. KV quantization directly changes bytes/token but adds its own accuracy and kernel questions. Activation quantization primarily targets compatible matrix execution and transient storage. A deployment can combine all three, yet the capacity workbook should retain three rows so a regression can be traced to the changed representation.
Capacity is an envelope bounded by memory, compute, and latency #
A concurrency number without a workload is incomplete. Memory capacity depends on resident tokens. Compute capacity depends on input and output token rates, batch shapes, kernels, and parallelism. Service capacity adds latency objectives: requests count as useful only if TTFT, ITL, or end-to-end deadlines are met. The relevant quantity is often goodput, completed work inside the stated SLO.
Nmemory [requests] = floor((Mdevice − Mweights − Mruntime,peak − Msafety) [bytes] / MKV,request [bytes/request]) Build the envelope by sweeping offered load and length buckets, not by quoting one saturation point. Hold model and system revision fixed. Record p50, p95, and p99 queue delay, TTFT, ITL, end-to-end latency, input tokens per second, output tokens per second, cache occupancy, and preemptions. Repeat for burst shapes that resemble production. Stop increasing load when the declared SLO fails, even if aggregate throughput continues upward.
Use an open-loop load generator when the question is overload behavior. A closed-loop client that waits for one response before sending the next automatically reduces arrivals as latency rises, concealing queue growth. Record offered rate and achieved completion rate separately. Warm-up should cover model load, graph compilation or capture, allocator stabilization, and cache state, while the measured window should still include the cold behavior users actually face when scale-to-zero or replica churn is part of production. Replicas add another queueing layer. More replicas can reduce queue delay and isolate failures, but each replica duplicates weights unless the topology shards a model. Tensor or pipeline parallelism changes communication and per-device memory. A router that sends traffic by request count can overload one replica with long prompts while another handles short completions. Token-aware routing and per-replica telemetry make the imbalance visible.
Failure lines reveal which serving layer you need #
OOM is a ledger violation or an allocation spike. Separate load-time weights, runtime peak, logical KV, physical KV blocks, allocator reserve, and communication buffers. Record the exact phase: model load, prefill admission, decode growth, or graph capture. Lowering maximum concurrency may hide an underestimated per-request cache; lowering maximum sequence length may hide an unbounded admission policy.
Queue collapse begins when sustained admitted work exceeds completion capacity. Queue age rises, TTFT stretches, clients retry, and retries increase offered load. Admission control, bounded queues, load shedding, and retry budgets prevent a slow service from becoming a positive-feedback loop. Scaling after saturation helps only if new replicas become ready before the queue and retry storm consume the deadline.
Cache thrash appears as frequent preemption, eviction, swapping, or prefix recomputation. The engine may remain below an OOM threshold while useful work falls because the same prompt tokens are processed repeatedly. Compare logical live KV bytes with reserved block bytes, allocation failures, prefix-cache hit rate, and recomputed tokens. A block-size change should be tested against the real sequence distribution rather than chosen from average length.
The selection boundary follows ownership. Choose an engine when you control model processes and need direct scheduling, kernels, cache policy, and hardware tuning. Choose an endpoint platform when you want a deployable model service with autoscaling, routing, observability, and operational APIs while retaining meaningful runtime choices. Choose managed inference when the provider can own accelerators, upgrades, availability, and most capacity planning under a contract you can test.
The boundary is set by required control, evidence, and staff time. Custom kernels and cache policies demand engine ownership. Data residency, fixed revisions, or strict tail SLOs may demand a dedicated endpoint. Variable traffic and a small operations team may favor managed capacity. In every case, preserve the same benchmark input record and latency vocabulary. Abstraction can transfer operation; it cannot make workload assumptions disappear.
Portability has limits. An engine configuration couples model architecture, quantized format, kernel availability, device generation, parallelism, and traffic. An endpoint contract couples scale behavior, quota, routing, observability, and revision policy. A managed API may expose only request-level metrics. Before choosing, identify the lowest layer where the team must diagnose a missed SLO. If the necessary cache, scheduler, or kernel evidence exists only below the product boundary, either require that evidence contractually or own the lower layer.
Compare the exact model and revision, supported precision, maximum input/output tokens, isolation, cold-start behavior, streaming semantics, cancellation, observability, measured TTFT/ITL under your traffic, failure policy, data handling, and total operating ownership. A vendor throughput row without those fields cannot support the decision.
- [01]
[Agrawal et al., “SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills,” 2023](https://arxiv.org/abs/2308.16369)[↩](#cite-agrawal-2023-sarathi) - [02]
[Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving,” OSDI 2024](https://arxiv.org/abs/2401.09670)[↩](#cite-zhong-2024-distserve) - [03]
[Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” EMNLP 2023](https://arxiv.org/abs/2305.13245)[↩](#cite-ainslie-2023-gqa) - [04]
[Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023](https://arxiv.org/abs/2309.06180)[↩](#cite-kwon-2023-pagedattention) - [05]
[Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models,” OSDI 2022](https://www.usenix.org/conference/osdi22/presentation/yu)[↩](#cite-yu-2022-orca) - [06]
[Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” NeurIPS 2022](https://arxiv.org/abs/2205.14135)[↩](#cite-dao-2022-flashattention) - [07]
[Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,” ICML 2023](https://arxiv.org/abs/2211.10438)[↩](#cite-xiao-2023-smoothquant)
Anything on this sheet still unclear — or anything you were too polite to ask out loud? File an RFI. Answers come from the drawing itself and cite their sheet numbers, and every question is recorded in the drawing log so the next revision can answer it in print.