Measuring Performance of Transformer Inference

A new guide on measuring transformer inference performance details metrics such as latency, time to first token (TTFT), time per output token (TPOT), throughput, memory usage, utilization, and cost per token, emphasizing the need to report high-percentile latencies (p90, p95, p99) alongside mean and median. The guide, part of a chapter on LLM inference optimization, provides Python code using time.perf_counter() and CUDA events for accurate measurement, and stresses separating prefill and decode phases due to different performance profiles.

When you optimize the inference performance of an LLM, you need to know how to measure it. Without measurement, it is easy to make a model more complicated without making it faster, or to improve throughput while making user-visible latency worse. An LLM service has several kinds of performance. A user cares about how long it takes to see the first token and how quickly the rest of the answer streams. An operator cares about how many requests the hardware can serve, how much memory is used, and how much each generated token costs. A researcher may care about whether an optimization changes the model’s output quality. In this chapter, you will learn about: - Latency and throughput metrics - Time to first token and time per output token - Measuring CPU and GPU inference - Using CUDA events - Benchmarking multiple requests - Thinking about multiple GPUs and multiple machines Let’s get started. Overview This chapter is divided into eight parts; they are: - Metrics for LLM Inference - Measuring a Single Request - Warmup and Synchronization - Measuring GPU Work with CUDA Events - Measuring Memory Usage - Measuring Concurrent Requests - Multiple GPUs and Multiple Machines - Cost per Token Metrics for LLM Inference The most common inference metrics are: Latency: How long a request takes from start to finish. Time to first token TTFT : How long the user waits before the first output token appears. Time per output token TPOT : The average time between generated tokens after the first token. Throughput: How many tokens or requests are processed per second. Memory usage: How much CPU memory or GPU memory is used. Utilization: How busy the accelerator is during the benchmark. Cost per token: The hardware or service cost divided by the number of tokens processed. For LLMs, a single latency number is usually not enough. Consider two requests: - Request A: 2,000 prompt tokens and 20 output tokens - Request B: 20 prompt tokens and 2,000 output tokens Request A stresses prefill. Request B stresses decode. They may have the same total number of tokens, but they have different performance profiles. This is why you should record prompt tokens and output tokens separately. Tail latency also matters. If most requests complete in one second but a few take ten seconds, users will notice. Report high-percentile latencies such as p90, p95, and p99 in addition to the mean or median. The high percentiles describe the worst cases better. You can easily find these percentiles from a list of values using NumPy: python import numpy as np def summarize values : values = np.asarray values, dtype=np.float64 return { "mean": values.mean , "median": np.percentile values, 50 , "p90": np.percentile values, 90 , "p95": np.percentile values, 95 , "p99": np.percentile values, 99 , } 1234567891011 import numpy as np def summarize values : values = np.asarray values, dtype=np.float64 return { "mean": values.mean , "median": np.percentile values, 50 , "p90": np.percentile values, 90 , "p95": np.percentile values, 95 , "p99": np.percentile values, 99 , } These numbers are simple, but they prevent a common mistake: optimizing the average while making the worst cases slower. Measuring a Single Request The simplest measurement uses time.perf counter . It is a built-in high-resolution wall-clock timer suitable for measuring elapsed time in Python. It is more accurate than time.time . The following example measures prefill and decode separately for a Hugging Face causal language model: python import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer def load model model name="sshleifer/tiny-gpt2", device="cpu" : tokenizer = AutoTokenizer.from pretrained model name model = AutoModelForCausalLM.from pretrained model name .to device model.eval return tokenizer, model @torch.no grad def measure one request model, tokenizer, prompt, max new tokens=50, device="cpu" : input ids = tokenizer prompt, return tensors="pt" .input ids.to device start = time.perf counter outputs = model input ids, use cache=True prefill end = time.perf counter past key values = outputs.past key values next token = outputs.logits :, -1, : .argmax dim=-1, keepdim=True generated = next token decode times = for in range max new tokens - 1 : step start = time.perf counter outputs = model next token, past key values=past key values, use cache=True, Note: You may need torch.cuda.synchronize here step end = time.perf counter decode times.append step end - step start past key values = outputs.past key values next token = outputs.logits :, -1, : .argmax dim=-1, keepdim=True generated.append next token if tokenizer.eos token id is not None: if next token.item == tokenizer.eos token id: break end = time.perf counter output ids = torch.cat input ids + generated, dim=1 return { "text": tokenizer.decode output ids 0 , skip special tokens=True , "prompt tokens": input ids.size 1 , "output tokens": len generated , "prefill seconds": prefill end - start, "decode seconds": sum decode times , "total seconds": end - start, "ttft seconds": prefill end - start, "seconds per output token": sum decode times / max 1, len decode times , } 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960 import timeimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer def load model model name="sshleifer/tiny-gpt2", device="cpu" : tokenizer = AutoTokenizer.from pretrained model name model = AutoModelForCausalLM.from pretrained model name .to device model.eval return tokenizer, model @torch.no grad def measure one request model, tokenizer, prompt, max new tokens=50, device="cpu" : input ids = tokenizer prompt, return tensors="pt" .input ids.to device start = time.perf counter outputs = model input ids, use cache=True prefill end = time.perf counter past key values = outputs.past key values next token = outputs.logits :, -1, : .argmax dim=-1, keepdim=True generated = next token decode times = for in range max new tokens - 1 : step start = time.perf counter outputs = model next token, past key values=past key values, use cache=True, Note: You may need torch.cuda.synchronize here step end = time.perf counter decode times.append step end - step start past key values = outputs.past key values next token = outputs.logits :, -1, : .argmax dim=-1, keepdim=True generated.append next token if tokenizer.eos token id is not None: if next token.item == tokenizer.eos token id: break end = time.perf counter output ids = torch.cat input ids + generated, dim=1 return { "text": tokenizer.decode output ids 0 , skip special tokens=True , "prompt tokens": input ids.size 1 , "output tokens": len generated , "prefill seconds": prefill end - start, "decode seconds": sum decode times , "total seconds": end - start, "ttft seconds": prefill end - start, "seconds per output token": sum decode times / max 1, len decode times , } This function does not use the model’s generate method. That is intentional. The goal is to expose prefill and decode so they can be measured separately. The number of output tokens includes the tokens generated by both prefill and decode. The seconds per output token is the average time per output token in the decode phase. There are two details to notice: use cache=True asks the model to return the KV cache.- During decode, the model receives only next token , not the whole sequence. This is the same idea as Chapter 1, but using a library model. Warmup and Synchronization When you measure performance, note that some one-time costs should not dominate the result. In Python, import of a module can be slow but subsequent import of the same module is instant. Similarly, the first execution of some code may be slower than subsequent executions due to initialization of data structures or warmup of caches. You want to measure steady-state work, not that setup overhead. Therefore, benchmarks should include warmup. The first few iterations may be slower for various reasons. Instead of measuring the total time and dividing by the number of iterations, you should measure the time for each iteration and analyze the steady-state ones. For example, if you use the model to generate multiple tokens, you will likely put the generation in a loop. Measure each iteration as follows, then ignore the first few results: python def iterations model, tokenizer, prompt, device, steps=100, warmup=10 : results = for in range steps : result = measure one request model, tokenizer, prompt, max new tokens=8, device=device, results.append result steady = results warmup: return summarize item "total seconds" for item in steady 12345678910111213 def iterations model, tokenizer, prompt, device, steps=100, warmup=10 : results = for in range steps : result = measure one request model, tokenizer, prompt, max new tokens=8, device=device, results.append result steady = results warmup: return summarize item "total seconds" for item in steady If you use GPU to run your LLM inference, you also need to initialize the kernels when you first run them. Unfortunately, many GPU operations are asynchronous. That is, while you launched an operation on GPU, Python may continue with your code immediately while the GPU is still working. Therefore, a naive approach to measure the time would be incorrect. Instead, you should use torch.cuda.synchronize to wait for the GPU to finish the operation before you stop the timer: python def sync if needed device : if device.startswith "cuda" : torch.cuda.synchronize start = time.perf counter outputs = model input ids, use cache=True sync if needed device elapsed = time.perf counter - start 12345678 def sync if needed device : if device.startswith "cuda" : torch.cuda.synchronize start = time.perf counter outputs = model input ids, use cache=True sync if needed device elapsed = time.perf counter - start This gives a wall-clock measurement that includes the actual GPU work. For accurate prefill and per-token decode timings on GPU, call sync if needed device after each timed model ... call in measure one request , not only once at the end of the request. Measuring GPU Work with CUDA Events CUDA events measure elapsed time the GPU spent executing kernels, not the end-to-end user latency. This time does not include any Python overhead. Below is an example of how to use CUDA events to measure the time: python def cuda event time fn : start = torch.cuda.Event enable timing=True end = torch.cuda.Event enable timing=True start.record result = fn end.record torch.cuda.synchronize milliseconds = start.elapsed time end return result, milliseconds / 1000.0 1234567891011 def cuda event time fn : start = torch.cuda.Event enable timing=True end = torch.cuda.Event enable timing=True start.record result = fn end.record torch.cuda.synchronize milliseconds = start.elapsed time end return result, milliseconds / 1000.0 You can use it to measure one forward pass: with torch.no grad : outputs, seconds = cuda event time lambda: model input ids, use cache=True print f"GPU forward time: {seconds:.6f} seconds" 123456 with torch.no grad : outputs, seconds = cuda event time lambda: model input ids, use cache=True print f"GPU forward time: {seconds:.6f} seconds" CUDA event timing and wall-clock timing answer different questions: - Wall-clock timing measures what the application experiences. - CUDA event timing measures how long the GPU work took. For an inference service, wall-clock timing is usually the primary metric because users experience queues, tokenization, scheduling, network overhead, and streaming. CUDA events are useful when you are optimizing kernels or comparing model execution paths. For deeper GPU profiling, use tools such as PyTorch Profiler, Nsight Systems, Nsight Compute, or CUPTI-based monitoring. These tools can report kernel timelines, memory copies, GPU utilization, and operator-level breakdowns. They are more complex than a timer, but they are necessary when a simple benchmark says the model is slow and you need to know why. Measuring Memory Usage Memory is a different dimension to measure because it limits not speed for one user so much as how many users your system can serve. Usually the GPU memory is the bottleneck. In PyTorch, you can report allocated and reserved memory like the following: python def gpu memory summary device="cuda" : torch.cuda.synchronize return { "allocated gb": torch.cuda.memory allocated device / 1e9, "reserved gb": torch.cuda.memory reserved device / 1e9, "max allocated gb": torch.cuda.max memory allocated device / 1e9, } 1234567 def gpu memory summary device="cuda" : torch.cuda.synchronize return { "allocated gb": torch.cuda.memory allocated device / 1e9, "reserved gb": torch.cuda.memory reserved device / 1e9, "max allocated gb": torch.cuda.max memory allocated device / 1e9, } The allocated value is memory used by tensors. The reserved value is memory held by PyTorch’s caching allocator. The maximum allocated value is often the most useful number for capacity planning. The allocated and reserved memory are real-time snapshots but the max allocated value is a peak over time. For accurate measurement, you should reset the peak statistic before a benchmark: torch.cuda.reset peak memory stats result = measure one request model, tokenizer, prompt, device="cuda" memory = gpu memory summary "cuda" print memory 1234 torch.cuda.reset peak memory stats result = measure one request model, tokenizer, prompt, device="cuda" memory = gpu memory summary "cuda" print memory Memory should be measured together with tokens. A run with a longer prompt or more generated tokens will naturally use more KV cache memory. Measuring Concurrent Requests To create a server that runs a language model, evaluate the system by how many requests you can serve per second. Throughput depends on both how fast you fulfill one request and how many requests you can run concurrently, though concurrency does not scale linearly under contention. Production systems should handle multiple users, and the scheduler may batch their work together. The following simple benchmark runs several requests concurrently using Python threads. This does not implement continuous batching. It only measures how a model wrapper behaves when several callers use it at the same time. python from concurrent.futures import ThreadPoolExecutor, as completed def run prompt model, tokenizer, prompt, device : start = time.perf counter result = measure one request model, tokenizer, prompt, max new tokens=32, device=device, end = time.perf counter result "wall seconds" = end - start return result def benchmark concurrent model, tokenizer, prompts, device="cpu", workers=4 : results = start = time.perf counter with ThreadPoolExecutor max workers=workers as pool: futures = pool.submit run prompt, model, tokenizer, prompt, device for prompt in prompts for future in as completed futures : results.append future.result end = time.perf counter total output tokens = sum item "output tokens" for item in results return { "requests": len results , "total seconds": end - start, "output tokens": total output tokens, "output tokens per second": total output tokens / end - start , "latency summary": summarize item "wall seconds" for item in results , } 1234567891011121314151617181920212223242526272829303132333435363738 from concurrent.futures import ThreadPoolExecutor, as completed def run prompt model, tokenizer, prompt, device : start = time.perf counter result = measure one request model, tokenizer, prompt, max new tokens=32, device=device, end = time.perf counter result "wall seconds" = end - start return result def benchmark concurrent model, tokenizer, prompts, device="cpu", workers=4 : results = start = time.perf counter with ThreadPoolExecutor max workers=workers as pool: futures = pool.submit run prompt, model, tokenizer, prompt, device for prompt in prompts for future in as completed futures : results.append future.result end = time.perf counter total output tokens = sum item "output tokens" for item in results return { "requests": len results , "total seconds": end - start, "output tokens": total output tokens, "output tokens per second": total output tokens / end - start , "latency summary": summarize item "wall seconds" for item in results , } This benchmark is for illustration only. It is not a replacement for a real serving benchmark. It does not model HTTP overhead, streaming, request queues, batching, cancellation, or cache eviction. But it is a useful next step after a single-request benchmark where you can run the model in parallel and observe the per-request latency. The Python GIL Global Interpreter Lock is usually not the main concern here because heavyweight model execution is often offloaded to compiled code. Concurrent use of the same model or tensors from multiple threads is unsafe without synchronization, so share one model on CUDA only with a lock or a single worker thread. When benchmarking a real server, record at least: - Number of concurrent users - Prompt token distribution - Output token distribution - Request rate - TTFT Time to first token percentiles - Inter-token latency percentiles - Total tokens per second - Error rate and timeout rate The distributions matter. A benchmark with all prompts at exactly 128 tokens and all outputs at exactly 128 tokens is easy to compare, but it may not represent your application. Multiple GPUs and Multiple Machines Multiple GPUs can be used for inference in several different ways. Different approaches can drastically change the performance of your system. The simplest approach is replication. You load one copy of the model on each GPU and route different requests to different replicas. This increases throughput and is easy to reason about, but each GPU must have enough memory for the full model and its KV cache. Another approach is to split one model across multiple GPUs. Tensor parallelism splits weight matrices across devices. Pipeline parallelism places different layers on different devices. Context parallelism partitions sequence work. Expert parallelism is used for mixture-of-experts models. These techniques allow larger models to run, but they introduce communication overhead and can increase latency. Multiple machines add another layer. A system may use many replicas across machines for high request volume. It may also split a single large model across machines, but this is more difficult because network communication is slower than communication within one machine. For low-latency serving, crossing machine boundaries inside one forward pass should be treated as expensive. Measuring performance of a system with multiple GPUs or multiple machines adds a new dimension of communication and synchronization overhead. Before choosing a multi-GPU or multi-machine design, answer these questions: - Are you serving one large model or many smaller models? - Are you limited by model weight memory or KV cache memory? - Do you need lower latency, higher throughput, or both? - Are requests independent, or do they share long prompt prefixes? - Can one GPU hold the model, or must the model be partitioned? These questions matter because the best design depends on the bottleneck. Adding GPUs does not automatically make a single request faster. It may help throughput through replication, or it may make a larger model possible through partitioning. The benchmark should show which effect you are getting. Cost per Token Cost is a performance metric. A faster system that uses much more expensive hardware may not be better for an application. A simple cost estimate is: cost per output token = hardware cost per second / output tokens per second 1 cost per output token = hardware cost per second / output tokens per second If a GPU instance costs 3 dollars per hour and the service generates 1,000 output tokens per second: hardware cost per second = 3.00 / 3600 = 0.000833 cost per output token = 0.000833 / 1000 = 0.000000833 123 hardware cost per second = 3.00 / 3600 = 0.000833cost per output token = 0.000833 / 1000 = 0.000000833 This is less than one millionth of a dollar per output token for hardware alone. A real calculation may also include idle capacity, storage, networking, engineering time, orchestration overhead, and failed requests. Cost should be compared with quality. Quantization, smaller models, and routing can reduce cost, but they may change model behavior. An efficient inference system is not merely the fastest one. It is the one that meets quality and reliability requirements at the lowest practical cost. Further Reading Below are some resources you may find useful: Little’s law https://en.wikipedia.org/wiki/Little%27s law , on Wikipedia. This is a useful queueing-theory result for relating average concurrency, arrival rate, and response time. It is a helpful mental model when reasoning about request rate, latency, and the number of in-flight inference requests. Metrics https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html , in NVIDIA NIM LLMs Benchmarking. This page defines common LLM inference metrics such as time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second. MLPerf Inference https://docs.mlcommons.org/inference/ , by MLCommons. This is a widely used benchmark suite for measuring inference performance across deployment scenarios. It is not limited to LLMs, but it provides useful discipline around repeatable benchmarking and reporting. torch.profiler https://docs.pytorch.org/docs/stable/profiler.html , in the PyTorch documentation. This is the main PyTorch profiling interface for collecting CPU and accelerator activity, operator timings, memory information, tensor shapes, and traces that can be inspected later. NVIDIA Nsight Systems User Guide https://docs.nvidia.com/nsight-systems/UserGuide/index.html , by NVIDIA. Nsight Systems is useful when wall-clock timers are not enough and you need a timeline of CUDA API calls, GPU kernels, memory copies, CPU work, and synchronization. Taming the Titans: A Survey of Efficient LLM Inference Serving https://arxiv.org/abs/2504.19720 , by Zhen et al. This survey gives a broader view of LLM inference serving, including request scheduling, model placement, storage management, disaggregation, load balancing, and cluster-level serving issues. Summary In this chapter, you learned how to measure LLM inference performance. You saw why prefill and decode should be measured separately, how to use wall-clock timers and CUDA events, how to record memory usage, and how to report latency percentiles. You also learned that multiple GPUs can mean replication for more throughput or partitioning for larger models, and that the benchmark should make this distinction clear. In the next part of the book, you will begin studying techniques for making one model faster, starting with floating-point precision.