The Ultimate Guide to LLM Inference Optimization- Part 1 LLM inference optimization is the process of making large language models respond faster, cheaper, or with fewer hardware resources while preserving accuracy, and it can be categorized into model-level, inference service-level, and hardware-level optimization. Model compression is essential because a 70B parameter model at FP16 requires approximately 140 GB of VRAM just for weights, and the KV cache adds significant memory that scales with context length and batch size. This guide, written for engineers and practitioners, focuses on model-level techniques in Parts 1 and 2, with Part 3 covering inference service-level optimization. 2026 is again one of those years you realize that big tech is not gonna stop on AI, even with bloated revenue, mounting infrastructure costs, and concerns of unhinged progress. New models come and go each day, but what never changes is optimizing AI for your use cases and practicality. As an engineer, the question I have to ask is, “Can I actually afford to run this model?” LLM inference optimization is the process of making LLMs respond faster, cheaper, or with fewer hardware resources while preserving “ Making a model faster often means sacrificing some accuracy . Reducing memory usage can lower hardware costs , but it may also affect output quality . Optimizing for higher throughput requests/sec or tokens/sec can increase latency for individual users, while minimizing latency may reduce the number of requests a system can handle at once. There is no single best optimization for every application. Optimization is a broad topic with many moving parts. For simplicity, it can be roughly categorized into model-level , inference service-level , and hardware-level optimization . Model-level optimization involves improving the model itself with architectural and structural modifications, whereas inference service-level optimization focuses on the software layer that manages serving. Hardware optimization is almost entirely about chip design and powerful accelerator technologies. Since hardware deserves a deep discussion of its own, we’ll leave it out of scope for this series. We will focus more on the former two. To make it easy to follow and understand, this guide is split into two parts. Part 1 & 2 will focus only on the model-level optimization, the techniques that make models smaller, faster, and more efficient without fundamentally changing how they’re served. Part-3 will be a separate article on inference service-level optimization , to be released shortly, where we’ll look at the software techniques. This guide is written for anyone who knows what LLMs are but doesn’t necessarily have a deep background in machine learning or hardware. And don’t worry if terms like “memory-bound” or “compute-bound” sound intimidating. You’ll come across plenty of jargon along the way, but I’ll introduce and explain each concept as we meet them. Let’s dive in Model optimization is an active area of research since the advent of neural networks. In the case of LLMs, the same techniques apply for optimizing network size. But when it comes to transformer-specific bottlenecks like the autoregressive nature and attention mechanism drawbacks, LLM-specific ones apply too. Let’s explore each one in detail. Why should models be compressed? As the name suggests, this is all about making the model smaller. But why should it be smaller? Let us understand by a simple intuition exercise. Imagine you’ve decided to deploy an open-source 70B parameter model. The first thing you need to store is the model’s parameters. These are the learned weights that contain everything the model knows. Each parameter occupies memory depending on its numerical precision. At FP16 16-bit floating point; full precision , every parameter requires 2 bytes of storage. 70 billion parameters × 2 bytes≈ 140 GB That means just loading the model into memory already requires around 140 GB of VRAM. Many people stop here; that’s the mistake. The KV key-value cache deserves special attention because it grows continuously with conversation we will discuss KV cache in detail later . Unlike model weights, whose size never changes, the KV cache scales with the number of layers, hidden dimension, number of attention heads, context length, and batch size. A simple rough approximation of KV cache for an initial budget is: KV Cache ≈ 2 × Layers × Hidden Size × Sequence Length × Bytes per Element For a Llama-class 70B model with an 8K-token context this value is very low for most use cases , the KV cache alone is roughly 10–12 GB in FP16. Increase the context window to 128K tokens, and the KV cache can consume well over 150 GB , often becoming larger than the model weights themselves. To maximize memory for KV cache, frameworks like vLLM reserve around 90% of the available GPU memory by default, leaving a small safety margin to avoid out-of-memory errors and CUDA fragmentation. This is one of the biggest reasons LLM inference is so memory-intensive. Then comes activation and runtime , which require some memory for storing computations and other stuff. Rounding it off, here is a quick summary. An NVIDIA H100 80 GB GPU costs around $25,000–30,000 USD , meaning the 2 GPUs alone for our setup would cost $50,000–70,000 USD . For most teams, buying dedicated AI hardware isn’t practical, so cloud GPUs are the preferred option. On Azure cloud, an H100 costs roughly $12.29 per GPU per hour on pay-as-you-go pricing, with reserved instances reducing that to around $5.47–7.93 per hour . While that may sound reasonable, running just two H100 GPUs continuously costs about $590 per day or over $215,000 per year . We’re discussing one model instance. If your application needs to serve hundreds or thousands of concurrent users, these costs multiply rapidly. This is precisely why model compression exists. How are models stored in memory? To understand compression, you need a small idea of how models are represented and stored in memory. As you may already know, floating-point numbers have traditionally been the gold standard. FP32, FP16, BF16, and TF32 are widely used for large language models. Each representation has 3 parts. Formats with more bits are considered higher precision. Here is an illustration for you to understand how bits are distributed in different formats. The understanding required here is that a number that is represented in FP32 may not be exactly representable in FP16 or other lower-precision formats. For example, consider the FP32 value 0.1. Although FP32 can represent this value with relatively high precision, 0.1 cannot be represented exactly in FP16. When converting the value from FP32 to FP16, it is rounded to the nearest representable FP16 value, which is approximately 0.0999756, causing information loss of 0.0001. Quantization — The Compression From the logic above, it is clear that as we represent the LLM weights in lower precisions, less memory is required to store and run the model. Quantization is the cheapest and most effective optimization. This is because lower precision not only reduces the model's memory footprint but also enhances inference speed, provided the hardware natively supports it. To be clearer, think of an 8-bit operation that works with half the amount of data compared to a 16-bit operation, so the hardware can often process more values at once, making inference faster. When it comes to quantization, numerous data types in different precisions have been explored by the research community. This includes FP4, FP8, INT8 https://arxiv.org/abs/2208.07339 2 , INT4 among many others. We even reached 1-bit introduced in BitNet https://arxiv.org/abs/2310.11453 4 , which is actually very extreme considering the data you can represent with that precision. Personally, as I worked on NVIDIA Blackwell GPUs and automotive Drive AGX Thor devices, NVFP4 https://arxiv.org/abs/2509.25149 1 worked well for me. It pushed quantization down to 4 bits while maintaining good model output accuracy through techniques such as microscaling . There is also something called mixed precision where some weights are stored in higher and some in lower precision. You can read this article https://machinelearning.apple.com/research/introducing-apple-foundation-models later to see how Apple achieved an average of 3.7 bits per weight using a mix of 2-bit and 4-bit quantization for its on-device models. Introducing Apple's On-Device and Server Foundation Models https://machinelearning.apple.com/research/introducing-apple-foundation-models Quantization can be roughly visualized as converting each point on a ruler to a corresponding point on a smaller ruler. Suppose our FP32 weight is 0.73. In FP32, this number is stored with 32 bits, giving us a lot of room to represent its value precisely. But INT8 has only 8 bits, so it can store only integer values from −128 to 127 . We therefore need a way to map our decimal FP32 values onto this much smaller set of integers. Imagine we have a group of weights ranging from −1.0 to +1.0 . We can use this range to determine a scaling factor also called quantization constant : scale = 1.0 / 127 ≈ 0.00787 This tells us that every step in the INT8 representation corresponds to roughly 0.00787 in the original FP32 range. Now we take our FP32 weight, 0.73 , and divide it by the scale: 0.73 / 0.00787 ≈ 92.7 Since INT8 can only store integers, we round this to the nearest integer: 92.7 → 93 So our original weight has now gone through: FP32: 0.73 → scaled value: 92.7 → INT8: 93 The INT8 model stores 93 , not 0.73. But we also keep the scale 0.00787 so that the value can be approximately reconstructed later: 93 × 0.00787 ≈ 0.732 The original value was 0.73 , but after quantization and reconstruction we get approximately 0.732 . This small difference, 0.002 , is the quantization error introduced by reducing the precision from FP32 to INT8. When it comes to quantizing large language models LLMs , there are two primary types of techniques: Post-Training Quantization PTQ As the name suggests, the LLM is quantized after training. The weights are converted from a higher precision to a lower precision data type. It can be applied to both weights and activations. Although speed, memory, and power usage are optimized, there is an accuracy trade-off that can be attributed to the Quantization-Aware Training This technique was developed to mitigate the potential loss of model accuracy in the case of PTQ. In contrast to PTQ, the quantization process is integrated with the training itself, hence making the process “Quantization Aware”. In QAT, the model architecture is initially modified to maintain both full-precision and quantized versions of elements, which includes weights and activations, thereby creating a dual storage system. During the forward pass of the training process, a simulated or “fake” quantization is introduced to the model, allowing it to experience the effects of quantization while still preserving the precision when calculating gradients, thereby enhancing the model’s robustness to quantization. I have discussed LLM quantization in detail with various methods in another blog, which you can check here https://medium.com/data-science/the-ultimate-handbook-for-llm-quantization-88bb7cb0d9d7 . The Ultimate Handbook for LLM Quantization https://medium.com/data-science/the-ultimate-handbook-for-llm-quantization-88bb7cb0d9d7 You must have come across LLMs named as DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-1.5B . By the end of this section, these names will make perfect sense. Even though distillation in neural networks has been around for a long time, one of the most influential examples for Transformer-based language models is the classic DistilBERT 5 paper. It builds on a simple intuition. The bigger model we are training from is called the teacher model, and the smaller model that receives the training is called the Broadly speaking, knowledge transfer in distillation can be achieved in two ways. KL divergence Kullback–Leibler divergence measures how different one probability distribution is from another reference distribution. In LLM distillation, the teacher model provides the reference distribution, typically through its output probabilities or logits , and the student model is trained to produce a similar distribution. The KL divergence loss penalizes the student when its predicted distribution differs from the teacher’s, encouraging the student to mimic the teacher’s behavior while using a smaller model. For this method, mainstream libraries like PyTorch have provided out-of-the-box implementations such as torchtune https://pytorch.org/blog/llama-into-torchtune/ . download teacher modeltune download meta-llama/Meta-Llama-3.1-8B-Instruct --output-dir /tmp/Meta-Llama-3.1-8B-Instruct --ignore-patterns "original/consolidated.00.pth" --hf token