2026 is again one of those years you realize that big tech is not gonna stop on AI, even with bloated revenue, mounting infrastructure costs, and concerns of unhinged progress. New models come and go each day, but what never changes is optimizing AI for your use cases and practicality. As an engineer, the question I have to ask is,
“Can I actually afford to run this model?”
** LLM inference optimization** is the process of making LLMs respond faster, cheaper, or with fewer hardware resources while preserving “
Making a model **faster **often means sacrificing some accuracy. Reducing **memory **usage can lower hardware costs, but it may also affect output quality. Optimizing for higher **throughput **(requests/sec or tokens/sec) can increase **latency **for individual users, while minimizing latency may reduce the number of requests a system can handle at once. There is no single best optimization for every application.
Optimization is a broad topic with many moving parts. For simplicity, it can be roughly categorized into model-level, inference service-level, and hardware-level optimization. Model-level optimization involves improving the model itself with architectural and structural modifications, whereas inference service-level optimization focuses on the software layer that manages serving. Hardware optimization is almost entirely about chip design and powerful accelerator technologies. Since hardware deserves a deep discussion of its own, we’ll leave it out of scope for this series. We will focus more on the former two.
To make it easy to follow and understand, this guide is split into two parts. Part 1 & 2 will focus only on the **model-level optimization, **the techniques that make models smaller, faster, and more efficient without fundamentally changing how they’re served. Part-3 will be a separate article on inference service-level optimization, to be released shortly, where we’ll look at the software techniques.
This guide is written for anyone who knows what LLMs are but doesn’t necessarily have a deep background in machine learning or hardware. And don’t worry if terms like “memory-bound” or “compute-bound” sound intimidating. You’ll come across plenty of jargon along the way, but I’ll introduce and explain each concept as we meet them. Let’s dive in!
Model optimization is an active area of research since the advent of neural networks. In the case of LLMs, the same techniques apply for optimizing network size. But when it comes to transformer-specific bottlenecks like the autoregressive nature and attention mechanism drawbacks, LLM-specific ones apply too. Let’s explore each one in detail.
Why should models be compressed?
As the name suggests, this is all about making the model smaller. But why should it be smaller? Let us understand by a simple intuition exercise.
Imagine you’ve decided to deploy an open-source 70B parameter model. The first thing you need to store is the model’s parameters. These are the learned weights that contain everything the model knows. Each parameter occupies memory depending on its numerical precision. At FP16 (16-bit floating point; full precision), every parameter requires 2 bytes of storage.
70 billion parameters × 2 bytes≈ 140 GB
That means just the model into memory already requires around** 140 GB **of VRAM. Many people stop here; that’s the mistake.
The KV (key-value) cache deserves special attention because it grows continuously with conversation (we will discuss KV cache in detail later). Unlike model weights, whose size never changes, the KV cache scales with the number of layers, hidden dimension, number of attention heads, context length, and batch size.
A simple rough approximation of KV cache for an initial budget is:
KV Cache ≈ 2 × Layers × Hidden Size × Sequence Length × Bytes per Element
For a Llama-class 70B model with an 8K-token context (this value is very low for most use cases), the KV cache alone is roughly 10–12 GB in FP16. Increase the context window to 128K tokens, and the KV cache can consume well over 150 GB, often becoming larger than the model weights themselves. To maximize memory for KV cache, frameworks like vLLM reserve around 90% of the available GPU memory by default, leaving a small safety margin to avoid out-of-memory errors and CUDA fragmentation. This is one of the biggest reasons LLM inference is so memory-intensive.
Then comes **activation **and runtime, which require some memory for storing computations and other stuff. Rounding it off, here is a quick summary.
An NVIDIA H100 80 GB GPU costs around $25,000–30,000 USD, meaning the 2 GPUs alone for our setup would cost $50,000–70,000 USD. For most teams, buying dedicated AI hardware isn’t practical, so cloud GPUs are the preferred option.
On Azure cloud, an H100 costs roughly $12.29 per GPU per hour on pay-as-you-go pricing, with reserved instances reducing that to around $5.47–7.93 per hour. While that may sound reasonable, running just two H100 GPUs continuously costs about $590 per day or over** $215,000 per year**. We’re discussing one model instance. If your application needs to serve hundreds or thousands of concurrent users, these costs multiply rapidly.
This is precisely why model compression exists.
How are models stored in memory?
To understand compression, you need a small idea of how models are represented and stored in memory. As you may already know, floating-point numbers have traditionally been the gold standard. FP32, FP16, BF16, and TF32 are widely used for large language models. Each representation has 3 parts.
Formats with more bits are considered higher precision. Here is an illustration for you to understand how bits are distributed in different formats.
The understanding required here is that a number that is represented in FP32 may not be exactly representable in FP16 or other lower-precision formats. For example, consider the FP32 value 0.1. Although FP32 can represent this value with relatively high precision, 0.1 cannot be represented exactly in FP16. When converting the value from FP32 to FP16, it is rounded to the nearest representable FP16 value, which is approximately 0.0999756, causing information loss of 0.0001.
Quantization — The Compression
From the logic above, it is clear that as we represent the LLM weights in lower precisions, less memory is required to store and run the model. **Quantization **is the cheapest and most effective optimization. This is because lower precision not only reduces the model's memory footprint but also enhances inference speed, provided the hardware natively supports it. To be clearer, think of an 8-bit operation that works with half the amount of data compared to a 16-bit operation, so the hardware can often process more values at once, making inference faster.
When it comes to quantization, numerous data types in different precisions have been explored by the research community. This includes FP4, FP8, INT8 [2], INT4 among many others. We even reached 1-bit (introduced in BitNet) [4], which is actually very extreme considering the data you can represent with that precision. Personally, as I worked on NVIDIA Blackwell GPUs and automotive Drive AGX Thor devices, NVFP4 [1] worked well for me. It pushed quantization down to 4 bits while maintaining good model output accuracy through techniques such as microscaling.
There is also something called **mixed precision **where some weights are stored in higher and some in lower precision. You can read this article later to see how Apple achieved an average of 3.7 bits per weight using a mix of 2-bit and 4-bit quantization for its on-device models.
Introducing Apple's On-Device and Server Foundation Models
Quantization can be roughly visualized as converting each point on a ruler to a corresponding point on a smaller ruler. Suppose our FP32 weight is *0.73. *In FP32, this number is stored with 32 bits, giving us a lot of room to represent its value precisely. But INT8 has only 8 bits, so it can store only integer values from −128 to 127. We therefore need a way to map our decimal FP32 values onto this much smaller set of integers.
Imagine we have a group of weights ranging from* −1.0* to +1.0. We can use this range to determine a scaling factor (also called quantization constant):
scale = 1.0 / 127 ≈ 0.00787
This tells us that every step in the INT8 representation corresponds to roughly 0.00787 in the original FP32 range. Now we take our FP32 weight, 0.73, and divide it by the scale:
0.73 / 0.00787 ≈ 92.7
Since INT8 can only store integers, we round this to the nearest integer:
92.7 → 93
So our original weight has now gone through:
FP32: 0.73 → scaled value: 92.7 → INT8: 93
The INT8 model stores* 93*, not 0.73. But we also keep the scale (0.00787) so that the value can be approximately reconstructed later:
93 × 0.00787 ≈ 0.732
The original value was 0.73, but after quantization and reconstruction we get approximately 0.732. This small difference, 0.002, is the quantization error introduced by reducing the precision from FP32 to INT8.
When it comes to quantizing large language models (LLMs), there are two primary types of techniques:
Post-Training Quantization (PTQ) As the name suggests, the LLM is quantized after training. The weights are converted from a higher precision to a lower precision data type. It can be applied to both weights and activations. Although speed, memory, and power usage are optimized, there is an accuracy trade-off that can be attributed to the
Quantization-Aware Training This technique was developed to mitigate the potential loss of model accuracy in the case of PTQ. In contrast to PTQ, the quantization process is integrated with the training itself, hence making the process “Quantization Aware”.
In QAT, the model architecture is initially modified to maintain both full-precision and quantized versions of elements, which includes weights and activations, thereby creating a dual storage system. During the forward pass of the training process, a simulated or “fake” quantization is introduced to the model, allowing it to experience the effects of quantization while still preserving the precision when calculating gradients, thereby enhancing the model’s robustness to quantization.
I have discussed LLM quantization in detail with various methods in another blog, which you can check here.
The Ultimate Handbook for LLM Quantization
You must have come across LLMs named as DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-1.5B. By the end of this section, these names will make perfect sense.
Even though distillation in neural networks has been around for a long time, one of the most influential examples for Transformer-based language models is the classic DistilBERT** [5] **paper. It builds on a simple intuition. The bigger model we are training from is called the ** teacher model, **and the smaller model that receives the training is called the
Broadly speaking, knowledge transfer in distillation can be achieved in two ways.
KL divergence (Kullback–Leibler divergence)measures how different one probability distribution is from another reference distribution. In LLM distillation, the teacher model provides the reference distribution, typically through its output probabilities (or logits), and the student model is trained to produce a similar distribution. The KL divergence loss penalizes the student when its predicted distribution differs from the teacher’s, encouraging the student to mimic the teacher’s behavior while using a smaller model.
For this method, mainstream libraries like PyTorch have provided out-of-the-box implementations such as torchtune.
HuggingFace also provides something straightforward, but this implements a slightly different variation called *Generalized Knowledge Distillation (GKD) introduced in *On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.
from datasets import load_datasetfrom trl.experimental.distillation import DistillationConfig, DistillationTrainer# 1. Load dataset and format as prompt-only chat messagesdataset = load_dataset("openai/gsm8k", "main", split="train")dataset = dataset.map( lambda x: {"messages": [{"role": "user", "content": x["question"]}]}, remove_columns=dataset.column_names,)# 2. Configure distillationconfig = DistillationConfig( output_dir="results/distill-qwen", num_train_epochs=1, bf16=True, save_strategy="no", # Distillation lmbda=1.0, # fully on-policy (student generates) beta=1.0, # reverse KL # Teacher teacher_model_init_kwargs={"dtype": "bfloat16"},)# 3. Traintrainer = DistillationTrainer( model="Qwen/Qwen2.5-1.5B-Instruct", # student model teacher_model="Qwen/Qwen2.5-7B-Instruct", args=config, train_dataset=dataset,)trainer.train()trainer.save_model()
A major limitation with this method is that it usually requires LLMs that use the same vocabulary, as with the Llama models in this example. This usually means they need to use compatible tokenizers and vocabulary mappings. If the models use different tokenizers or vocabularies, their output distributions are defined over different sets of tokens, so they cannot be compared directly with a standard KL-divergence loss. But there has been research on cross-tokenizer approaches like Universal Logit Distillation that you could explore.
- Distilling through generated data: The student model is trained on a dataset that contains samples generated by the teacher model. In Orca, Microsoft Research [6] found that this becomes more effective when the student model is trained along with reasoning and intermediate explanations behind those samples, helping it acquire more of the teacher’s problem-solving capabilities.
Model pruning involves reducing the LLM size by removing the weights or layers that do not contribute much to the output. It results in a much smaller model that retains somewhat good performance of the original model. Since pruning is not commonly used in LLMs, I am not discussing it in detail. SparseGPT, Wanda, and Wanda++ are some methods I would recommend going through.
Massive LLMs are expensive to use. The main bottleneck for inference is their autoregressive nature, which consumes a lot of GPU bandwidth (memory-bound). What if we can run a smaller, cheaper model to generate text autoregressively and then let our actual larger model just verify the output?
In speculative decoding, the idea is simple. Decoding is slow because it is sequential, while verifying is parallel. So use a small, fast model to guess the next several tokens. Then have the real model verify all those guesses in a single pass, which costs roughly what one token would have cost. The smaller LLM is called the draft model, and the actual LLM is called the target model. Hearing about this the first time, I had the same questions many of you might have: *What exactly is this “verification” process? And if we still have to run the large model, why not just use the large model directly? *Let’s get right to it.
Imagine the input tokens are My, neighbour, heard.
- The speculative decoding algorithm then compares the target model’s probability for each drafted token with the probability assigned by the draft model (verification). If the target model considers a drafted token sufficiently likely, that token is accepted; otherwise, it is rejected.
In our example, the target model strongly agrees with a and dog, but considers driving unlikely. The first two tokens are therefore accepted, while driving is rejected. The target model can then provide the correct continuation, such as bark, giving us “My neighbour heard a dog bark…” without requiring the target model to generate a and dog one token at a time. After bark , the draft model continues generating again, and the same process repeats.
In practice, speculative decoding is not used as it is because it involves two separate models. A more efficient method called self-speculative decoding integrates the drafting capability directly into the target model. Instead of a separate small model, the target model uses a lightweight auxiliary component (such as an EAGLE** head**) to draft multiple tokens in parallel [7]. This approach eliminates the memory and compute overhead of managing a second model while still achieving significant speedups.
Another variation of this is Medusa, which you can read about in the paper Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. Here, multiple extra decoding heads are attached to the main model’s hidden states.
These heads predict the next K tokens simultaneously from the same current state, allowing the model to verify multiple candidates in a single forward pass and speeding up inference by roughly 2x to 3x.
There are some things you may need to note to decide whether speculative decoding works for you.
TPOT(Time per output token)measures how long an LLM takes, on average, to generate each output token after generation has started.Total Generation Time / No of tokens generated
So far, we’ve seen how techniques like quantization, distillation, pruning, and speculative decoding can make LLMs smaller and more affordable. But we’re far from done. There are still some clever ways to carry this even further by altering the foundation: **Attention Mechanisms. **Respecting the length of this article, let’s meet at Part 2. 👁️
[1] NVIDIA, Felix Abecassis, Anjulie Agrusa, et al. (2025).Pretraining Large Language Models with NVFP4
[2] Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer(2022).LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
[3] Chip Huyen (2024).AI Engineering: Building Applications with Foundation Models
[4] Hongyu Wang, Shuming Ma, Li Dong, et al. (2023).BitNet: Scaling 1-bit Transformers for Large Language Models
[5] Victor Sanh, Lysandre Debut, Julien Chaumond, et al. (2019).DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
[6] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, et al. (2023).Orca: Progressive Learning from Complex Explanation Traces of GPT-4
[7] Yuhui Li, Fangyun Wei, et al.(2024).EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
[8] Ashish Abraham (2024).The Ultimate Handbook for LLM Quantization
If not otherwise stated, all images are created by the author.
The Ultimate Guide to LLM Inference Optimization- Part 1 was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.