# Stop Wasting GPU Memory: A Deep Dive Into vLLM’s PagedAttention

> Source: <https://pub.towardsai.net/stop-wasting-gpu-memory-a-deep-dive-into-vllms-pagedattention-a7155f1aa0ec?source=rss----98111c9905da---4>
> Published: 2026-09-03 12:01:02+00:00

Every time you type a prompt into an LLM, a silent battle for GPU memory takes place under the hood.

If you’ve ever hosted an LLM or looked at serving costs, you know that the bottleneck isn’t just raw computing power (FLOPS) — it is **GPU memory bandwidth and capacity**. In fact, serving a single request can be up to 10 times more expensive than a traditional search query.

The chief culprit? The **Key-Value (KV) Cache**.

Today, we are diving deep into how **vLLM** revolutionized LLM serving by taking a page (literally) out of classical operating system design: **PagedAttention**. We will use three visual scenarios to show exactly how it works.

The Problem: Contiguous Memory is a Trap

To generate text autoregressively (one word at a time), an LLM must look at all previous words to calculate where to focus its attention next. To avoid recalculating the Key (K) and Value (V) projection vectors for every past token over and over, these vectors are saved in GPU memory. This is the **KV Cache**.

In traditional serving frameworks (like FasterTransformer or Orca), deep learning engines require tensors to be stored in **contiguous memory**. This means when a request starts, the system must pre-allocate a continuous block of GPU DRAM equal to the **maximum possible sequence length** (e.g., 2048 tokens)

This causes three catastrophic types of memory waste:

As a result, traditional serving systems waste up to **60% to 80%** of their available KV cache memory!

```
Traditional Contiguous Allocation (e.g., Max Length = 16)[ Token 1 ][ Token 2 ][ Token 3 ][ Reserved ][ Reserved ][ Reserved ]... (Wasted DRAM)
```

The breakthrough of PagedAttention is treating GPU DRAM like traditional computer memory.

Instead of forcing a sequence’s KV cache to reside in one massive, continuous block of memory, PagedAttention partitions the cache into **KV Blocks**. Each block contains keys and values for a fixed, small number of tokens (denoted as block size B, with 16 being the optimal default).

These physical blocks are scattered **non-contiguously** throughout the GPU memory. vLLM coordinates this using a **Block Table** — a translation registry that maps contiguous **Logical Blocks** (the model’s view) to scattered **Physical Blocks** (the hardware’s view).

Let’s trace how this executes in the three core generation scenarios:

Scenario 1: Single Sequence Generation (The Autoregressive Loop)

``` php
[Logical Block L0]  ---> Block Table (L0 -> P7) ---> [Physical Block P7 in DRAM][Logical Block L1]  ---> Block Table (L1 -> P3) ---> [Physical Block P3 in DRAM]
```

When you send a prompt, vLLM executes the **Prefill Phase** 7. If your prompt is *“Four score and seven years”* (5 tokens) and block size B=4:

By only allocating what is needed on-demand, memory waste is capped strictly to the unfilled slots of the *very last block*. This slashes memory waste from ~70% down to **less than 4%**

Scenario 2: Parallel Sampling & Copy-on-Write (CoW)

When generating multiple diverse outputs from a single prompt (e.g., generating three code completions for a programming assistant), the prompt is exactly the same.

Traditionally, you would have to duplicate the prompt’s KV cache across multiple contiguous memory blocks.

PagedAttention handles this elegantly:

This allows LLM servers to share up to **30% of their KV memory** effortlessly during parallel generation

Scenario 3: Beam Search (Pruning the Tree)

Beam Search is a highly dynamic decoding method where multiple candidate sentences (”beams”) compete. At each step, only the most probable paths are kept, while low-probability paths are killed off.

In standard systems, this requires frequent, heavy memory copies to synchronize the KV cache of newly selected beams.

With PagedAttention, Beam Search functions like a dynamic process tree:

This block-sharing optimization translates to a massive **37% to 55% memory saving** during beam search workloads.

By squeezing out every ounce of wasted memory fragmentation, vLLM can increase the server’s **effective batch size** (the number of requests processed simultaneously) by up to **4x** on the exact same hardware, leading to a dramatic 2–4× improvement in overall serving throughput.

*To help you see this in action, we’ve built a fully interactive dashboard simulator where you can play with Single Generation, Parallel Sampling, and Beam Search. Try stepping through the simulation to watch block tables update and Copy-on-Write execute in real-time!*

*Refer : *[https://github.com/rahulprasanth487/PagedAttention](https://github.com/rahulprasanth487/PagedAttention)

[Stop Wasting GPU Memory: A Deep Dive Into vLLM’s PagedAttention](https://pub.towardsai.net/stop-wasting-gpu-memory-a-deep-dive-into-vllms-pagedattention-a7155f1aa0ec) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.