# Scaling LLM Inference: Multi-Node KV Cache Offloading with GKE & Managed Lustre
> Source:
> Published: 2026-07-01 07:00:00+00:00
*Significant contributors to this article include ***Sneha Aradhey**, Software Engineer, Google Kubernetes Engine, and **Michael MacDonald**, Sr Software Engineer, Google Cloud Managed Lustre.
Enterprise production environments are shifting to distributed, multi-node architectures to serve long-context window lengths and agentic AI. As these workloads scale, KVCaches often outgrow local CPU RAM and host SSD cache tiers.
To handle this, some setups attempt to pool node-local storage into a distributed layer (such as multi-node pooled NVMe arrays). Pooling SSDs aggregates raw capacity and often leverages spare local drives, presenting clear advantages. However, there are some limitations: the approach requires the compute cluster to manage its own complex data distribution and cross-node replication.
An alternative is to offload the attention state to a dedicated, high-performance external parallel filesystem. We utilize **Google Cloud Managed Lustre with the llm-d offloading stack** as a cluster-wide decentralized attention cache tier, bypassing host-level capacity limits and eliminating the networking overhead of managing local pooled drives.
With this approach, we achieve efficiency at scale:
**Google Cloud Managed Lustre enables over 50% TCO savings and reduces GPU-hour requirements for Llama-3.3-70B inference on a six-node A3 Mega cluster by nearly 60%. These gains are realized by offloading shared, prefilled KV caches to Lustre’s high-performance tier with a 95% cache hit rate.**
#### Benchmark Configuration
**Model:** Llama-3.3-70B
**Context Dynamics:** Prompt length of 50,000 tokens, input question length of 256 tokens, and output length of 512 tokens.
#### Extension of Lustre KV Cache solution with CPU RAM offload
The Managed Lustre KV Cache offload architecture can be extended via integration of offload to CPU RAM. This hybrid approach [significantly improves performance compared to CPU offload only](https://github.com/llm-d/llm-d/tree/main/guides/tiered-prefix-cache#llm-d-fs-connector--lustre), delivering approximately 40% improvement in Time to First Token (TTFT) and a 30% reduction in end-to-end latency, for Llama-3.3-70B inference.
### User Guide
#### Architectural Components
**GKE GPU Nodes:** Dedicated accelerator resources provisioned exclusively for high-throughput model execution and tensor-parallel operations.
**Managed Lustre:** A shared, high-bandwidth parallel filesystem acting as a centralized external tier that caches prefilled attention states to eliminate redundant prefill computation.
**PVC Evictor****:** A scalable, distributed garbage collection service that tracks file access patterns and automatically removes Least-Recently-Used (LRU) cache chunks to maintain healthy storage headroom.
#### Target Models
This guide provides two distinct, validated tracks for deployment depending on your model preference:
**Qwen Series:** `Qwen/Qwen3.5-35B-A3B`
**Gemma 4 Architecture:** `google/gemma-4-31B-it`
#### Architectural Diagram
#### Before You Begin
Before starting this deployment, ensure your Google Cloud project is properly configured:
**Quota:** Verify you have sufficient quota for the selected accelerators in your chosen region, as well as adequate general CPU, memory, and Managed Lustre quotas.
**Validate Required IAM Permissions for Managed Lustre**
**Prepare your Environment to Connect to Managed Lustre:** Complete the “[Before You Begin](https://docs.cloud.google.com/managed-lustre/docs/lustre-csi-driver-new-volume#before_you_begin)” steps to enable APIs, set up environment variables, and set up your VPC.
**GKE Version:** The [Managed Lustre CSI driver](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/managed-lustre) is supported on GKE versions **1.33 or later**. For the best experience and default port (988) usage, GKE version **1.33.2-gke.4780000 or later** is recommended.
#### Overview of Required Steps
- Create the GKE Cluster
- Create the GPU Compute node pool
- Provision Lustre storage
- Deploy vLLM Serving Engine with Lustre
- Deploy the PVC Evictor
- Clean Up
#### 1. Create the GKE Cluster
Create a rapid-channel GKE cluster with Workload Identity and all necessary CSI storage add-ons enabled (Lustre, GCSFuse and Persistent Disk).
- code_block
- )])]>
#### 2. Create the GPU Compute Node Pool
Provision an GPU VM node pool ( e.g. `a3-megagpu-4g`
, `a4-highgpu-4g`
, etc.).
- code_block
- )])]>
#### 3. Provision Lustre Storage (Auto-provisioned)
Before deploying vLLM, you need to provision the Lustre storage. We use an auto-provisioned Lustre instance via a `StorageClass`
and a `PersistentVolumeClaim`
(PVC).
Create a file named `lustre-pvc.yaml`
with the following content:
- code_block
- " # See options below.\r\n network: ""\r\n---\r\napiVersion: v1\r\nkind: PersistentVolumeClaim\r\nmetadata:\r\n name: lustre-pvc\r\nspec:\r\n accessModes:\r\n - ReadWriteMany\r\n resources:\r\n requests:\r\n storage: # Range from 9000Gi to 84016000Gi, increments and ranges are Lustre tier-dependent.\r\n storageClassName: lustre-class'), ('language', ''), ('caption', )])]>
Notes: Performance tier options are “125”, “250”, “500”, and “1000”. Per-tier capacity ranges and increments can be found [here](https://docs.cloud.google.com/managed-lustre/docs/performance-tiers).
Apply this manifest to provision the Lustre instance and observe provisioning:
- code_block
- )])]>
#### 4. Deploy vLLM Serving Engine with Lustre
**Step 4a: Create the Hugging Face Access Secret**
Before submitting the deployment manifest, you must provision your Hugging Face API [token](https://huggingface.co/docs/hub/en/security-tokens) as a secure secret within the cluster.
Run the following command, replacing `` with your token:
- code_block
- " \\\r\n --namespace=default'), ('language', ''), ('caption', )])]>
**Step 4b: Create the vLLM Deployment Manifest**
This complete Kubernetes manifest deploys the vLLM engine, configures the `llmd-fs-connector`
for high-performance KV-caching, and mounts your parallel Lustre storage (`lustre-pvc`
).
Common Manifest (Choose between Qwen3.5 or gemma-4)
Replace example values between <> with appropriate values for your environment.
- code_block
- \r\n runAsUser: \r\n volumes:\r\n - name: lustre-storage\r\n persistentVolumeClaim:\r\n claimName: lustre-pvc\r\n - name: shm\r\n emptyDir:\r\n medium: Memory\r\n sizeLimit: "200Gi"\r\n containers:\r\n - name: vllm-storage\r\n image: vllm/vllm-openai:v0.23.0-cu129\r\n volumeMounts:\r\n - mountPath: /mnt/files-storage\r\n name: lustre-storage\r\n command:\r\n - "/bin/bash"\r\n args:\r\n - "-c"\r\n - |\r\n set -x\r\n export USER=vllm\r\n export LOGNAME=vllm\r\n pip install --user msgpack\r\n pip install \'llmd-fs-connector==0.23\' --extra-index-url https://llm-d.github.io/llm-d-kv-cache/simple/\r\n \r\n vllm serve \\ # google/gemma-4-31B-it OR Qwen/Qwen3.5-35B-A3B\r\n --download-dir /model/models \\\r\n --load-format auto \\\r\n --kv-transfer-config \'{\r\n "kv_connector": "MultiConnector",\r\n "kv_role": "kv_both",\r\n "kv_connector_extra_config": {\r\n "connectors": [\r\n {\r\n "kv_connector": "OffloadingConnector",\r\n "kv_role": "kv_both",\r\n "kv_connector_extra_config": {\r\n "cpu_bytes_to_use": 64424509440,\r\n "lazy_offload": true\r\n }\r\n },\r\n {\r\n "kv_connector": "OffloadingConnector",\r\n "kv_role": "kv_both",\r\n "kv_connector_extra_config": {\r\n "spec_name": "SharedStorageOffloadingSpec",\r\n "spec_module_path": "llmd_fs_backend.spec",\r\n "shared_storage_path": "/mnt/files-storage/llmd-kv-cache/",\r\n "threads_per_gpu": 32,\r\n "block_size": # 256 for gemma or 528 for Qwen3.5\r\n }\r\n }\r\n ]\r\n }\r\n }\' \\\r\n --distributed_executor_backend "mp" \\\r\n --port 8000 \\\r\n --max_num_batched_tokens 16384 \\\r\n --enable-chunked-prefill \\\r\n --max-model-len 32000 \\\r\n --gpu-memory-utilization 0.92 \\\r\n --tensor-parallel-size "4" \\\r\n --prefix-caching-hash-algo sha256_cbor \\\r\n --enable_prefix_caching \\\r\n --enforce-eager \\\r\n --no-disable-hybrid-kv-cache-manager\r\n env:\r\n - name: HUGGING_FACE_HUB_TOKEN\r\n valueFrom:\r\n secretKeyRef:\r\n name: hf-token-secret\r\n key: token\r\n # ... probes ...\r\n resources:\r\n requests:\r\n nvidia.com/gpu: "4"\r\n limits:\r\n nvidia.com/gpu: "4"'), ('language', ''), ('caption', )])]>
Note: Qwen-3.5 specifically requires a block size of `528`
to avoid fragmentation, while Gemma 4 functions perfectly with the default `256`
.
**Step 4c: Apply and Verify Deployment**
To apply this manifest to your cluster, run:
- code_block
- )])]>
**Step 4d: Track Model Download Status**
Because large models can take some time to download on first boot, track the initialization logs directly by streaming the container logs:
Bash
- code_block
- )])]>
#### 5. Deploy the PVC Evictor
##### PVC Evictor Overview
**Architecture & Role**
The `llmd_fs_backend`
connector offloads KV-cache blocks to Lustre but does not natively delete old cache files. Over time, the cache will fill the shared filesystem. The **PVC Evictor** acts as an external garbage collector that continuously monitors disk usage and evicts least-recently-used (LRU) files to maintain healthy storage headroom.
**Scaling & Sharding**
The PVC Evictor supports sharding and can be scaled to multiple replicas to match the capacity and performance of your Lustre instance. As a rule of thumb, you should deploy **1 evictor replica for each 72 TB of Lustre capacity** to distribute the eviction load effectively without overwhelming the metadata servers.
For large-scale deployments, the evictor can be configured to run with multiple shards. When running in multi-replica mode, the workload is partitioned across pods, with each pod managing a specific shard of the cache namespace. This prevents redundant metadata scans and race conditions.
**High-Performance Resource Requirements**
Running the evictor at high scale (e.g., with 16 parallel crawler processes) requires significant CPU and memory resources to handle the rapid scanning and queue management of millions of files. Ensure that the pods are provisioned with sufficient resources (e.g., 12 CPU requests and 8Gi Memory requests) and scheduled on appropriate node types (such as `c4-standard-16`
).
**PVC Evictor Deployment Steps**
The PVC Evictor is deployed via Helm using the chart located in `kv_connectors/pvc_evictor/helm`
.
**Step 5a: Create a Dedicated Node Pool for the Evictor**
Running the evictor at high scale requires significant CPU and memory. First, create a dedicated node pool using a high-performance machine type (such as c4-standard-16) to accommodate the 12 CPU and 8Gi memory requests needed per pod.
- code_block
- )])]>
**Step 5b: Install via Helm (High-Performance Configuration)**
Deploy a scaled, high-performance evictor pool with 2 replicas to monitor lustre-pvc. This configuration uses 16 crawler processes per pod to handle massive file namespaces.
**Note on Security Contexts**: To allow the evictor pod to delete files created by vLLM, it must run with matching security context IDs. Ensure the placeholders ``
and ``
exactly match the non-root values used in the `securityContext`
of your vLLM deployment to ensure shared POSIX file permissions.
- code_block
- \\\r\n --set securityContext.container.runAsUser= \\\r\n --set resources.requests.cpu=12 \\\r\n --set resources.requests.memory=8Gi \\\r\n --set resources.limits.cpu=15 \\\r\n --set resources.limits.memory=16Gi \\\r\n --set nodeSelector."cloud\\.google\\.com/gke-nodepool"=evictor-pool \\\r\n --set securityContext.pod.seLinuxOptions.level="s0:c0\\,c1"'), ('language', ''), ('caption', )])]>
#### Critical Parameters Explained:
`replicaCount=2`
: Deploys 2 evictor pods. The Helm chart automatically configures sharding (`totalShards=2`
) when multiple replicas are used.
`config.numCrawlerProcesses=16`
: Runs 16 parallel crawler threads per pod to scan the filesystem rapidly.
`config.deletionBatchSize=5000`
: Deletes files in batches of 5000 to reduce metadata overhead.
`config.fileQueueMinSize`
& `config.fileQueueMaxsize`
: Configures large memory queues (1M min, 2M max) to buffer files for deletion, matching the high crawler throughput.
`config.fileAccessTimeThresholdMinutes=10`
: Aggressively evicts files that haven't been accessed in the last 10 minutes when the cleanup threshold is triggered.
`securityContext.container.runAsNonRoot=false`
: Required if the evictor needs root-like permissions to manage/delete files across different user ownerships on the shared storage.
`resources.requests`
& `limits`
: Allocates 12-15 CPUs and 8-16Gi of memory per pod to ensure the high number of crawler processes do not get CPU-throttled or run Out-Of-Memory (OOM).
**Step 5c: Verify and Monitor**
- code_block
- )])]>
#### Step 6: Clean Up
Because this deployment provisions significant and high-cost hardware, be sure to clean up your environment when you are done to avoid unnecessary charges.
Bash
- code_block
- )])]>
### Appendix: Reference Configuration for Llama-3.3-70B Benchmark
The following configuration is a representation of the deployment manifest used to generate the Llama-3.3-70B benchmark results referenced in this post. It is provided for completeness and transparency.
Note: This configuration utilizes an earlier iteration of the software stack (vLLM v0.15.0) and specific infrastructure flags that were active in the benchmarking environment at the time the data was collected.
- code_block
- )])]>