Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices

A developer published a walkthrough of using Kubernetes persistent storage to keep AI workloads' state alive across Pod restarts, arguing that ephemeral container disks are costly for vector indices and multi-gigabyte model weights. The post explains PersistentVolumes, PersistentVolumeClaims and StorageClasses, and details how access modes map to use cases: ReadWriteOnce for a vector database that must avoid uncoordinated writers, and ReadOnlyMany for a single copy of model weights serving many inference Pods. It recommends StatefulSets over Deployments for vector databases, since stable Pod names and per-Pod volumes that follow restarts keep replication consistent, illustrated with a Qdrant manifest using volumeClaimTemplates.

Containers are ephemeral . When a Pod crashes or Kubernetes moves it, everything on its local disk is gone. For a web server that is fine. For AI it is costly: vector indices take time to rebuild, and model weights are many gigabytes that you do not want to download again every restart. In plain terms: A container's disk is a whiteboard that gets wiped whenever the room is reassigned. A persistent volume is a filing cabinet in the corridor that every new room can open. Diagram: A container's local filesystem is discarded when its Pod crashes or is rescheduled. Vector indices and multi-gigabyte model weights must outlive the Pod, so they need persistent volumes. See the animated version https://sureshpallapothu.in/blog/day-20-kubernetes-storage . Kubernetes separates the storage from the application with two resources. Diagram: A PersistentVolume is a piece of real storage in the cluster. A PersistentVolumeClaim is the application's request for storage. The Pod references the claim, which binds to a matching volume, or a StorageClass can create one on demand. See the animated version https://sureshpallapothu.in/blog/day-20-kubernetes-storage . The access mode decides how Pods may use the volume. Check that your storage backend actually supports the mode you ask for. Shared read-only mounts typically come from NFS or a similar network filesystem, while plain block disks are usually single-node. Diagram: ReadWriteOnce mounts a volume read-write on a single node, which suits a vector database that must not have uncoordinated writers. ReadOnlyMany mounts read-only on many nodes at once, so one copy of the model weights can serve many inference Pods. The storage backend has to support the mode you request. See the animated version https://sureshpallapothu.in/blog/day-20-kubernetes-storage . For a vector database, a standard Deployment is usually the wrong tool. A StatefulSet gives each Pod a stable name db-0 , db-1 , starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That keeps replication between the database Pods consistent through cluster restarts. Diagram: A StatefulSet gives each Pod a stable name such as db-0 and db-1, starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That is why stateful databases use StatefulSets instead of Deployments. See the animated version https://sureshpallapothu.in/blog/day-20-kubernetes-storage . Here is an example. It assumes your cluster has a default StorageClass, and the Qdrant image and sizes are illustrative. 1. Shared, read-only model weights needs a backend that supports ReadOnlyMany apiVersion: v1 kind: PersistentVolumeClaim metadata: name: model-weights spec: accessModes: ReadOnlyMany resources: requests: { storage: 20Gi } --- 2. A vector database with its own volume per Pod apiVersion: apps/v1 kind: StatefulSet metadata: name: qdrant spec: serviceName: qdrant replicas: 1 selector: matchLabels: { app: qdrant } template: metadata: labels: { app: qdrant } spec: containers: - name: qdrant image: qdrant/qdrant:v1.9.0 pin a version you have tested volumeMounts: - { name: data, mountPath: /qdrant/storage } volumeClaimTemplates: one PVC is created per Pod: data-qdrant-0, ... - metadata: { name: data } spec: accessModes: ReadWriteOnce resources: requests: { storage: 50Gi } Done well, this eliminates repeated weight downloads, preserves your database indices and protects the system's memory across cluster updates. Day 21: High-performance networking: ingress and egress for AI clusters. Originally published at https://sureshpallapothu.in/blog/day-20-kubernetes-storage https://sureshpallapothu.in/blog/day-20-kubernetes-storage , where this post includes animated diagrams.