Containers are ephemeral. When a Pod crashes or Kubernetes moves it, everything on its local disk is gone. For a web server that is fine. For AI it is costly: vector indices take time to rebuild, and model weights are many gigabytes that you do not want to download again every restart.
In plain terms: A container's disk is a whiteboard that gets wiped whenever the room is reassigned. A persistent volume is a filing cabinet in the corridor that every new room can open.
Diagram: A container's local filesystem is discarded when its Pod crashes or is rescheduled. Vector indices and multi-gigabyte model weights must outlive the Pod, so they need persistent volumes. See the animated version.
Kubernetes separates the storage from the application with two resources.
Diagram: A PersistentVolume is a piece of real storage in the cluster. A PersistentVolumeClaim is the application's request for storage. The Pod references the claim, which binds to a matching volume, or a StorageClass can create one on demand. See the animated version.
The access mode decides how Pods may use the volume.
Check that your storage backend actually supports the mode you ask for. Shared read-only mounts typically come from NFS or a similar network filesystem, while plain block disks are usually single-node.
Diagram: ReadWriteOnce mounts a volume read-write on a single node, which suits a vector database that must not have uncoordinated writers. ReadOnlyMany mounts read-only on many nodes at once, so one copy of the model weights can serve many inference Pods. The storage backend has to support the mode you request. See the animated version.
For a vector database, a standard Deployment is usually the wrong tool. A StatefulSet gives each Pod a stable name (db-0, db-1), starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That keeps replication between the database Pods consistent through cluster restarts.
Diagram: A StatefulSet gives each Pod a stable name such as db-0 and db-1, starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That is why stateful databases use StatefulSets instead of Deployments. See the animated version.
Here is an example. It assumes your cluster has a default StorageClass, and the Qdrant image and sizes are illustrative.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-weights
spec:
accessModes: [ReadOnlyMany]
resources:
requests: { storage: 20Gi }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: qdrant
spec:
serviceName: qdrant
replicas: 1
selector:
matchLabels: { app: qdrant }
template:
metadata:
labels: { app: qdrant }
spec:
containers:
- name: qdrant
image: qdrant/qdrant:v1.9.0 # pin a version you have tested
volumeMounts:
- { name: data, mountPath: /qdrant/storage }
volumeClaimTemplates: # one PVC is created per Pod: data-qdrant-0, ...
- metadata: { name: data }
spec:
accessModes: [ReadWriteOnce]
resources:
requests: { storage: 50Gi }
Done well, this eliminates repeated weight downloads, preserves your database indices and protects the system's memory across cluster updates.
Day 21: High-performance networking: ingress and egress for AI clusters.
Originally published at https://sureshpallapothu.in/blog/day-20-kubernetes-storage, where this post includes animated diagrams.