{"slug": "day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector", "title": "Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices", "summary": "A developer published a walkthrough of using Kubernetes persistent storage to keep AI workloads' state alive across Pod restarts, arguing that ephemeral container disks are costly for vector indices and multi-gigabyte model weights. The post explains PersistentVolumes, PersistentVolumeClaims and StorageClasses, and details how access modes map to use cases: ReadWriteOnce for a vector database that must avoid uncoordinated writers, and ReadOnlyMany for a single copy of model weights serving many inference Pods. It recommends StatefulSets over Deployments for vector databases, since stable Pod names and per-Pod volumes that follow restarts keep replication consistent, illustrated with a Qdrant manifest using volumeClaimTemplates.", "body_md": "Containers are **ephemeral**. When a Pod crashes or Kubernetes moves it, everything on its local disk is gone. For a web server that is fine. For AI it is costly: vector indices take time to rebuild, and model weights are many gigabytes that you do not want to download again every restart.\n\n**In plain terms:** A container's disk is a whiteboard that gets wiped whenever the room is reassigned. A persistent volume is a filing cabinet in the corridor that every new room can open.\n\n**Diagram:** A container's local filesystem is discarded when its Pod crashes or is rescheduled. Vector indices and multi-gigabyte model weights must outlive the Pod, so they need persistent volumes. [See the animated version](https://sureshpallapothu.in/blog/day-20-kubernetes-storage).\n\nKubernetes separates the storage from the application with two resources.\n\n**Diagram:** A PersistentVolume is a piece of real storage in the cluster. A PersistentVolumeClaim is the application's request for storage. The Pod references the claim, which binds to a matching volume, or a StorageClass can create one on demand. [See the animated version](https://sureshpallapothu.in/blog/day-20-kubernetes-storage).\n\nThe access mode decides how Pods may use the volume.\n\nCheck that your storage backend actually supports the mode you ask for. Shared read-only mounts typically come from NFS or a similar network filesystem, while plain block disks are usually single-node.\n\n**Diagram:** ReadWriteOnce mounts a volume read-write on a single node, which suits a vector database that must not have uncoordinated writers. ReadOnlyMany mounts read-only on many nodes at once, so one copy of the model weights can serve many inference Pods. The storage backend has to support the mode you request. [See the animated version](https://sureshpallapothu.in/blog/day-20-kubernetes-storage).\n\nFor a vector database, a standard Deployment is usually the wrong tool. A **StatefulSet** gives each Pod a stable name (`db-0`, `db-1`), starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That keeps replication between the database Pods consistent through cluster restarts.\n\n**Diagram:** A StatefulSet gives each Pod a stable name such as db-0 and db-1, starts them in order, and binds each Pod to its own persistent volume that follows it across restarts. That is why stateful databases use StatefulSets instead of Deployments. [See the animated version](https://sureshpallapothu.in/blog/day-20-kubernetes-storage).\n\nHere is an example. It assumes your cluster has a default StorageClass, and the Qdrant image and sizes are illustrative.\n\n```\n# 1. Shared, read-only model weights (needs a backend that supports ReadOnlyMany)\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: model-weights\nspec:\n  accessModes: [ReadOnlyMany]\n  resources:\n    requests: { storage: 20Gi }\n---\n# 2. A vector database with its own volume per Pod\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: qdrant\nspec:\n  serviceName: qdrant\n  replicas: 1\n  selector:\n    matchLabels: { app: qdrant }\n  template:\n    metadata:\n      labels: { app: qdrant }\n    spec:\n      containers:\n        - name: qdrant\n          image: qdrant/qdrant:v1.9.0        # pin a version you have tested\n          volumeMounts:\n            - { name: data, mountPath: /qdrant/storage }\n  volumeClaimTemplates:                       # one PVC is created per Pod: data-qdrant-0, ...\n    - metadata: { name: data }\n      spec:\n        accessModes: [ReadWriteOnce]\n        resources:\n          requests: { storage: 50Gi }\n```\n\nDone well, this eliminates repeated weight downloads, preserves your database indices and protects the system's memory across cluster updates.\n\n**Day 21:** High-performance networking: ingress and egress for AI clusters.\n\n*Originally published at [https://sureshpallapothu.in/blog/day-20-kubernetes-storage](https://sureshpallapothu.in/blog/day-20-kubernetes-storage), where this post includes animated diagrams.*", "url": "https://wpnews.pro/news/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector", "canonical_source": "https://dev.to/suresh_kumar_de3920bedd1c/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector-indices-3cef", "published_at": "2026-10-20 18:05:45+00:00", "updated_at": "2026-10-20 18:18:33.005307+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools"], "entities": ["Kubernetes", "Qdrant"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector", "markdown": "https://wpnews.pro/news/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector.md", "text": "https://wpnews.pro/news/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector.txt", "jsonld": "https://wpnews.pro/news/day-20-persistent-storage-in-kubernetes-managing-state-weights-and-vector.jsonld"}}