{"slug": "evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf", "title": "Evaluating Kubernetes Pod Checkpoint/Restore on AWS (CRIU, PLEG, eBPF)", "summary": "A study by Musa Khasmammadov evaluating checkpoint/restore-based pod migration on a 3-node AWS EC2 cluster in eu-central-1 found that decoupled CRIU (v4.3) migration preserved 100% of in-memory heap state while adding a median cutover blackout of 1064.5 ms versus 1019.0 ms for native cold reschedule, a difference of 45.5 ms (+4.5%) across 36 bidirectional migrations and 20 cold restarts. A two-sided Mann-Whitney U test found the cutover penalty was not statistically significant, and the pipeline's timing was dominated by the Kubelet's 1-second PLEG relist cycle. The testbed coordinated CRIU, the Kubelet Checkpoint API (KEP-2008), Buildah (v1.37), CRI-O (v1.31) and Cilium eBPF (v1.16) to migrate stateful workloads such as machine learning embedding caches without application refactoring.", "body_md": "# Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes: An Empirical Performance and Bottleneck Study\n\nThis repository contains the complete experimental testbed, deployment automation, Kubernetes operator, benchmark harnesses, and LaTeX source code for the research paper:\n\n**Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes: An Empirical Performance and Bottleneck Study**\n\n*Musa Khasmammadov*\n\n[📄 Read the 5-Page IEEE-Formatted PDF](https://github.com/musaxasmammedov7/final-research/blob/main/paper/Stateful_Container_Live_Migration_IEEE_NativeTeX.pdf) | [📝 View TeX Source](https://github.com/musaxasmammedov7/final-research/blob/main/paper/paper_stateful_live_migration.tex)\n\nModern cloud architectures typically assume container workloads are stateless and ephemeral. Critical production workloads (e.g., machine learning embedding caches, in-memory stream processors, stateful gaming engines), however, maintain gigabytes of volatile heap state that cannot be persisted to external databases without prohibitive latency or intrusive application refactoring.\n\nWe designed and empirically evaluated an end-to-end, snapshot-consistent pod migration pipeline coordinating **CRIU (v4.3)**, the **Kubelet Checkpoint API (KEP-2008)**, **Buildah (v1.37)**, **CRI-O (v1.31)**, and **Cilium eBPF (v1.16)** on a dedicated multi-node **AWS EC2** cluster (`eu-central-1`).\n\n```\n +----------------------------------------------------------------------------------------------------+\n |                                  AWS EC2 3-Node Topology                                           |\n |                                                                                                    |\n |  +-----------------------+     +-----------------------+     +----------------------------------+  |\n |  |      k8s-master       |     |     k8s-worker-1      |     |           k8s-worker-2           |  |\n |  |     (c7i-flex.large)  |     |     (m7i-flex.large)  |     |         (m7i-flex.large)         |  |\n |  |  - Control Plane      |     |  - CRI-O 1.31 + CRIU  |     |  - CRI-O 1.31 + CRIU             |  |\n |  |  - OCI Registry :5000 |     |  - Buildah 1.37       |     |  - Buildah 1.37                  |  |\n |  |  - Cilium Ingress     |     |  - Source Pod (State) | === |  - Restored Pod (Bit-Exact RAM)  |  |\n |  +-----------------------+     +-----------------------+     +----------------------------------+  |\n |              |                             |                                    |                  |\n |              +-----------------------------+------------------------------------+                  |\n |                               Private VPC Subnet (10.10.1.0/24)                                    |\n |                             Cilium eBPF Host Routing (No iptables)                                 |\n +----------------------------------------------------------------------------------------------------+\n```\n\nExternal 50 Hz HTTP probing (20 ms interval) across 36 bidirectional migrations and 20 cold restarts reveals:\n\n| Metric | Decoupled CRIU Migration ( | Native Cold Reschedule ( | Delta / Statistical Test | \n|---|---|---|---|\n| **Median Cutover Blackout (** | **1064.5 ms** | **1019.0 ms** | **+45.5 ms (+4.5%)** | \n| **Mean Cutover Blackout**  |  |  | Difference: | \n| **Mann-Whitney U Test** |  | — | (Null hypothesis not rejected at | \n| **Welch's Two-Sample t-test** |  | — |  | \n| **RAM State Preservation** | **100% Intact** (Heap preserved) | **0% (Reset to zero)** | Bit-exact state continuity | \n| **Kubelet PLEG Synchronization** |  |  | Dominated by 1 s PLEG relist cycle ( | \n\nNote\n\nThe two-sided Mann-Whitney U test confirms that decoupled process heap restoration **does not provide evidence of a large cutover penalty** over cold pod initialization (\n\n| Dimension | Mode A: Decoupled Migration (Default) | Mode B: Synchronous Stop-and-Copy | Native Cold Reschedule | \n|---|---|---|---|\n| **Source Pod Lifecycle** | Continues serving during packaging/push | Frozen/killed *before* packaging/push | Terminated immediately | \n| **Client Cutover Blackout (** | **1064.5 ms** (p50) /**797 ms** (best) | **1878 ms** | 1019.0 ms (p50) | \n| **State Drift (RPO)** | Bounded drift: | **0.000 ms (Zero drift)** | Total loss (100% loss) | \n| **Best Suited Workloads** | Monotonic counters, caches, gaming lobbies | Bit-exact financial ledgers, ACID logs | Ephemeral/stateless services | \n\nWe evaluated numerical pseudo-random float64 heaps up to 1024 MB (1 GB) with cross-node cryptographic validation:\n\n| Allocated Heap | OCI Layer Size |  |  | `-D` ) | RPO (gzip / `-D` ) | Worst `-D` ) | % 120s Spot SLA |  | SHA-256 Digest Status | \n|---|---|---|---|---|---|---|---|---|---|\n| **26 MB** (p50) | 5.16 MiB | 0.144 s | 0.371 s | 0.261 s | 0.67 s / 0.56 s | 5.70 s | 4.8% | 1.066 s | Verified ( `Counter` ) | \n| **64 MB** | 69.11 MiB | 0.216 s | 0.761 s | 0.361 s | 1.38 s / 0.98 s | 6.19 s | 5.2% | 0.951 s | **100% Match** (`7785bfc7` ) | \n| **128 MB** | 133.11 MiB | 0.258 s | 1.082 s | 0.528 s | 2.01 s / 1.46 s | 6.71 s | 5.6% | 1.695 s | **100% Match** (`cce52f48` ) | \n| **256 MB** | 261.13 MiB | 0.375 s | 5.796 s | **1.473 s** ( | 7.48 s / 3.16 s | 8.53 s | 7.1% | 2.108 s | **100% Match** (`4b1b1359` ) | \n| **512 MB** | 517.15 MiB | 0.760 s | 16.120 s | **4.210 s** ( | 19.39 s / 7.48 s | 13.24 s | 11.0% | 7.807 s | **100% Match** (`ab028a06` ) | \n| **1024 MB (1 GB)** | 1029.21 MiB | **7.464 s** | 32.060 s | **13.743 s** ( | 42.63 s / **24.31 s** | **36.78 s** | **30.6%** | 17.328 s | **100% Match** (`c0da7edf` ) | \n\n1. \n**AWS Spot SLA Guarantee:** Under worst-case Spot preemption detection ($T_{\\text{detect}}^{\\text{worst}} = 5.0\\text{ s}$ ) with Buildah`--disable-compression` , total source evacuation completes in**36.78 s** (only**30.6%** of the 120 s AWS Spot preemption deadline).\n2. \n**Buildah Gzip Bypass Optimization:** Disabling single-threaded gzip serialization (`--disable-compression` /`-D` ) slashes 1 GB packaging from 32.06 s to**13.74 s** (saving**18.32 s** , a$2.3\\times$ speedup).\n3. \n**Linux Kernel Dirty-Page Writeback:** From 26 MB to 512 MB,$T_{\\text{freeze}}$ remains sub-second ($144\\text{ ms} \\to 760\\text{ ms}$ ) as writes are buffered in page cache. At 1 GB, uncommitted dirty memory hits 783.6 MB (`vm.dirty_background_ratio = 10%` ), triggering kernel flusher serialization to EBS gp3 and increasing$T_{\\text{freeze}}$ to 7.464 s.\n4. \n**Restoration Phase Breakdown:** Target-side restoration ($T_{\\text{restore}} = 17.328\\text{ s}$ ) breaks down into four phases:\n  - Registry layer pull over VPC ($\\approx 8.5\\text{ s}$ , matching$\\approx 1\\text{ Gbps}$ baseline instance bandwidth).\n  - CRI-O storage extraction and unpacking ($\\approx 4.8\\text{ s}$ ).\n  - \n`runc/criu restore` memory mapping and process unfreezing ($\\approx 3.2\\text{ s}$ ).\n  - Kubelet PLEG status synchronization ($\\approx 0.8\\text{ s}$ ).\n5. Registry layer pull over VPC (\n6. \n**Why Migrate?** Reconstructing a 1 GB state snapshot from cloud object storage (Amazon S3) over assumed multi-tenant cloud bandwidth (300–800 Mbps) requires an estimated**11.2–29.8 s** recovery window. Checkpointing eliminates external database persistence tiers and application-level serialization logic, with cutover becoming advantageous over cold rebuild from$\\approx 256\\text{ MB}$ (2.10 s vs. 3.5–7.5 s).\n\nWe deployed a two-container test pod (`stream-server` listening on `127.0.0.1:8090` + `envoy-sidecar` on `0.0.0.0:15006` forwarding over loopback TCP).\n\n- \n**Finding:** KEP-2008 operates at**single-container granularity** . Checkpointing`stream-server` and restoring it on the target node isolated it from the co-scheduled sidecar ($T_{\\text{blackout}} = \\infty$ ), because CRIU's`tcp-close` closes active loopback sockets.\n- \n**Architectural Recommendation:** Single-container checkpointing severs loopback sidecar peering; node-level routing architectures (such as**Cilium eBPF host-routing** or Kubernetes ClusterIP proxies) decouple L4/L7 service redirection from pod network namespaces, providing a resilient foundation.\n\n```\n.\n├── README.md                                    # Project documentation & benchmark overview\n├── ansible/                                     # Infrastructure automation playbooks\n│   ├── ansible.cfg\n│   └── site.yml                                 # Bootstrap Linux 6.8+, CRI-O 1.31, CRIU v4.3, Buildah\n├── manifests/                                   # Kubernetes CRDs, Operator, and Workloads\n│   ├── 01-cert-manager-issuer.yaml             # Let's Encrypt TLS ClusterIssuer\n│   ├── 02-google-online-boutique.yaml          # 11-microservice background benchmark workload\n│   ├── 03-live-migration-stream-demo.yaml      # Stateful in-memory heap benchmark pod\n│   ├── 03a-live-stream-template.yaml           # Checkpoint restore pod template\n│   ├── 04-cilium-ingress-https.yaml            # Cilium Service Mesh Ingress configuration\n│   └── 05-live-migration-operator.yaml         # PodLiveMigration CRD & autonomous DaemonSet controller\n├── paper/                                       # Academic IEEE Manuscript\n│   ├── Stateful_Container_Live_Migration_IEEE_NativeTeX.pdf # Compiled 5-page IEEE PDF\n│   └── paper_stateful_live_migration.tex        # Native LaTeX source (IEEEtran, TikZ, PGFPlots)\n├── scripts/                                     # Benchmark harnesses and empirical test tools\n│   ├── benchmark_migration.py                   # 50 Hz high-frequency prober & N-run benchmark suite\n│   └── run_mode_b_cluster.py                    # Mode B (Stop-and-Copy) synchronous test harness\n└── terraform/                                   # AWS EC2 multi-node cluster provisioning\n    ├── main.tf                                  # VPC, security groups, 1 master + 2 workers\n    ├── outputs.tf                               # Cluster public/private IPs\n    └── variables.tf                             # Instance types (c7i-flex.large, m7i-flex.large)\ncd terraform\nterraform init\nterraform apply -auto-approve\ncd ../ansible\nansible-playbook site.yml\nkubectl apply -f ../manifests/01-cert-manager-issuer.yaml\nkubectl apply -f ../manifests/02-google-online-boutique.yaml\nkubectl apply -f ../manifests/03-live-migration-stream-demo.yaml\nkubectl apply -f ../manifests/04-cilium-ingress-https.yaml\nkubectl apply -f ../manifests/05-live-migration-operator.yaml\n```\n\nTo execute an automated statistical run with high-frequency HTTP probing:\n\n```\nssh -i terraform/k8s-research-key.pem ubuntu@<MASTER_PUBLIC_IP> \\\n  \"python3 /home/ubuntu/scripts/benchmark_migration.py --runs 36\"\n```\n\nThe paper uses native `IEEEtran` and compiles cleanly with `tectonic` or `xelatex`:\n\n```\ntectonic paper/paper_stateful_live_migration.tex --outdir paper/\n```\n\n1. **Kubelet Checkpoint API:**[Kubernetes Official Documentation](https://kubernetes.io/docs/reference/node/kubelet-checkpoint-api/)\n2. **KEP-2008:**[Forensic Container Checkpointing Enhancement Proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2008-forensic-container-checkpointing)\n3. **CRIU Architecture:**[CRIU Documentation](https://criu.org)\n4. **CRI-O Container Engine:**[CRI-O GitHub Repository](https://github.com/cri-o/cri-o)\n5. **Cilium eBPF Service Mesh:**[Cilium Documentation](https://docs.cilium.io)\n\nThis research project and its accompanying artifacts are open-source under the Apache License 2.0.", "url": "https://wpnews.pro/news/evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf", "canonical_source": "https://github.com/musaxasmammedov7/final-research", "published_at": "2026-10-28 19:58:28+00:00", "updated_at": "2026-10-28 20:12:03.092671+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "mlops"], "entities": ["Musa Khasmammadov", "Kubernetes", "CRIU", "AWS EC2", "Buildah", "CRI-O", "Cilium", "Kubelet Checkpoint API"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf", "markdown": "https://wpnews.pro/news/evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf.md", "text": "https://wpnews.pro/news/evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf.txt", "jsonld": "https://wpnews.pro/news/evaluating-kubernetes-pod-checkpoint-restore-on-aws-criu-pleg-ebpf.jsonld"}}