Evaluating Kubernetes Pod Checkpoint/Restore on AWS (CRIU, PLEG, eBPF) A study by Musa Khasmammadov evaluating checkpoint/restore-based pod migration on a 3-node AWS EC2 cluster in eu-central-1 found that decoupled CRIU (v4.3) migration preserved 100% of in-memory heap state while adding a median cutover blackout of 1064.5 ms versus 1019.0 ms for native cold reschedule, a difference of 45.5 ms (+4.5%) across 36 bidirectional migrations and 20 cold restarts. A two-sided Mann-Whitney U test found the cutover penalty was not statistically significant, and the pipeline's timing was dominated by the Kubelet's 1-second PLEG relist cycle. The testbed coordinated CRIU, the Kubelet Checkpoint API (KEP-2008), Buildah (v1.37), CRI-O (v1.31) and Cilium eBPF (v1.16) to migrate stateful workloads such as machine learning embedding caches without application refactoring. Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes: An Empirical Performance and Bottleneck Study This repository contains the complete experimental testbed, deployment automation, Kubernetes operator, benchmark harnesses, and LaTeX source code for the research paper: Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes: An Empirical Performance and Bottleneck Study Musa Khasmammadov 📄 Read the 5-Page IEEE-Formatted PDF https://github.com/musaxasmammedov7/final-research/blob/main/paper/Stateful Container Live Migration IEEE NativeTeX.pdf | 📝 View TeX Source https://github.com/musaxasmammedov7/final-research/blob/main/paper/paper stateful live migration.tex Modern cloud architectures typically assume container workloads are stateless and ephemeral. Critical production workloads e.g., machine learning embedding caches, in-memory stream processors, stateful gaming engines , however, maintain gigabytes of volatile heap state that cannot be persisted to external databases without prohibitive latency or intrusive application refactoring. We designed and empirically evaluated an end-to-end, snapshot-consistent pod migration pipeline coordinating CRIU v4.3 , the Kubelet Checkpoint API KEP-2008 , Buildah v1.37 , CRI-O v1.31 , and Cilium eBPF v1.16 on a dedicated multi-node AWS EC2 cluster eu-central-1 . +----------------------------------------------------------------------------------------------------+ | AWS EC2 3-Node Topology | | | | +-----------------------+ +-----------------------+ +----------------------------------+ | | | k8s-master | | k8s-worker-1 | | k8s-worker-2 | | | | c7i-flex.large | | m7i-flex.large | | m7i-flex.large | | | | - Control Plane | | - CRI-O 1.31 + CRIU | | - CRI-O 1.31 + CRIU | | | | - OCI Registry :5000 | | - Buildah 1.37 | | - Buildah 1.37 | | | | - Cilium Ingress | | - Source Pod State | === | - Restored Pod Bit-Exact RAM | | | +-----------------------+ +-----------------------+ +----------------------------------+ | | | | | | | +-----------------------------+------------------------------------+ | | Private VPC Subnet 10.10.1.0/24 | | Cilium eBPF Host Routing No iptables | +----------------------------------------------------------------------------------------------------+ External 50 Hz HTTP probing 20 ms interval across 36 bidirectional migrations and 20 cold restarts reveals: | Metric | Decoupled CRIU Migration | Native Cold Reschedule | Delta / Statistical Test | |---|---|---|---| | Median Cutover Blackout | 1064.5 ms | 1019.0 ms | +45.5 ms +4.5% | | Mean Cutover Blackout | | | Difference: | | Mann-Whitney U Test | | — | Null hypothesis not rejected at | | Welch's Two-Sample t-test | | — | | | RAM State Preservation | 100% Intact Heap preserved | 0% Reset to zero | Bit-exact state continuity | | Kubelet PLEG Synchronization | | | Dominated by 1 s PLEG relist cycle | Note The two-sided Mann-Whitney U test confirms that decoupled process heap restoration does not provide evidence of a large cutover penalty over cold pod initialization | Dimension | Mode A: Decoupled Migration Default | Mode B: Synchronous Stop-and-Copy | Native Cold Reschedule | |---|---|---|---| | Source Pod Lifecycle | Continues serving during packaging/push | Frozen/killed before packaging/push | Terminated immediately | | Client Cutover Blackout | 1064.5 ms p50 / 797 ms best | 1878 ms | 1019.0 ms p50 | | State Drift RPO | Bounded drift: | 0.000 ms Zero drift | Total loss 100% loss | | Best Suited Workloads | Monotonic counters, caches, gaming lobbies | Bit-exact financial ledgers, ACID logs | Ephemeral/stateless services | We evaluated numerical pseudo-random float64 heaps up to 1024 MB 1 GB with cross-node cryptographic validation: | Allocated Heap | OCI Layer Size | | | -D | RPO gzip / -D | Worst -D | % 120s Spot SLA | | SHA-256 Digest Status | |---|---|---|---|---|---|---|---|---|---| | 26 MB p50 | 5.16 MiB | 0.144 s | 0.371 s | 0.261 s | 0.67 s / 0.56 s | 5.70 s | 4.8% | 1.066 s | Verified Counter | | 64 MB | 69.11 MiB | 0.216 s | 0.761 s | 0.361 s | 1.38 s / 0.98 s | 6.19 s | 5.2% | 0.951 s | 100% Match 7785bfc7 | | 128 MB | 133.11 MiB | 0.258 s | 1.082 s | 0.528 s | 2.01 s / 1.46 s | 6.71 s | 5.6% | 1.695 s | 100% Match cce52f48 | | 256 MB | 261.13 MiB | 0.375 s | 5.796 s | 1.473 s | 7.48 s / 3.16 s | 8.53 s | 7.1% | 2.108 s | 100% Match 4b1b1359 | | 512 MB | 517.15 MiB | 0.760 s | 16.120 s | 4.210 s | 19.39 s / 7.48 s | 13.24 s | 11.0% | 7.807 s | 100% Match ab028a06 | | 1024 MB 1 GB | 1029.21 MiB | 7.464 s | 32.060 s | 13.743 s | 42.63 s / 24.31 s | 36.78 s | 30.6% | 17.328 s | 100% Match c0da7edf | 1. AWS Spot SLA Guarantee: Under worst-case Spot preemption detection $T {\text{detect}}^{\text{worst}} = 5.0\text{ s}$ with Buildah --disable-compression , total source evacuation completes in 36.78 s only 30.6% of the 120 s AWS Spot preemption deadline . 2. Buildah Gzip Bypass Optimization: Disabling single-threaded gzip serialization --disable-compression / -D slashes 1 GB packaging from 32.06 s to 13.74 s saving 18.32 s , a$2.3\times$ speedup . 3. Linux Kernel Dirty-Page Writeback: From 26 MB to 512 MB,$T {\text{freeze}}$ remains sub-second $144\text{ ms} \to 760\text{ ms}$ as writes are buffered in page cache. At 1 GB, uncommitted dirty memory hits 783.6 MB vm.dirty background ratio = 10% , triggering kernel flusher serialization to EBS gp3 and increasing$T {\text{freeze}}$ to 7.464 s. 4. Restoration Phase Breakdown: Target-side restoration $T {\text{restore}} = 17.328\text{ s}$ breaks down into four phases: - Registry layer pull over VPC $\approx 8.5\text{ s}$ , matching$\approx 1\text{ Gbps}$ baseline instance bandwidth . - CRI-O storage extraction and unpacking $\approx 4.8\text{ s}$ . - runc/criu restore memory mapping and process unfreezing $\approx 3.2\text{ s}$ . - Kubelet PLEG status synchronization $\approx 0.8\text{ s}$ . 5. Registry layer pull over VPC 6. Why Migrate? Reconstructing a 1 GB state snapshot from cloud object storage Amazon S3 over assumed multi-tenant cloud bandwidth 300–800 Mbps requires an estimated 11.2–29.8 s recovery window. Checkpointing eliminates external database persistence tiers and application-level serialization logic, with cutover becoming advantageous over cold rebuild from$\approx 256\text{ MB}$ 2.10 s vs. 3.5–7.5 s . We deployed a two-container test pod stream-server listening on 127.0.0.1:8090 + envoy-sidecar on 0.0.0.0:15006 forwarding over loopback TCP . - Finding: KEP-2008 operates at single-container granularity . Checkpointing stream-server and restoring it on the target node isolated it from the co-scheduled sidecar $T {\text{blackout}} = \infty$ , because CRIU's tcp-close closes active loopback sockets. - Architectural Recommendation: Single-container checkpointing severs loopback sidecar peering; node-level routing architectures such as Cilium eBPF host-routing or Kubernetes ClusterIP proxies decouple L4/L7 service redirection from pod network namespaces, providing a resilient foundation. . ├── README.md Project documentation & benchmark overview ├── ansible/ Infrastructure automation playbooks │ ├── ansible.cfg │ └── site.yml Bootstrap Linux 6.8+, CRI-O 1.31, CRIU v4.3, Buildah ├── manifests/ Kubernetes CRDs, Operator, and Workloads │ ├── 01-cert-manager-issuer.yaml Let's Encrypt TLS ClusterIssuer │ ├── 02-google-online-boutique.yaml 11-microservice background benchmark workload │ ├── 03-live-migration-stream-demo.yaml Stateful in-memory heap benchmark pod │ ├── 03a-live-stream-template.yaml Checkpoint restore pod template │ ├── 04-cilium-ingress-https.yaml Cilium Service Mesh Ingress configuration │ └── 05-live-migration-operator.yaml PodLiveMigration CRD & autonomous DaemonSet controller ├── paper/ Academic IEEE Manuscript │ ├── Stateful Container Live Migration IEEE NativeTeX.pdf Compiled 5-page IEEE PDF │ └── paper stateful live migration.tex Native LaTeX source IEEEtran, TikZ, PGFPlots ├── scripts/ Benchmark harnesses and empirical test tools │ ├── benchmark migration.py 50 Hz high-frequency prober & N-run benchmark suite │ └── run mode b cluster.py Mode B Stop-and-Copy synchronous test harness └── terraform/ AWS EC2 multi-node cluster provisioning ├── main.tf VPC, security groups, 1 master + 2 workers ├── outputs.tf Cluster public/private IPs └── variables.tf Instance types c7i-flex.large, m7i-flex.large cd terraform terraform init terraform apply -auto-approve cd ../ansible ansible-playbook site.yml kubectl apply -f ../manifests/01-cert-manager-issuer.yaml kubectl apply -f ../manifests/02-google-online-boutique.yaml kubectl apply -f ../manifests/03-live-migration-stream-demo.yaml kubectl apply -f ../manifests/04-cilium-ingress-https.yaml kubectl apply -f ../manifests/05-live-migration-operator.yaml To execute an automated statistical run with high-frequency HTTP probing: ssh -i terraform/k8s-research-key.pem ubuntu@