This repository contains the complete experimental testbed, deployment automation, Kubernetes operator, benchmark harnesses, and LaTeX source code for the research paper:
Evaluating Checkpoint/Restore-Based Pod Migration in Kubernetes: An Empirical Performance and Bottleneck Study
Musa Khasmammadov
📄 Read the 5-Page IEEE-Formatted PDF | 📝 View TeX Source
Modern cloud architectures typically assume container workloads are stateless and ephemeral. Critical production workloads (e.g., machine learning embedding caches, in-memory stream processors, stateful gaming engines), however, maintain gigabytes of volatile heap state that cannot be persisted to external databases without prohibitive latency or intrusive application refactoring.
We designed and empirically evaluated an end-to-end, snapshot-consistent pod migration pipeline coordinating CRIU (v4.3), the Kubelet Checkpoint API (KEP-2008), Buildah (v1.37), CRI-O (v1.31), and Cilium eBPF (v1.16) on a dedicated multi-node AWS EC2 cluster (eu-central-1).
+----------------------------------------------------------------------------------------------------+
| AWS EC2 3-Node Topology |
| |
| +-----------------------+ +-----------------------+ +----------------------------------+ |
| | k8s-master | | k8s-worker-1 | | k8s-worker-2 | |
| | (c7i-flex.large) | | (m7i-flex.large) | | (m7i-flex.large) | |
| | - Control Plane | | - CRI-O 1.31 + CRIU | | - CRI-O 1.31 + CRIU | |
| | - OCI Registry :5000 | | - Buildah 1.37 | | - Buildah 1.37 | |
| | - Cilium Ingress | | - Source Pod (State) | === | - Restored Pod (Bit-Exact RAM) | |
| +-----------------------+ +-----------------------+ +----------------------------------+ |
| | | | |
| +-----------------------------+------------------------------------+ |
| Private VPC Subnet (10.10.1.0/24) |
| Cilium eBPF Host Routing (No iptables) |
+----------------------------------------------------------------------------------------------------+
External 50 Hz HTTP probing (20 ms interval) across 36 bidirectional migrations and 20 cold restarts reveals:
| Metric | Decoupled CRIU Migration ( | Native Cold Reschedule ( | Delta / Statistical Test |
|---|---|---|---|
| Median Cutover Blackout ( | 1064.5 ms | 1019.0 ms | +45.5 ms (+4.5%) |
| Mean Cutover Blackout | Difference: | ||
| Mann-Whitney U Test | — | (Null hypothesis not rejected at | |
| Welch's Two-Sample t-test | — | ||
| RAM State Preservation | 100% Intact (Heap preserved) | 0% (Reset to zero) | Bit-exact state continuity |
| Kubelet PLEG Synchronization | Dominated by 1 s PLEG relist cycle ( |
Note
The two-sided Mann-Whitney U test confirms that decoupled process heap restoration does not provide evidence of a large cutover penalty over cold pod initialization (
| Dimension | Mode A: Decoupled Migration (Default) | Mode B: Synchronous Stop-and-Copy | Native Cold Reschedule |
|---|---|---|---|
| Source Pod Lifecycle | Continues serving during packaging/push | Frozen/killed before packaging/push | Terminated immediately |
| Client Cutover Blackout ( | 1064.5 ms (p50) /797 ms (best) | 1878 ms | 1019.0 ms (p50) |
| State Drift (RPO) | Bounded drift: | 0.000 ms (Zero drift) | Total loss (100% loss) |
| Best Suited Workloads | Monotonic counters, caches, gaming lobbies | Bit-exact financial ledgers, ACID logs | Ephemeral/stateless services |
We evaluated numerical pseudo-random float64 heaps up to 1024 MB (1 GB) with cross-node cryptographic validation:
| Allocated Heap | OCI Layer Size | -D ) |
RPO (gzip / -D ) |
Worst -D ) |
% 120s Spot SLA | SHA-256 Digest Status | |||
|---|---|---|---|---|---|---|---|---|---|
| 26 MB (p50) | 5.16 MiB | 0.144 s | 0.371 s | 0.261 s | 0.67 s / 0.56 s | 5.70 s | 4.8% | 1.066 s | Verified ( Counter ) |
| 64 MB | 69.11 MiB | 0.216 s | 0.761 s | 0.361 s | 1.38 s / 0.98 s | 6.19 s | 5.2% | 0.951 s | 100% Match (7785bfc7 ) |
| 128 MB | 133.11 MiB | 0.258 s | 1.082 s | 0.528 s | 2.01 s / 1.46 s | 6.71 s | 5.6% | 1.695 s | 100% Match (cce52f48 ) |
| 256 MB | 261.13 MiB | 0.375 s | 5.796 s | 1.473 s ( | 7.48 s / 3.16 s | 8.53 s | 7.1% | 2.108 s | 100% Match (4b1b1359 ) |
| 512 MB | 517.15 MiB | 0.760 s | 16.120 s | 4.210 s ( | 19.39 s / 7.48 s | 13.24 s | 11.0% | 7.807 s | 100% Match (ab028a06 ) |
| 1024 MB (1 GB) | 1029.21 MiB | 7.464 s | 32.060 s | 13.743 s ( | 42.63 s / 24.31 s | 36.78 s | 30.6% | 17.328 s | 100% Match (c0da7edf ) |
AWS Spot SLA Guarantee: Under worst-case Spot preemption detection ($T_{\text{detect}}^{\text{worst}} = 5.0\text{ s}$ ) with Buildah--disable-compression , total source evacuation completes in36.78 s (only30.6% of the 120 s AWS Spot preemption deadline).
2.
Buildah Gzip Bypass Optimization: Disabling single-threaded gzip serialization (--disable-compression /-D ) slashes 1 GB packaging from 32.06 s to13.74 s (saving18.32 s , a$2.3\times$ speedup).
3.
Linux Kernel Dirty-Page Writeback: From 26 MB to 512 MB,$T_{\text{freeze}}$ remains sub-second ($144\text{ ms} \to 760\text{ ms}$ ) as writes are buffered in page cache. At 1 GB, uncommitted dirty memory hits 783.6 MB (vm.dirty_background_ratio = 10% ), triggering kernel flusher serialization to EBS gp3 and increasing$T_{\text{freeze}}$ to 7.464 s.
4.
Restoration Phase Breakdown: Target-side restoration ($T_{\text{restore}} = 17.328\text{ s}$ ) breaks down into four phases:
- Registry layer pull over VPC ($\approx 8.5\text{ s}$ , matching$\approx 1\text{ Gbps}$ baseline instance bandwidth).
- CRI-O storage extraction and unpacking ($\approx 4.8\text{ s}$ ).
runc/criu restore memory mapping and process unfreezing ($\approx 3.2\text{ s}$ ).
- Kubelet PLEG status synchronization ($\approx 0.8\text{ s}$ ).
- Registry layer pull over VPC (
Why Migrate? Reconstructing a 1 GB state snapshot from cloud object storage (Amazon S3) over assumed multi-tenant cloud bandwidth (300–800 Mbps) requires an estimated11.2–29.8 s recovery window. Checkpointing eliminates external database persistence tiers and application-level serialization logic, with cutover becoming advantageous over cold rebuild from$\approx 256\text{ MB}$ (2.10 s vs. 3.5–7.5 s).
We deployed a two-container test pod (stream-server listening on 127.0.0.1:8090 + envoy-sidecar on 0.0.0.0:15006 forwarding over loopback TCP).
Finding: KEP-2008 operates atsingle-container granularity . Checkpointingstream-server and restoring it on the target node isolated it from the co-scheduled sidecar ($T_{\text{blackout}} = \infty$ ), because CRIU'stcp-close closes active loopback sockets. #
Architectural Recommendation: Single-container checkpointing severs loopback sidecar peering; node-level routing architectures (such asCilium eBPF host-routing or Kubernetes ClusterIP proxies) decouple L4/L7 service redirection from pod network namespaces, providing a resilient foundation.
.
├── README.md # Project documentation & benchmark overview
├── ansible/ # Infrastructure automation playbooks
│ ├── ansible.cfg
│ └── site.yml # Bootstrap Linux 6.8+, CRI-O 1.31, CRIU v4.3, Buildah
├── manifests/ # Kubernetes CRDs, Operator, and Workloads
│ ├── 01-cert-manager-issuer.yaml # Let's Encrypt TLS ClusterIssuer
│ ├── 02-google-online-boutique.yaml # 11-microservice background benchmark workload
│ ├── 03-live-migration-stream-demo.yaml # Stateful in-memory heap benchmark pod
│ ├── 03a-live-stream-template.yaml # Checkpoint restore pod template
│ ├── 04-cilium-ingress-https.yaml # Cilium Service Mesh Ingress configuration
│ └── 05-live-migration-operator.yaml # PodLiveMigration CRD & autonomous DaemonSet controller
├── paper/ # Academic IEEE Manuscript
│ ├── Stateful_Container_Live_Migration_IEEE_NativeTeX.pdf # Compiled 5-page IEEE PDF
│ └── paper_stateful_live_migration.tex # Native LaTeX source (IEEEtran, TikZ, PGFPlots)
├── scripts/ # Benchmark harnesses and empirical test tools
│ ├── benchmark_migration.py # 50 Hz high-frequency prober & N-run benchmark suite
│ └── run_mode_b_cluster.py # Mode B (Stop-and-Copy) synchronous test harness
└── terraform/ # AWS EC2 multi-node cluster provisioning
├── main.tf # VPC, security groups, 1 master + 2 workers
├── outputs.tf # Cluster public/private IPs
└── variables.tf # Instance types (c7i-flex.large, m7i-flex.large)
cd terraform
terraform init
terraform apply -auto-approve
cd ../ansible
ansible-playbook site.yml
kubectl apply -f ../manifests/01-cert-manager-issuer.yaml
kubectl apply -f ../manifests/02-google-online-boutique.yaml
kubectl apply -f ../manifests/03-live-migration-stream-demo.yaml
kubectl apply -f ../manifests/04-cilium-ingress-https.yaml
kubectl apply -f ../manifests/05-live-migration-operator.yaml
To execute an automated statistical run with high-frequency HTTP probing:
ssh -i terraform/k8s-research-key.pem ubuntu@<MASTER_PUBLIC_IP> \
"python3 /home/ubuntu/scripts/benchmark_migration.py --runs 36"
The paper uses native IEEEtran and compiles cleanly with tectonic or xelatex:
tectonic paper/paper_stateful_live_migration.tex --outdir paper/
- Kubelet Checkpoint API:Kubernetes Official Documentation
- KEP-2008:Forensic Container Checkpointing Enhancement Proposal
- CRIU Architecture:CRIU Documentation
- CRI-O Container Engine:CRI-O GitHub Repository
- Cilium eBPF Service Mesh:Cilium Documentation
This research project and its accompanying artifacts are open-source under the Apache License 2.0.