Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot

NVIDIA released Snapshot, an open-source checkpoint-and-restore component of the NVIDIA DSX AI Factory Platform, which captures a fully initialized GPU workload's CPU and GPU memory and restores it on any node with the same GPU model and driver, bringing inference capacity online in seconds instead of minutes. Design partner Crusoe cut GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300, and Saturn Cloud cut vLLM startup from 2–8.5 minutes to 6–34 seconds on the same hardware, with benchmarks spanning vLLM, SGLang and NVIDIA TensorRT LLM and models from 600 million parameters to MoE models with 1 TB checkpoints on 8 GPUs. Snapshot uses Kubernetes custom resources PodSnapshot and SnapshotJob for checkpointing and the nvidia.com/restore-from annotation for restore, and ships ready-made examples for vLLM, SGLang and TensorRT LLM.

NVIDIA Snapshot https://github.com/ai-dynamo/snapshot readme , a new open-source component of the NVIDIA DSX AI Factory Platform https://www.nvidia.com/en-us/data-center/products/dsx/ , makes initialized GPU workloads reusable across inference replicas—speeding startup and recovery. When inference demand spikes or a replica fails, available GPUs don’t translate immediately into serving capacity. Initializing an LLM workload—a process known as a cold start—can take minutes, delaying scaling and recovery. Snapshot is a checkpoint and restore system for GPU workloads on Kubernetes. It captures a fully initialized workload—including CPU and GPU memory—and restores that state on compatible infrastructure, bringing inference capacity online in seconds instead of minutes. Figure 1 shows the startup sequence. Snapshot skips repeated workload initialization; infrastructure provisioning and container image loading remain outside the measured restore path. This post explains how Snapshot checkpoints and restores GPU workloads, how platform teams can integrate it, and how restore times compare with cold starts across three inference engines. Snapshot for AI factory operations Originally developed for NVIDIA Dynamo inference workers, Snapshot provides checkpoint and restore capabilities that other inference engines, schedulers, controllers, and platforms can integrate. Snapshot provides two operations for GPU Pods: - Checkpoint : Once a workload is fully initialized, capture its CPU and GPU memory as a portable artifact. - Restore : Start a new Pod from that artifact on any node with the same GPU model and driver. The process continues from where it was checkpointed, so weight loading, compilation, and graph capture are skipped. Both operations use Kubernetes custom resources. To checkpoint a Pod that is already running, create a PodSnapshot that names it. To start a Pod from a template and checkpoint it as soon as it is ready, create a SnapshotJob instead. To restore, add the annotation nvidia.com/restore-from to a new Pod, naming the PodSnapshot to restore from. The workload only needs a small handshake to pause before the checkpoint and resume after the restore. The repository has ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM https://github.com/ai-dynamo/snapshot/blob/main/docs/guides/README.md 1-make-the-workload-snapshot-ready . Snapshot can checkpoint and restore compatible workloads that span multiple GPUs on a single node as a single checkpoint artifact across ranks. This supports fast startup for large mixture-of-experts MoE models using parallelism strategies such as tensor parallelism TP , data and expert parallelism DEP , and decode context parallelism DCP . Workloads must meet Snapshot’s checkpoint/restore and runtime compatibility requirements https://github.com/ai-dynamo/snapshot/blob/main/docs/reference/workload-contract.md . See the example configurations https://github.com/ai-dynamo/snapshot/blob/main/docs/reference/workload-contract.md multi-gpu-workloads . Benchmarks Design partners have also evaluated Snapshot on their own workloads: - Crusoe https://www.crusoe.ai/resources/blog/nvidia-snapshot-crusoe-cloud , an upstream contributor to Snapshot, reduced GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300. - Saturn Cloud https://saturncloud.io/blog/benchmarking-nvidia-dynamo-snapshot , a token factory platform, benchmarked NVIDIA Snapshot for scale-from-zero inference, cutting vLLM startup from 2–8.5 minutes to 6–34 seconds on NVIDIA HGX B300 https://www.nvidia.com/en-us/data-center/hgx/ . The following benchmarks show how Snapshot accelerates inference startup across vLLM, SGLang, and NVIDIA TensorRT LLM TRT-LLM —with models from 600 million parameters to mixture-of-experts https://www.nvidia.com/en-us/glossary/mixture-of-experts/ MoE models with 1 TB checkpoints on 8 GPUs. Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start. Methodology Each configuration was tested using two strategies on the same hardware and software stack: - Cold start: The workload starts from scratch. The clock runs from process launch until engine initialization returns, before inference. The measurement is divided into four phases Table 2 : - Startup: Starting the engine and worker processes, importing Python libraries, and setting up communication between GPUs. Some just-in-time JIT kernel compilation also happens here. - Model load: Reading the model weights and constructing the model’s tensors, including any engine-specific conversion or repacking done while loading. - Initialization: Preparing the loaded model to serve: memory profiling, allocating the key-value KV cache and runtime buffers, and other kernel and runtime preparation. - Compilation, autotuning, and graph capture: Compiling kernels for example, with torch.compile , autotuning them for the GPU, and capturing CUDA graphs. - Snapshot: The workload is restored from its checkpoint. The clock runs from the moment Snapshot is asked to restore until the workload is serving again. The measurement is divided into four stages Table 3 : - Agent setup: The Snapshot agent on the node locates the checkpoint artifact and prepares the target Pod and GPUs. - Checkpoint/Restore in Userspace CRIU restore: CRIU, the standard Linux tool for checkpointing processes, reads the process image from storage into host memory. Speed is set mostly by the storage backend. - CUDA restore: Copying GPU state back into GPU memory. - Wake/remap: Adapting the process to the new node and waking the engine. Both measurements exclude pulling the container image and setting up the container runtime. The KV cache release is triggered before every checkpoint, so it is not part of the artifact. Table 1 also reports checkpoint time and size: how long it takes to capture a fully initialized replica and how much storage the stored artifact requires. This is a one-time cost per model and configuration. Setup - GPU: NVIDIA B200 - NVIDIA driver: 615.71.09 - Storage backend: VAST NFS - Inference engines: - vLLM 0.31.0 - SGLang 0.5.21 - TensorRT LLM 1.3.0 Results The benchmarks show four main findings. - Restore times stayed under 43 seconds: Across all tested configurations, Snapshot restored replicas in 3.6 to 42.7 seconds—8x to 36x faster than cold start. The largest MoE models, with checkpoints of about 600 GB to 1 TB across eight GPUs, restored in under 43 seconds compared with cold-start times of 6 to 19 minutes. - Restoring skips more than weight loading: Cold-start time includes loading weights, compiling and autotuning kernels, and capturing CUDA graphs. Snapshot restores the initialized workload, skipping these repeated steps. - Checkpointing pays off with the first restored replica: In every tested configuration, checkpoint time plus restore time was shorter than a single cold start. Checkpoint creation took 15 seconds to 5.5 minutes, depending on the model and configuration. - Moving GPU state dominates large restores: For the eight-GPU MoE configurations, restoring GPU state took 18 to 29 seconds. Restoring CPU process state with CRIU took 4.2 to 10.8 seconds. Figures 2 to 4 compare cold-start and Snapshot restore times for vLLM, SGLang, and TensorRT LLM across the tested models and parallelism configurations. Table 1 provides the data for every configuration, including checkpoint time and size. All times are in seconds. In the figures and tables, TEP8 denotes tensor and expert parallelism across eight GPUs https://github.com/ai-dynamo/snapshot/blob/main/docs/development/benchmarks.md . | Model | Engine | Parallelism | Weight size GB | Checkpoint size GB | Checkpoint time s | Cold start s | Snapshot agent restore s | Speedup | |---|---|---|---|---|---|---|---|---| | Qwen3-0.6B | vLLM | TP1 | 1.50 | 8.13 | 15.30 | 104.43 | 4.66 | 22.40x | | Qwen3-0.6B | SGLang | TP1 | 1.50 | 9.42 | 20.71 | 87.08 | 3.59 | 24.26x | | Qwen3-0.6B | TRT-LLM | TP1 | 1.50 | 8.04 | 20.07 | 121.26 | 8.15 | 14.88x | | Qwen3-14B | vLLM | TP1 | 29.54 | 37.71 | 19.98 | 122.76 | 5.72 | 21.45x | | Qwen3-14B | vLLM | TP2 | 29.54 | 48.29 | 38.37 | 139.63 | 6.19 | 22.55x | | Qwen3-14B | SGLang | TP1 | 29.54 | 40.19 | 26.17 | 106.26 | 5.60 | 18.96x | | Qwen3-14B | SGLang | TP2 | 29.54 | 51.93 | 33.88 | 124.72 | 7.45 | 16.75x | | Qwen3-14B | TRT-LLM | TP1 | 29.54 | 36.38 | 23.72 | 127.02 | 12.50 | 10.16x | | Qwen3-14B | TRT-LLM | TP2 | 29.54 | 47.97 | 42.90 | 158.66 | 15.99 | 9.92x | | gpt-oss-120b | vLLM | TP1 | 65.25 | 81.56 | 44.37 | 172.80 | 10.09 | 17.12x | | gpt-oss-120b | vLLM | TP4 | 65.25 | 119.69 | 133.01 | 155.66 | 11.62 | 13.40x | | gpt-oss-120b | SGLang | TP1 | 65.25 | 172.32 | 55.63 | 268.96 | 15.94 | 16.87x | | gpt-oss-120b | SGLang | TP4 | 65.25 | 693.89 | 174.20 | 240.17 | 29.64 | 8.10x | | gpt-oss-120b | TRT-LLM | TP1 | 65.25 | 77.87 | 29.98 | 200.55 | 7.79 | 25.75x | | gpt-oss-120b | TRT-LLM | TP4 | 65.25 | 111.34 | 85.29 | 223.96 | 18.58 | 12.05x | | GLM-5.3-NVFP4 | vLLM | TEP8 | 464.82 | 604.00 | 282.80 | 393.65 | 28.64 | 13.74x | | GLM-5.3-NVFP4 | SGLang | TEP8 | 464.82 | 669.49 | 327.20 | 426.70 | 35.34 | 12.07x | | GLM-5.3-NVFP4 | TRT-LLM | TEP8 | 464.82 | 628.30 | 231.33 | 363.90 | 33.55 | 10.85x | | DeepSeek-V4.1-Flash | vLLM | TEP8 | 510.30 | 647.18 | 274.75 | 1110.65 | 30.96 | 35.87x | | DeepSeek-V4.1-Flash | SGLang | TEP8 | 510.30 | 743.35 | 286.77 | 953.09 | 36.54 | 26.08x | | DeepSeek-V4-Pro-NVFP4 | vLLM | TEP8 | 913.09 | 996.53 | 193.84 | 908.48 | 39.56 | 22.96x | | DeepSeek-V4-Pro-NVFP4 | SGLang | TEP8 | 913.09 | 1039.09 | 253.72 | 621.55 | 42.69 | 14.56x | Table 1. Cold start compared with a Snapshot restore on NVIDIA B200 with VAST NFS storage, on 1 to 8 GPUs. Checkpoint time is a one-time cost per configuration. Speedup is cold start time divided by Snapshot restore time Table 2 breaks down the cold-start time, showing which initialization phases a restore skips. | Model | Engine | Parallelism | Startup s | Model load s | Initialization s | Compilation, autotuning & graph capture s | Total s | |---|---|---|---|---|---|---|---| | Qwen3-0.6B | vLLM | TP1 | 31.37 | 17.27 | 14.58 | 41.21 | 104.43 | | Qwen3-0.6B | SGLang | TP1 | 33.74 | 11.67 | 8.41 | 33.26 | 87.08 | | Qwen3-0.6B | TRT-LLM | TP1 | 25.79 | 3.79 | 40.64 | 51.04 | 121.26 | | Qwen3-14B | vLLM | TP1 | 31.16 | 26.15 | 15.30 | 50.15 | 122.76 | | Qwen3-14B | vLLM | TP2 | 42.18 | 18.96 | 15.29 | 63.20 | 139.63 | | Qwen3-14B | SGLang | TP1 | 33.45 | 20.79 | 8.28 | 43.74 | 106.26 | | Qwen3-14B | SGLang | TP2 | 47.62 | 18.93 | 8.29 | 49.88 | 124.72 | | Qwen3-14B | TRT-LLM | TP1 | 25.35 | 11.54 | 39.36 | 50.77 | 127.02 | | Qwen3-14B | TRT-LLM | TP2 | 27.42 | 8.88 | 70.62 | 51.74 | 158.66 | | gpt-oss-120b | vLLM | TP1 | 35.15 | 38.69 | 10.49 | 88.47 | 172.80 | | gpt-oss-120b | vLLM | TP4 | 49.91 | 24.71 | 12.13 | 68.91 | 155.66 | | gpt-oss-120b | SGLang | TP1 | 42.54 | 34.44 | 0.36 | 191.62 | 268.96 | | gpt-oss-120b | SGLang | TP4 | 57.97 | 49.06 | 1.33 | 131.81 | 240.17 | | gpt-oss-120b | TRT-LLM | TP1 | 26.21 | 47.50 | 42.85 | 83.99 | 200.55 | | gpt-oss-120b | TRT-LLM | TP4 | 34.28 | 42.72 | 22.34 | 124.62 | 223.96 | | GLM-5.3-NVFP4 | vLLM | TEP8 | 65.47 | 65.88 | 48.88 | 213.42 | 393.65 | | GLM-5.3-NVFP4 | SGLang | TEP8 | 60.77 | 73.98 | 2.66 | 289.29 | 426.70 | | GLM-5.3-NVFP4 | TRT-LLM | TEP8 | 31.58 | 90.39 | 120.43 | 121.50 | 363.90 | | DeepSeek-V4.1-Flash | vLLM | TEP8 | 63.96 | 184.90 | 77.25 | 784.54 | 1110.65 | | DeepSeek-V4.1-Flash | SGLang | TEP8 | 56.18 | 74.21 | 38.50 | 784.20 | 953.09 | | DeepSeek-V4-Pro-NVFP4 | vLLM | TEP8 | 64.42 | 95.14 | 64.87 | 684.05 | 908.48 | | DeepSeek-V4-Pro-NVFP4 | SGLang | TEP8 | 58.80 | 87.68 | 42.55 | 432.52 | 621.55 | Table 2. Where the cold start time goes, by phase, in seconds. Phase timings may differ from the total because of rounding or intervals not captured by logging and instrumentation Table 3 breaks down restore time into agent setup, CRIU restore, CUDA restore, and wake/remap. | Model | Engine | Parallelism | Snapshot: agent setup | Snapshot: CRIU restore | Snapshot: CUDA restore | Snapshot: wake/remap | Total s | |---|---|---|---|---|---|---|---| | Qwen3-0.6B | vLLM | TP1 | 2.30 | 1.36 | 0.78 | 0.22 | 4.66 | | Qwen3-0.6B | SGLang | TP1 | 0.24 | 1.33 | 1.02 | 0.99 | 3.59 | | Qwen3-0.6B | TRT-LLM | TP1 | 5.72 | 1.43 | 0.84 | 0.16 | 8.15 | | Qwen3-14B | vLLM | TP1 | 0.13 | 1.58 | 3.16 | 0.85 | 5.72 | | Qwen3-14B | vLLM | TP2 | 0.57 | 2.03 | 3.43 | 0.17 | 6.19 | | Qwen3-14B | SGLang | TP1 | 0.24 | 1.35 | 3.67 | 0.34 | 5.60 | | Qwen3-14B | SGLang | TP2 | 0.92 | 2.17 | 3.49 | 0.87 | 7.45 | | Qwen3-14B | TRT-LLM | TP1 | 6.80 | 2.70 | 2.82 | 0.18 | 12.50 | | Qwen3-14B | TRT-LLM | TP2 | 9.65 | 3.52 | 2.41 | 0.41 | 15.99 | | gpt-oss-120b | vLLM | TP1 | 0.48 | 2.60 | 6.71 | 0.30 | 10.09 | | gpt-oss-120b | vLLM | TP4 | 1.02 | 4.40 | 5.57 | 0.62 | 11.62 | | gpt-oss-120b | SGLang | TP1 | 0.26 | 1.69 | 13.46 | 0.54 | 15.94 | | gpt-oss-120b | SGLang | TP4 | 2.69 | 4.34 | 21.69 | 0.93 | 29.64 | | gpt-oss-120b | TRT-LLM | TP1 | 0.28 | 1.51 | 5.74 | 0.26 | 7.79 | | gpt-oss-120b | TRT-LLM | TP4 | 9.99 | 3.30 | 4.34 | 0.95 | 18.58 | | GLM-5.3-NVFP4 | vLLM | TEP8 | 3.79 | 4.18 | 20.61 | 0.06 | 28.64 | | GLM-5.3-NVFP4 | SGLang | TEP8 | 3.27 | 10.80 | 19.96 | 1.31 | 35.34 | | GLM-5.3-NVFP4 | TRT-LLM | TEP8 | 9.81 | 5.32 | 18.17 | 0.25 | 33.55 | | DeepSeek-V4.1-Flash | vLLM | TEP8 | 4.84 | 5.71 | 19.37 | 1.04 | 30.96 | | DeepSeek-V4.1-Flash | SGLang | TEP8 | 4.34 | 9.77 | 21.40 | 1.03 | 36.54 | | DeepSeek-V4-Pro-NVFP4 | vLLM | TEP8 | 5.44 | 4.73 | 28.98 | 0.42 | 39.56 | | DeepSeek-V4-Pro-NVFP4 | SGLang | TEP8 | 3.13 | 10.13 | 28.60 | 0.83 | 42.69 | Table 3. Where the Snapshot restore time goes, by stage, in seconds. Together, the four stages make up the Snapshot column in Table 1 Figure 5 shows the restore-time breakdown for vLLM. Benchmarking tools and reproduction instructions https://github.com/ai-dynamo/snapshot/blob/main/docs/development/benchmarks-guide.md are available in the repository. Get started To try Snapshot with a GPU workload on Kubernetes, start with the installation instructions and usage guides in the NVIDIA Snapshot repository https://github.com/ai-dynamo/snapshot . Try it out with NVIDIA Dynamo to restore workers by following the user guide https://docs.nvidia.com/dynamo/latest/kubernetes/model-deployment/model-loading/dynamo-snapshot-experimental prerequisites . To discuss an integration or contribute feedback, open an issue https://github.com/ai-dynamo/snapshot/issues/new/choose or join the NVIDIA Dynamo Slack workspace http://ai-dynamo.org/slack and visit the snapshot-dev channel. Visit the NVIDIA booth 170 at KubeCon https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/ for Kubernetes deep dives and live demos.