How we cut inference cold start to seconds with NVIDIA Dynamo Photoroom cut GPU inference cold starts from roughly 220 seconds to 35–45 seconds using NVIDIA Dynamo Snapshot, which freezes a fully loaded inference process with NVIDIA's cuda-checkpoint and CRIU and restores it on any GPU in a Kubernetes fleet. Photoroom wired the approach into its LitServe serving framework and ArgoCD deployment pipeline, targeting the serial model-loading cold start that costs tens of seconds of library imports, up to a minute to download a ~55 GB model at ~1 GB/s, and up to ~40 seconds of warm-up for its regionally compiled image-edit pipeline. The restore depends on NVIDIA driver r580 or later, the first version that allows a checkpoint to be restored onto a different physical GPU. Photoroom cut GPU inference cold starts from ~220s to 35–45s using NVIDIA Dynamo Snapshot. The approach snapshots a fully-loaded process with NVIDIA's cuda-checkpoint and CRIU, then restores it on any GPU in a Kubernetes fleet. Here's how we wired it into LitServe and ArgoCD. At Photoroom, we run dozens of computer vision and generative AI models in production. Every background removal, studio scene, retouch, or expand is powered by its own model, each with its own latency budget and traffic pattern. Some models run under continuous load; others are bursty or rarely touched. For the quiet ones, we scale to zero, because there's no sense holding an expensive GPU idle for a model that gets a handful of requests an hour. Scale-to-zero has one well-known downside: the cold start. The first request after a scale-up pays the full cost of bringing a replica online, and for a large diffusion or transformer model that can mean several minutes. This post covers how we cut that time to seconds using GPU memory checkpointing. What actually happens when a container serving with LitServe starts LitServe https://lightning.ai/litserve is an open-source, PyTorch-native serving framework from LightningAI https://lightning.ai/ : you just fill in setup to load the model and predict to run inference, and it handles the HTTP server, batching, and workers. We reach for it whenever we want to get a model served quickly. The trade-off: it loads models sequentially, one setup run start to finish before the server accepts a request, whereas frameworks like Triton load them in parallel. That serial cold start is why we focus on LitServe here, and why a fast restore helps so much. When a LitServe container starts, it will not serve a single request until the model is fully loaded. It launches a worker process, runs your setup inside it, and only brings up the HTTP server once that worker reports READY. setup is where all the heavy lifting happens, roughly in this order: 1. Import the inference libraries PyTorch, TensorRT, diffusers, … . 2. Download the model weights. 3. Move the weights from host memory into GPU memory. 4. Optional Warm up the model with JIT / torch.compile or CUDA graph capture. Only then does the HTTP server come up and start serving. Where does the time actually go? On our setup: imports take about 15 s the ML libraries are surprisingly heavy to load ; downloading a ~55 GB model at ~1 GB/s ~8 Gbps is about a minute, or zero if the weights ship in the image or on a mounted volume; moving them host-to-GPU is only a second or two, because the link is PCIe Gen5 x16 ~50 GB/s ; and warm-up depends entirely on the model, from near-zero for an AOT-compiled engine to ~40 s for our regionally-compiled compile repeated blocks image-edit pipeline. Where a cold start's time goes: the setup phases and their rough durations For a typical large diffusion or transformer setup, expect tens of seconds of imports, seconds to a few minutes of download depending on the bandwidth to the storage that contains the weights, a second or two to load the weights onto the GPU, and anywhere from typically several minutes of warm-up. Put together, that is a few minutes on every cold start, and most of it the download and warm-up is repeated every single time. What is GPU memory checkpointing? What if we could freeze the process after all of this is done, and thaw it back on the next cold start? That's memory checkpointing. In short, GPU memory checkpointing freezes a fully-loaded inference process, including its GPU memory, to disk, and restores it on a later cold start instead of reloading and recompiling the model. Serverless platforms have used CPU-memory checkpointing for years, but GPU state was out of reach until NVIDIA shipped its cuda-checkpoint utility. It's been around for a little while, but only from driver r580 onward can a checkpoint be restored onto a different physical GPU than the one it was captured on, which is exactly what you need in a dynamic, autoscaled fleet. Memory checkpointing in one picture: producer and consumer How to add GPU memory checkpointing to a LitServe app NVIDIA Dynamo Snapshot NVIDIA Dynamo https://github.com/ai-dynamo/dynamo is a framework for orchestrating LLMs at scale, with backends for vLLM, SGLang, Tokenspeed and NVIDIA TensorRT LLM, and features like disaggregated serving, prefix-aware routing to maximize KV-cache hits, and efficient cross-node KV-cache transfer. In June 2026 NVIDIA shipped stable memory checkpointing for Dynamo v1.2 as a standalone Helm chart https://github.com/ai-dynamo/dynamo/tree/main/deploy/helm/charts/snapshot , after previewing it in March. Photoroom, a member of the NVIDIA Inception https://www.nvidia.com/en-us/startups/ program for startups, worked with NVIDIA to integrate this capability into their production infrastructure. It combines the CUDA driver's native checkpointing capability cuda-checkpoint command-line utility that saves and restores a running process's GPU state with CRIU https://criu.org/ open-source Linux tool that freezes a running process to disk and restores it to snapshot a fully-loaded process and restore it later, potentially on another node and another GPU. It's also native to Kubernetes. Requirements: a cluster with the Dynamo Snapshot v1.2.0 chart and agent https://github.com/ai-dynamo/dynamo/tree/main/deploy/helm/charts/snapshot installed driver r580+, a fast shared RWX volume, and the rest of their prerequisites https://github.com/ai-dynamo/dynamo/tree/main/deploy/helm/charts/snapshot prerequisites , plus your inference image built on a glibc 2.38+ base, so the CRIU / cuda-checkpoint binaries run. What we reuse and what we don't Dynamo exposes checkpointing at three levels: 1. A low-level snapshotctl CLI, meant for debugging and validation. 2. A DynamoCheckpoint CRD driven by the Dynamo operator, the primary interface that launches the producer Job and wires restore through a mutating webhook. 3. The snapshot-agent itself, which restores a process based on pod labels. We reuse only the agent. What we take from Dynamo Snapshot: only the snapshot-agent The current Dynamo operator's DynamoCheckpoint path assumes it owns the serving workload through its own CRDs DynamoGraphDeployment / DynamoComponentDeployment . Our models, however, are LitServe apps served as ArgoCD Rollouts from our own Helm chart, with specific canary steps, autoscaling and init containers that Dynamo's CRDs do not model, so we can't put the Rollout under the operator. Checkpointing can run outside the operator using standalone DynamoCheckpoint CRs , but its production path is coupled to Dynamo's serving stack. For our case, we install just the snapshot-agent from the Dynamo Snapshot chart, stamp its restore labels on our Rollout ourselves, and drive the produce step with our own Job the sync-wave flow below . How checkpoint and restore actually works A per-node agent DaemonSet drives checkpoint and restore, coordinating with the process through sentinel files on a shared control volume: 1. Once the process is fully loaded and serving, it writes a ready-for-checkpoint file. 2. The agent runs cuda-checkpoint to move GPU memory back to the host, then CRIU dumps the whole process to the shared volume. It writes to a temporary directory and atomically renames it only on success, so a partial dump is never mistaken for a valid checkpoint. 3. To restore, a placeholder pod starts, the agent CRIU-restores the process into it and toggles cuda-checkpoint back. The process resumes exactly where it was frozen, with the model already in VRAM. The checkpoint/restore handshake: the process and the agent coordinate through sentinel files The checkpoint/restore binaries criu , cuda-checkpoint , and the nsrestore namespace helper have to be present in the container, so we wrap our inference image in a thin one that copies those binaries in. The runtime identity, and therefore the checkpoint, is still driven by the normal inference image. In practice it is a few lines of Dockerfile, and the binaries come straight from the pinned, public snapshot-agent image, so we never build CRIU or the CUDA plugin ourselves: ARG BASE IMAGE ARG SNAPSHOT AGENT IMAGE=nvcr.io/nvidia/ai-dynamo/snapshot-agent:1.2.0 FROM ${SNAPSHOT AGENT IMAGE} AS snapshot FROM ${BASE IMAGE} must have glibc 2.38+ see below USER root CRIU runtime dependencies RUN apt-get update && apt-get install -y --no-install-recommends \ libbsd0 libcap2 libnet1 libnl-3-200 libnl-route-3-200 libprotobuf-c1 \ libgnutls30t64 libnftables1 iproute2 iptables procps uuid-runtime \ && rm -rf /var/lib/apt/lists/ copy the prebuilt binaries + CRIU plugins from the agent image COPY --from=snapshot /usr/local/sbin/criu /usr/local/sbin/criu COPY --from=snapshot /usr/local/sbin/cuda-checkpoint /usr/local/sbin/cuda-checkpoint COPY --from=snapshot /usr/local/bin/nsrestore /usr/local/bin/nsrestore COPY --from=snapshot /usr/local/lib/criu /usr/local/lib/criu RUN criu --version fail the build early if the base image's glibc is too old The same image runs on both the checkpoint-source pod and the restore pods; only the command differs. The source pod runs the real server, loads the model, and is dumped, while a restore pod runs a sleep placeholder that the agent CRIU-restores the checkpointed process into. They cannot use different images: CRIU restores the process's memory, but the restored process still references the executables and shared libraries on disk, so the restore filesystem has to match the source exactly. Connecting LitServe to Dynamo Snapshot Dynamo ships sentinel hooks for vLLM, SGLang and TensorRT LLM. For LitServe we add our own. It comes down to two pieces: a small lifecycle hook that speaks the sentinel protocol, and making setup checkpoint-safe . The hook waits until the server is serving, signals ready-for-checkpoint , and blocks until the agent is done. Where and how it runs is deliberate: in the main process PID 1 , as a daemon thread, and only after the server starts never inside setup . PID 1 is the process the agent checkpoints as a whole and the one it signals to exit on the source pod ; the daemon thread keeps it from blocking the server; and the timing guarantees we freeze a fully-loaded process, since LitServe marks a worker READY only after setup returns. snapshot lifecycle.py -- speaks Dynamo's sentinel protocol from LitServe def start checkpoint lifecycle port: int - None: only active in a checkpoint pod control dir set by the agent ; no-op otherwise control dir = os.environ.get "DYN SNAPSHOT CONTROL DIR" if not control dir: return threading.Thread target= run, args= control dir, port , daemon=True .start def run control dir, port : 1. wait until every worker is READY /health 200 -- the state we want frozen while not health ok port : time.sleep 1 2. tell the agent we are ready to be checkpointed Path control dir, "ready-for-checkpoint" .write text "ready" 3. block until the agent writes snapshot-complete or restore-complete while not snapshot complete exists or restore complete exists : time.sleep 0.1 4. which sentinel fired tells us our role if snapshot complete exists : source pod: exit 0 so the producer Job completes and frees the GPU os.kill 1, signal.SIGINT PID 1 is LitServe - clean exit 0 restored pod: nothing to do, just keep serving The helpers are deliberately tiny: - health ok port does a local GET