Show HN: Warmctl – Fast cold starts for vLLM inference

TensorRails released warmctl, a beta GPU checkpoint/restore tool for vLLM that cuts cold-start times by snapshotting a warmed-up engine and restoring it in seconds on hosts with the same GPUs and NVIDIA driver branch. On 8×H100, GLM-5.3-Flash starts serving in 37 s with `warmctl up` versus a 650 s vLLM cold start, and the public catalog lists 21 models across H100, H200, B200, B300, L40S and RTX PRO 6000. The tool uses CRIU for process trees, cuda-checkpoint for GPU state and NVIDIA cuInterpose to preserve NCCL and symmetric memory, making scale-to-zero, fast autoscaling and spot GPU replacement practical for large models.

warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine and restores it in seconds on any host with the same GPUs and NVIDIA driver branch, so a new replica skips weight loading, torch.compile, kernel autotuning and CUDA graph capture. GLM-5.3-Flash on 8×H100: vLLM cold start 650 s, warmctl up 37 s, then stock serving speed. That makes scale to zero, fast autoscaling and spot GPU replacement practical for large models. Docs https://warmctl.io/docs/ · Catalog https://warmctl.io/catalog/ · Blog https://warmctl.io/blog/ · Status and limits status-and-limits · Alternatives https://warmctl.io/docs/explanation/alternatives/ Linux GPU host, Docker with the NVIDIA runtime, root: curl -fsSL https://warmctl.io/install.sh | sh newest release in /opt/warmctl sysctl -w kernel.io uring disabled=2 CRIU cannot dump io uring warmctl doctor check the host and its GPUs warmctl catalog models this host can restore warmctl up qwen3-8b --listen 0.0.0.0:8000 pull, restore, serve The public catalog https://warmctl.io/catalog/ has 21 models on H100, H200, B200, B300, L40S and RTX PRO 6000. Any other model vLLM serves with sleep mode, build yourself tutorial https://warmctl.io/docs/tutorials/build-your-own/ : warmctl build --profile qwen3-8b-tp1 --model-dir /models/qwen3-8b --gpus 0 --output /snaps/qwen3-8b warmctl push /snaps/qwen3-8b ghcr.io/you/qwen3-8b warmctl up ghcr.io/you/qwen3-8b warmctl up to serving, snapshot in page cache, against a vLLM cold start weights on local disk, empty compile caches . | Model | GPUs | Image | warmctl up | vLLM cold start | |---|---|---|---|---| | GLM-5.3-Flash | 8×H100 | 404 GiB | 37 s | 650 s | | DeepSeek-V4.1-Flash | 8×H100 | 678 GiB | 78 s | 357 s | | DeepSeek-V4.1-Flash-1017 | 8×B300 | 729 GiB | 76 s | ~1026 s | | DeepSeek-V4.1-Flash-1017, DEP8 | 8×B200 | 809 GiB | 84 s | 1077 s | | DeepSeek-V4.1-Flash-1017, DEP8 | 8×B300 | 851 GiB | 88 s | 930 s | | gpt-oss-120b | 1×B200 | 78 GiB | 13 s | 351 s | | Gemma 4 26B-A4B | 1×B200 | 60 GiB | 12 s | ~330 s | | Qwen3-32B | 1×B200 | 72 GiB | 13 s | ~300 s | | Qwen3.6-35B-A3B | 1×B200 | 46 GiB | 10 s | ~240 s | | Qwen3-8B | 1×H100 | 23 GiB | 9 s | ~80 s | ~ is the time to a ready engine while building the snapshot; where we measured both, a plain cold start was 1–24% shorter. On B200, GLM-5.3-Flash restores in 44 s 8×B200 and DeepSeek-V4.1-Flash in 73 s 4×B200 . Every platform is in the catalog https://warmctl.io/catalog/ . GLM-5.3-Flash on 8×H100 by phase, from page cache and from a cold network file share about 10 GB/s : | Phase | Page cache | File share, cold | |---|---|---| | CRIU restores the process tree | 8.1 s | 15.3 s | | GPU state restored | 10.5 s | 13.3 s | | Weights reloaded 320 GiB | 12.3 s | 19.3 s | | Serving | 37.3 s | 56.8 s | Restored engines serve at stock vLLM speed or faster GLM-5.3-Flash: 100–136% of stock throughput . build starts vLLM in a container, warms it over every shape in the profile and checkpoints the processes CRIU , the GPUs cuda-checkpoint and the weights. push stores the snapshot as an OCI artifact, one per platform. up checks the host, restores everything at the same GPU addresses, runs a canary against the original engine and opens an OpenAI-compatible API. Multi-GPU engines keep NCCL and symmetric memory through NVIDIA cuInterpose. Details: how a restore works https://warmctl.io/docs/explanation/how-it-works/ , architecture https://github.com/TensorRails/warmctl/blob/main/docs/architecture.md , shared GPU memory https://github.com/TensorRails/warmctl/blob/main/docs/interpose.md , registries and caches https://github.com/TensorRails/warmctl/blob/main/docs/distribution.md . warmctl is in beta: releases are GitHub pre-releases, and every user-facing change is in the CHANGELOG https://github.com/TensorRails/warmctl/blob/main/CHANGELOG.md . Each restore is checked before it serves: the host must match the snapshot before anything is downloaded, every file carries its SHA-256, every weights allocation is compared by a GPU fingerprint, and a canary prompt must get the original engine's answer bitwise for large models . | Works today | Not yet | |---|---| | vLLM | SGLang, TensorRT-LLM, TGI, LMDeploy planned | | 1 to 8 GPUs on one node, tensor and expert parallel | Engines that span several nodes | | H100, H200, B200, B300, L40S, RTX PRO 6000 | Another GPU model or count, or another driver branch: build a snapshot for that platform | | Docker hosts, Kubernetes, KServe | GPUs in confidential-computing mode cuda-checkpoint does not support it | | Any OCI registry, or a shared file system | | The warmctl runtime image, registry.warmctl.io/warmctl/runtime:latest anonymous pull; pin a release tag such as :v2.0.0-beta.15 for a cluster that must not change under you , is the catalog's vLLM image with warmctl on top make pod-image builds it . In a privileged pod it runs the same warmctl up NAME and restores in place, with readiness on /health : kubectl apply -f deploy/kubernetes/warmctl.yaml deploy/kubernetes https://github.com/TensorRails/warmctl/blob/main/deploy/kubernetes also holds node preparation, a KEDA autoscaler and a KServe LLMInferenceService preset; details in docs/kubernetes.md https://github.com/TensorRails/warmctl/blob/main/docs/kubernetes.md . Tutorials: Kubernetes https://warmctl.io/docs/tutorials/kubernetes/ , KServe https://warmctl.io/docs/tutorials/kserve/ . Measured on GPU clusters: GLM-5.3-Flash on 8×H100 restores in 58 s from a shared file system post https://warmctl.io/blog/boosting-dedicated-inference/ , and a new KServe replica of Qwen3-8B on an L40S serves in 14 s post https://warmctl.io/blog/kserve-llm-replicas-in-seconds/ . schema: warmctl.profile/v1 id: llama31-8b-tp1 model: {id: meta-llama/Llama-3.1-8B-Instruct, revision: 0e9e39f2} parallel: {tp: 1} hardware: h100-80gb-sxm vllm: {max-model-len: 8192, kv-cache-memory-bytes: 21474836480} vllm: takes any vllm serve flag; the rest comes from profiles/ defaults.yaml . Linux x86-64 with glibc, Docker with the NVIDIA runtime, root, driver 580+, kernel.io uring disabled=2 . A snapshot restores under any release of the driver branch it was built on 580.x, 610.x . Proven on H100 SXM, H200, B200, B300, L40S and RTX PRO 6000. warmctl hardware lists the GPU models warmctl knows and marks this host's; another model is a row in a hardware.yaml next to your profiles --profile-dir . Keep warmctl , cuinterpose/ and the versioned dmtcp/ directory from the linux-amd64 archive side by side; the installer does this automatically. Linux arm64 archives include DMTCP but no cuInterpose yet and are CPU-tested, not GPU-qualified. The native DMTCP packages require Debian 12-compatible glibc = 2.36 , libstdc++ and libatomic libstdc++6 and libatomic1 on Debian/Ubuntu . macOS builds run the registry side: catalog , images , mirror , push , pull . Source builds use make build cuinterpose ; see DMTCP runtime installation and upgrades https://github.com/TensorRails/warmctl/blob/main/docs/dmtcp.md . Every release is signed and has build provenance verify https://github.com/TensorRails/warmctl/blob/main/RELEASING.md verify-the-cli-release . make test · make lint · make e2e-test CPU, Docker, root · make release the release archives as a local snapshot . Releases: RELEASING.md https://github.com/TensorRails/warmctl/blob/main/RELEASING.md . GPU acceptance: docs/gpu-runbook.md https://github.com/TensorRails/warmctl/blob/main/docs/gpu-runbook.md . Apache-2.0, including the bundled NVIDIA cuInterpose build NOTICE https://github.com/TensorRails/warmctl/blob/main/NOTICE .