# Show HN: Warmctl – Fast cold starts for vLLM inference

> Source: <https://github.com/TensorRails/warmctl>
> Published: 2026-11-10 22:28:15+00:00

warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine
and restores it in seconds on any host with the same GPUs and NVIDIA driver
branch, so a new replica skips weight loading, torch.compile, kernel
autotuning and CUDA graph capture. GLM-5.3-Flash on 8×H100: vLLM cold start
650 s, `warmctl up` 37 s, then stock serving speed. That makes scale to zero,
fast autoscaling and spot GPU replacement practical for large models.

[Docs](https://warmctl.io/docs/) · [Catalog](https://warmctl.io/catalog/) ·
[Blog](https://warmctl.io/blog/) · [Status and limits](#status-and-limits) ·
[Alternatives](https://warmctl.io/docs/explanation/alternatives/)

Linux GPU host, Docker with the NVIDIA runtime, root:

```
curl -fsSL https://warmctl.io/install.sh | sh  # newest release in /opt/warmctl
sysctl -w kernel.io_uring_disabled=2           # CRIU cannot dump io_uring
warmctl doctor                                 # check the host and its GPUs
warmctl catalog                                # models this host can restore
warmctl up qwen3-8b --listen 0.0.0.0:8000      # pull, restore, serve
```

The [public catalog](https://warmctl.io/catalog/) has 21 models on H100, H200,
B200, B300, L40S and RTX PRO 6000. Any other model vLLM serves with sleep mode,
build yourself ([tutorial](https://warmctl.io/docs/tutorials/build-your-own/)):

```
warmctl build --profile qwen3-8b-tp1 --model-dir /models/qwen3-8b --gpus 0 --output /snaps/qwen3-8b
warmctl push /snaps/qwen3-8b ghcr.io/you/qwen3-8b
warmctl up ghcr.io/you/qwen3-8b
```

`warmctl up` to serving, snapshot in page cache, against a vLLM cold start
(weights on local disk, empty compile caches).

| Model | GPUs | Image | warmctl up | vLLM cold start | 
|---|---|---|---|---|
| GLM-5.3-Flash | 8×H100 | 404 GiB | **37 s** | 650 s | 
| DeepSeek-V4.1-Flash | 8×H100 | 678 GiB | **78 s** | 357 s | 
| DeepSeek-V4.1-Flash-1017 | 8×B300 | 729 GiB | **76 s** | ~1026 s | 
| DeepSeek-V4.1-Flash-1017, DEP8 | 8×B200 | 809 GiB | **84 s** | 1077 s | 
| DeepSeek-V4.1-Flash-1017, DEP8 | 8×B300 | 851 GiB | **88 s** | 930 s | 
| gpt-oss-120b | 1×B200 | 78 GiB | **13 s** | 351 s | 
| Gemma 4 26B-A4B | 1×B200 | 60 GiB | **12 s** | ~330 s | 
| Qwen3-32B | 1×B200 | 72 GiB | **13 s** | ~300 s | 
| Qwen3.6-35B-A3B | 1×B200 | 46 GiB | **10 s** | ~240 s | 
| Qwen3-8B | 1×H100 | 23 GiB | **9 s** | ~80 s | 

~ is the time to a ready engine while building the snapshot; where we
measured both, a plain cold start was 1–24% shorter. On B200, GLM-5.3-Flash
restores in 44 s (8×B200) and DeepSeek-V4.1-Flash in 73 s (4×B200).
Every platform is in the [catalog](https://warmctl.io/catalog/).

GLM-5.3-Flash on 8×H100 by phase, from page cache and from a cold network file share (about 10 GB/s):

| Phase | Page cache | File share, cold | 
|---|---|---|
| CRIU restores the process tree | 8.1 s | 15.3 s | 
| GPU state restored | 10.5 s | 13.3 s | 
| Weights reloaded (320 GiB) | 12.3 s | 19.3 s | 
| **Serving** | **37.3 s** | **56.8 s** | 

Restored engines serve at stock vLLM speed or faster (GLM-5.3-Flash: 100–136% of stock throughput).

`build` starts vLLM in a container, warms it over every shape in the profile
and checkpoints the processes (CRIU), the GPUs (cuda-checkpoint) and the
weights. `push` stores the snapshot as an OCI artifact, one per platform.
`up` checks the host, restores everything at the same GPU addresses, runs a
canary against the original engine and opens an OpenAI-compatible API.
Multi-GPU engines keep NCCL and symmetric memory through NVIDIA cuInterpose.

Details: [how a restore works](https://warmctl.io/docs/explanation/how-it-works/),
[architecture](https://github.com/TensorRails/warmctl/blob/main/docs/architecture.md),
[shared GPU memory](https://github.com/TensorRails/warmctl/blob/main/docs/interpose.md),
[registries and caches](https://github.com/TensorRails/warmctl/blob/main/docs/distribution.md).

warmctl is in beta: releases are GitHub pre-releases, and every user-facing
change is in the [CHANGELOG](https://github.com/TensorRails/warmctl/blob/main/CHANGELOG.md). Each restore is checked before it
serves: the host must match the snapshot before anything is downloaded, every
file carries its SHA-256, every weights allocation is compared by a GPU
fingerprint, and a canary prompt must get the original engine's answer
(bitwise for large models).

| Works today | Not yet | 
|---|---|
| vLLM | SGLang, TensorRT-LLM, TGI, LMDeploy (planned) | 
| 1 to 8 GPUs on one node, tensor and expert parallel | Engines that span several nodes | 
| H100, H200, B200, B300, L40S, RTX PRO 6000 | Another GPU model or count, or another driver branch: build a snapshot for that platform | 
| Docker hosts, Kubernetes, KServe | GPUs in confidential-computing mode (cuda-checkpoint does not support it) | 
| Any OCI registry, or a shared file system |  | 

The warmctl runtime image, `registry.warmctl.io/warmctl/runtime:latest`
(anonymous pull; pin a release tag such as `:v2.0.0-beta.15` for a cluster
that must not change under you), is the catalog's vLLM image with warmctl on
top (`make pod-image` builds it).
In a privileged pod it runs the same `warmctl up NAME` and restores in place,
with readiness on `/health`:

```
kubectl apply -f deploy/kubernetes/warmctl.yaml
```

[deploy/kubernetes](https://github.com/TensorRails/warmctl/blob/main/deploy/kubernetes) also holds node preparation, a KEDA
autoscaler and a KServe LLMInferenceService preset; details in [docs/kubernetes.md](https://github.com/TensorRails/warmctl/blob/main/docs/kubernetes.md).
Tutorials: [Kubernetes](https://warmctl.io/docs/tutorials/kubernetes/),
[KServe](https://warmctl.io/docs/tutorials/kserve/). Measured on GPU clusters:
GLM-5.3-Flash on 8×H100 restores in 58 s from a shared file system
([post](https://warmctl.io/blog/boosting-dedicated-inference/)), and a new
KServe replica of Qwen3-8B on an L40S serves in 14 s
([post](https://warmctl.io/blog/kserve-llm-replicas-in-seconds/)).

```
schema: warmctl.profile/v1
id: llama31-8b-tp1
model: {id: meta-llama/Llama-3.1-8B-Instruct, revision: 0e9e39f2}
parallel: {tp: 1}
hardware: [h100-80gb-sxm]
vllm: {max-model-len: 8192, kv-cache-memory-bytes: 21474836480}
```

`vllm:` takes any `vllm serve` flag; the rest comes from
`profiles/_defaults.yaml`.

Linux x86-64 with glibc, Docker with the NVIDIA runtime, root, driver 580+,
`kernel.io_uring_disabled=2`. A snapshot restores under any release of the
driver branch it was built on (580.x, 610.x). Proven on H100 SXM, H200, B200,
B300, L40S and RTX PRO 6000. `warmctl hardware` lists the GPU models warmctl knows and marks this
host's; another model is a row in a `hardware.yaml` next to your profiles
(`--profile-dir`).
Keep `warmctl`, `cuinterpose/` and the versioned `dmtcp/` directory from the
linux-amd64 archive side by side; the installer does this automatically.
Linux arm64 archives include DMTCP but no cuInterpose yet and are CPU-tested,
not GPU-qualified. The native DMTCP packages require Debian 12-compatible
glibc (>= 2.36), libstdc++ and libatomic (`libstdc++6` and `libatomic1` on
Debian/Ubuntu). macOS builds run the registry side: `catalog`,
`images`, `mirror`, `push`, `pull`. Source builds use `make build cuinterpose`;
see [DMTCP runtime installation and upgrades](https://github.com/TensorRails/warmctl/blob/main/docs/dmtcp.md).
Every release is signed and has build provenance
([verify](https://github.com/TensorRails/warmctl/blob/main/RELEASING.md#verify-the-cli-release)).

`make test` · `make lint` · `make e2e-test` (CPU, Docker, root) ·
`make release` (the release archives as a local snapshot). Releases:
[RELEASING.md](https://github.com/TensorRails/warmctl/blob/main/RELEASING.md). GPU acceptance:
[docs/gpu-runbook.md](https://github.com/TensorRails/warmctl/blob/main/docs/gpu-runbook.md).

Apache-2.0, including the bundled NVIDIA cuInterpose build ([NOTICE](https://github.com/TensorRails/warmctl/blob/main/NOTICE)).