{"slug": "show-hn-warmctl-fast-cold-starts-for-vllm-inference", "title": "Show HN: Warmctl – Fast cold starts for vLLM inference", "summary": "TensorRails released warmctl, a beta GPU checkpoint/restore tool for vLLM that cuts cold-start times by snapshotting a warmed-up engine and restoring it in seconds on hosts with the same GPUs and NVIDIA driver branch. On 8×H100, GLM-5.3-Flash starts serving in 37 s with `warmctl up` versus a 650 s vLLM cold start, and the public catalog lists 21 models across H100, H200, B200, B300, L40S and RTX PRO 6000. The tool uses CRIU for process trees, cuda-checkpoint for GPU state and NVIDIA cuInterpose to preserve NCCL and symmetric memory, making scale-to-zero, fast autoscaling and spot GPU replacement practical for large models.", "body_md": "warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine\nand restores it in seconds on any host with the same GPUs and NVIDIA driver\nbranch, so a new replica skips weight loading, torch.compile, kernel\nautotuning and CUDA graph capture. GLM-5.3-Flash on 8×H100: vLLM cold start\n650 s, `warmctl up` 37 s, then stock serving speed. That makes scale to zero,\nfast autoscaling and spot GPU replacement practical for large models.\n\n[Docs](https://warmctl.io/docs/) · [Catalog](https://warmctl.io/catalog/) ·\n[Blog](https://warmctl.io/blog/) · [Status and limits](#status-and-limits) ·\n[Alternatives](https://warmctl.io/docs/explanation/alternatives/)\n\nLinux GPU host, Docker with the NVIDIA runtime, root:\n\n```\ncurl -fsSL https://warmctl.io/install.sh | sh  # newest release in /opt/warmctl\nsysctl -w kernel.io_uring_disabled=2           # CRIU cannot dump io_uring\nwarmctl doctor                                 # check the host and its GPUs\nwarmctl catalog                                # models this host can restore\nwarmctl up qwen3-8b --listen 0.0.0.0:8000      # pull, restore, serve\n```\n\nThe [public catalog](https://warmctl.io/catalog/) has 21 models on H100, H200,\nB200, B300, L40S and RTX PRO 6000. Any other model vLLM serves with sleep mode,\nbuild yourself ([tutorial](https://warmctl.io/docs/tutorials/build-your-own/)):\n\n```\nwarmctl build --profile qwen3-8b-tp1 --model-dir /models/qwen3-8b --gpus 0 --output /snaps/qwen3-8b\nwarmctl push /snaps/qwen3-8b ghcr.io/you/qwen3-8b\nwarmctl up ghcr.io/you/qwen3-8b\n```\n\n`warmctl up` to serving, snapshot in page cache, against a vLLM cold start\n(weights on local disk, empty compile caches).\n\n| Model | GPUs | Image | warmctl up | vLLM cold start | \n|---|---|---|---|---|\n| GLM-5.3-Flash | 8×H100 | 404 GiB | **37 s** | 650 s | \n| DeepSeek-V4.1-Flash | 8×H100 | 678 GiB | **78 s** | 357 s | \n| DeepSeek-V4.1-Flash-1017 | 8×B300 | 729 GiB | **76 s** | ~1026 s | \n| DeepSeek-V4.1-Flash-1017, DEP8 | 8×B200 | 809 GiB | **84 s** | 1077 s | \n| DeepSeek-V4.1-Flash-1017, DEP8 | 8×B300 | 851 GiB | **88 s** | 930 s | \n| gpt-oss-120b | 1×B200 | 78 GiB | **13 s** | 351 s | \n| Gemma 4 26B-A4B | 1×B200 | 60 GiB | **12 s** | ~330 s | \n| Qwen3-32B | 1×B200 | 72 GiB | **13 s** | ~300 s | \n| Qwen3.6-35B-A3B | 1×B200 | 46 GiB | **10 s** | ~240 s | \n| Qwen3-8B | 1×H100 | 23 GiB | **9 s** | ~80 s | \n\n~ is the time to a ready engine while building the snapshot; where we\nmeasured both, a plain cold start was 1–24% shorter. On B200, GLM-5.3-Flash\nrestores in 44 s (8×B200) and DeepSeek-V4.1-Flash in 73 s (4×B200).\nEvery platform is in the [catalog](https://warmctl.io/catalog/).\n\nGLM-5.3-Flash on 8×H100 by phase, from page cache and from a cold network file share (about 10 GB/s):\n\n| Phase | Page cache | File share, cold | \n|---|---|---|\n| CRIU restores the process tree | 8.1 s | 15.3 s | \n| GPU state restored | 10.5 s | 13.3 s | \n| Weights reloaded (320 GiB) | 12.3 s | 19.3 s | \n| **Serving** | **37.3 s** | **56.8 s** | \n\nRestored engines serve at stock vLLM speed or faster (GLM-5.3-Flash: 100–136% of stock throughput).\n\n`build` starts vLLM in a container, warms it over every shape in the profile\nand checkpoints the processes (CRIU), the GPUs (cuda-checkpoint) and the\nweights. `push` stores the snapshot as an OCI artifact, one per platform.\n`up` checks the host, restores everything at the same GPU addresses, runs a\ncanary against the original engine and opens an OpenAI-compatible API.\nMulti-GPU engines keep NCCL and symmetric memory through NVIDIA cuInterpose.\n\nDetails: [how a restore works](https://warmctl.io/docs/explanation/how-it-works/),\n[architecture](https://github.com/TensorRails/warmctl/blob/main/docs/architecture.md),\n[shared GPU memory](https://github.com/TensorRails/warmctl/blob/main/docs/interpose.md),\n[registries and caches](https://github.com/TensorRails/warmctl/blob/main/docs/distribution.md).\n\nwarmctl is in beta: releases are GitHub pre-releases, and every user-facing\nchange is in the [CHANGELOG](https://github.com/TensorRails/warmctl/blob/main/CHANGELOG.md). Each restore is checked before it\nserves: the host must match the snapshot before anything is downloaded, every\nfile carries its SHA-256, every weights allocation is compared by a GPU\nfingerprint, and a canary prompt must get the original engine's answer\n(bitwise for large models).\n\n| Works today | Not yet | \n|---|---|\n| vLLM | SGLang, TensorRT-LLM, TGI, LMDeploy (planned) | \n| 1 to 8 GPUs on one node, tensor and expert parallel | Engines that span several nodes | \n| H100, H200, B200, B300, L40S, RTX PRO 6000 | Another GPU model or count, or another driver branch: build a snapshot for that platform | \n| Docker hosts, Kubernetes, KServe | GPUs in confidential-computing mode (cuda-checkpoint does not support it) | \n| Any OCI registry, or a shared file system |  | \n\nThe warmctl runtime image, `registry.warmctl.io/warmctl/runtime:latest`\n(anonymous pull; pin a release tag such as `:v2.0.0-beta.15` for a cluster\nthat must not change under you), is the catalog's vLLM image with warmctl on\ntop (`make pod-image` builds it).\nIn a privileged pod it runs the same `warmctl up NAME` and restores in place,\nwith readiness on `/health`:\n\n```\nkubectl apply -f deploy/kubernetes/warmctl.yaml\n```\n\n[deploy/kubernetes](https://github.com/TensorRails/warmctl/blob/main/deploy/kubernetes) also holds node preparation, a KEDA\nautoscaler and a KServe LLMInferenceService preset; details in [docs/kubernetes.md](https://github.com/TensorRails/warmctl/blob/main/docs/kubernetes.md).\nTutorials: [Kubernetes](https://warmctl.io/docs/tutorials/kubernetes/),\n[KServe](https://warmctl.io/docs/tutorials/kserve/). Measured on GPU clusters:\nGLM-5.3-Flash on 8×H100 restores in 58 s from a shared file system\n([post](https://warmctl.io/blog/boosting-dedicated-inference/)), and a new\nKServe replica of Qwen3-8B on an L40S serves in 14 s\n([post](https://warmctl.io/blog/kserve-llm-replicas-in-seconds/)).\n\n```\nschema: warmctl.profile/v1\nid: llama31-8b-tp1\nmodel: {id: meta-llama/Llama-3.1-8B-Instruct, revision: 0e9e39f2}\nparallel: {tp: 1}\nhardware: [h100-80gb-sxm]\nvllm: {max-model-len: 8192, kv-cache-memory-bytes: 21474836480}\n```\n\n`vllm:` takes any `vllm serve` flag; the rest comes from\n`profiles/_defaults.yaml`.\n\nLinux x86-64 with glibc, Docker with the NVIDIA runtime, root, driver 580+,\n`kernel.io_uring_disabled=2`. A snapshot restores under any release of the\ndriver branch it was built on (580.x, 610.x). Proven on H100 SXM, H200, B200,\nB300, L40S and RTX PRO 6000. `warmctl hardware` lists the GPU models warmctl knows and marks this\nhost's; another model is a row in a `hardware.yaml` next to your profiles\n(`--profile-dir`).\nKeep `warmctl`, `cuinterpose/` and the versioned `dmtcp/` directory from the\nlinux-amd64 archive side by side; the installer does this automatically.\nLinux arm64 archives include DMTCP but no cuInterpose yet and are CPU-tested,\nnot GPU-qualified. The native DMTCP packages require Debian 12-compatible\nglibc (>= 2.36), libstdc++ and libatomic (`libstdc++6` and `libatomic1` on\nDebian/Ubuntu). macOS builds run the registry side: `catalog`,\n`images`, `mirror`, `push`, `pull`. Source builds use `make build cuinterpose`;\nsee [DMTCP runtime installation and upgrades](https://github.com/TensorRails/warmctl/blob/main/docs/dmtcp.md).\nEvery release is signed and has build provenance\n([verify](https://github.com/TensorRails/warmctl/blob/main/RELEASING.md#verify-the-cli-release)).\n\n`make test` · `make lint` · `make e2e-test` (CPU, Docker, root) ·\n`make release` (the release archives as a local snapshot). Releases:\n[RELEASING.md](https://github.com/TensorRails/warmctl/blob/main/RELEASING.md). GPU acceptance:\n[docs/gpu-runbook.md](https://github.com/TensorRails/warmctl/blob/main/docs/gpu-runbook.md).\n\nApache-2.0, including the bundled NVIDIA cuInterpose build ([NOTICE](https://github.com/TensorRails/warmctl/blob/main/NOTICE)).", "url": "https://wpnews.pro/news/show-hn-warmctl-fast-cold-starts-for-vllm-inference", "canonical_source": "https://github.com/TensorRails/warmctl", "published_at": "2026-11-10 22:28:15+00:00", "updated_at": "2026-11-10 22:52:30.159711+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "large-language-models", "mlops", "developer-tools"], "entities": ["warmctl", "TensorRails", "vLLM", "NVIDIA", "GLM-5.3-Flash", "DeepSeek-V4.1-Flash", "Qwen3-8B", "CRIU"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-warmctl-fast-cold-starts-for-vllm-inference", "markdown": "https://wpnews.pro/news/show-hn-warmctl-fast-cold-starts-for-vllm-inference.md", "text": "https://wpnews.pro/news/show-hn-warmctl-fast-cold-starts-for-vllm-inference.txt", "jsonld": "https://wpnews.pro/news/show-hn-warmctl-fast-cold-starts-for-vllm-inference.jsonld"}}