{"slug": "cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore", "title": "Cold starting a 32B model in 1.1s with GPU checkpoint/restore", "summary": "Numerata Engineering reported on October 19, 2026 that its GPU checkpoint/restore engine cold starts Qwen3-0.6B on vLLM in 0.15 seconds and Qwen3-32B on vLLM in 1.1 seconds, versus 13 to 17 seconds for existing GPU snapshot approaches that spin up a 0.6B model at roughly 100 MB/s. The company said it avoids snapshotting the CPU process, using its Lupine client to hand over a live GPU pointer list at checkpoint time and restoring allocations at identical memory addresses by working around libcuda's opaque allocation APIs. The result matters because it moves GPU cloud cold starts toward true serverless GPUs by exploiting PCIe Gen5's 64 GB/s and Grace Hopper NVLink-C2C's 400+ GB/s of usable bandwidth.", "body_md": "You know what I want out of a GPU cloud? Instant cold starts. That enables true serverless GPUs.\n\nOCTOBER 19, 2026 · NUMERATA ENGINEERING\n\nState-of-the-art GPU snapshots have shown that restoring from a snapshot beats booting a container and initializing PyTorch from scratch. However, they still take 13 to 17 seconds to spin up a tiny 0.6B model, a paltry ~100 MB/s transfer rate. Meanwhile, PCIe Gen5 gives us 64 GB/s and Grace Hopper's NVLink-C2C gives us 400+ GB/s of usable bandwidth. What gives?\n\nOver the past few weeks, we went down the rabbit hole of GPU checkpoint/restore to max out the hardware link. Today, our checkpoint/restore engine cold starts Qwen3-0.6B on vLLM in 0.15 seconds and Qwen3-32B on vLLM in 1.1 seconds.\n\nHere is where all the time was actually going in traditional GPU snapshots, and how we engineered around it.\n\n1. Scrubbing the CPU process (by not snapshotting it)\n\nIf you want to checkpoint a GPU workload today, the standard recipe combines CRIU with NVIDIA’s cuda-checkpoint utility. Because traditional GPU clouds stuff your Python interpreter, web server, threads, and CUDA driver into a single monolithic container, snapshotting the GPU means freezing the entire container.\n\nThat API actually does way too much, and is quite slow as a result:\n\nPython is heavy: You have to freeze and dump gigabytes of Python heap, shared libraries, and open sockets before you can release the GPU.\n\ncuda-checkpoint stages through host RAM: NVIDIA’s tool is a black box that locks the process and copies all your VRAM into internal CPU staging buffers through a slow, single-threaded bottleneck.\n\nZero overlap: You must wait for cuda-checkpoint to finish evacuating the GPU before CRIU can dump the CPU, and on restore you must wait for CRIU to finish restoring the CPU before a single byte starts uploading back to the GPU.\n\nWalking page tables is slow: Scanning virtual memory mappings and page tables to discover what memory is live and dirty burns precious seconds before data transfer even starts.\n\nKeeping the CPU instance alive gives us two big shortcuts:\n\nWe never walk memory: Because the Lupine client intercepts allocation calls on the way out, it already holds the full list of every live GPU pointer and its byte size. At checkpoint time, we pause incoming calls, the client hands over its pointer list, and transfers start on millisecond one.\n\nLive socket handoff: When we rotate the GPU instance underneath you, both ends pause their HTTP/2 stream cleanly between frames. Once the new GPU instance is warm, our regional gateway splices your existing connection onto a resume tunnel. To PyTorch on the CPU instance, rotating onto a new physical GPU just looks like one CUDA call took an extra 200ms.\n\n2. Putting pointers back where we found them\n\nKeeping the CPU instance up narrows our problem to a simple contract: save every GPU allocation and put the bytes back at exactly the same memory address.\n\n“Exactly” is the hard part. Modern ML workloads like PyTorch, torch.compile, and CUDA graphs embed raw 64-bit GPU pointers directly inside VRAM buffers. If a tensor lived at 0x7f04a0000000 before the checkpoint, it must land at 0x7f04a0000000 on restore. Move it by a single page and internal pointers break. And scanning a 32B model's opaque floating-point soup to guess which 8 bytes are secretly a pointer is a non-starter.\n\nWhy we had to go around libcuda.so\n\nIt turns out libcuda’s APIs are too blunt to do this though, and they come with a ton of baggage:\n\nOpaque do-everything APIs: Standard cuMemAlloc gives you zero control over what address comes back. Even if you use CUDA’s lower-level virtual memory API (cuMemAddressReserve), the target address parameter is just a hint. If libcuda dislikes the alignment, it silently picks a different address. Worse, cuMemAlloc packs small tensors into shared 2 MiB chunks inside 32 MiB address blocks, so the public APIs refuse to reserve individual tensors at smaller offsets.\n\nToo much userspace baggage: Every public allocation call updates internal lookup trees, context tables, and per-pointer bookkeeping inside libcuda.so on top of slow driver page allocation. That's not very cash money.\n\nSo our first prototype went around libcuda.so completely. If you only read NVIDIA’s public CUDA Driver API docs, cuMemAddressReserve is as low as you can go, and it treats your target address as a polite suggestion. To find out how libcuda.so actually tells the kernel where to put a page, we had to go spelunking through NVIDIA’s open-gpu-kernel-modules repository.\n\nUVM_CREATE_EXTERNAL_RANGE: Unlike cuMemAddressReserve, this command does not take a hint. You pass a base and length, and the kernel reserves that exact [base, base + length) range in the GPU’s virtual address space.\n\nUVM_MAP_EXTERNAL_ALLOCATION: Maps physical VRAM pages directly into the GPU page tables at that reserved range, using the raw /dev/nvidiactl file descriptor and internal Resource Manager handles created by NV_ESC_RM_ALLOC.\n\nWith these two totally cavalier, undocumented incantations, we blast our way to mapping physical pages onto their exact target address in ~1 millisecond.\n\nThat worked, right up until we tried to copy the weights back in. Because we created those mappings behind libcuda.so’s back, its internal lookup tables had no idea our restored pointers existed. Calling cuMemcpyHtoD immediately returned CUDA_ERROR_INVALID_VALUE. (Excuse me, I just mapped that.) Working around libcuda’s bouncer required uploading data into a temporary VRAM staging buffer and launching a GPU copy kernel to move it into the final address, cutting effective bandwidth in half.\n\nStripping the baggage\n\nBy looking at how the driver manages virtual addresses under the hood, we got rid of both libcuda’s per-tensor baggage and the staging buffer. Normally libcuda tracks every tiny allocation so it can answer pointer-size queries. But in Lupine, the CPU client already tracks individual tensor bounds locally! The GPU server doesn't need per-tensor bookkeeping at all. We merge adjacent allocations into coarse 32 MiB virtual memory blocks at the exact target addresses so cuMemcpyHtoDAsync copies weights straight into place in a single pass.\n\n3. Hashing on the GPU\n\nOnce address mapping takes ~1ms and the two-copy tax is gone, we have a pure streaming pipeline problem. We replaced sequential 1 MiB transfers with a pipeline of reusable pinned memory buffers and 8 concurrent workers streaming 64 MiB chunks.\n\nTo optimize storage, we also content-address every 64 MiB chunk by its cryptographic hash. Because base model weights and untouched memory pages stay identical across snapshots and across users running the same model, content addressing lets us deduplicate traffic right at the beginning and skip transferring chunks that are already present in storage or local cache.\n\nHowever, content addressing at multi-gigabyte-per-second speeds poses a major problem: CPU hashing throughput isn't fast enough. Computing SHA-256 on a 32B BF16 model takes over 30 CPU-seconds (37.76 seconds total), turning the CPU into a massive bottleneck.\n\nHow do we fix that? At checkpoint and cold-start time, the weights are already sitting on a massively parallel processor: the GPU.\n\nWe switched our chunk store to BLAKE3, whose tree structure is easy to parallelize across GPU thread blocks, and wrote custom CUDA kernels to hash chunks directly on the GPU. Our CUDA BLAKE3 kernel hashes GPU memory at 878 GB/s (73 ms for an entire 32B BF16 model) on a side stream, returning just a 32-byte digest per chunk to the host. When we took all the above into account, our 32B cold start time dropped from 37.76s to 1.11s.\n\n4. Tricking NVIDIA's Resource Manager\n\nPutting raw VRAM bytes back at the right addresses restores your model weights and KV caches. But that is not enough, because NVIDIA also has a Resource Manager that owns the driver's bookkeeping.\n\nWhen a process uses CUDA, that bookkeeping lives in two places: userspace structs inside libcuda.so on the CPU, and kernel-side state inside the NVIDIA Resource Manager behind /dev/nvidia*. The key insight is that this driver bookkeeping is tiny compared to an entire application process. For the userspace half, we use the exact same approach on the CPU that we used on the GPU: we track the worker's CPU allocations and restore those few megabytes directly into the new process memory in ~122 milliseconds. And for the kernel-side Resource Manager state, we record the ioctl calls made against /dev/nvidia* and replay them on the new instance:\n\nIntercept device calls: We hook open, ioctl, and mmap on /dev/nvidia* via LD_PRELOAD as the worker runs, logging every Resource Manager allocation and mapping so we can replay them against the fresh driver.\n\nSkip unnecessary ioctls: We also optimize the recorded ioctl sequence to strip out calls that are unnecessary on restore, since 90% of startup ioctls are just libcuda asking read-only questions whose answers are already cached in CPU memory.\n\nOverlap CPU & GPU restore: We stream model weights into the GPU at the same time we restore the 122ms of tracked CPU bookkeeping into process memory, then tell the NVIDIA Resource Manager to attach our already-loaded VRAM directly to the saved handle IDs.\n\n5. Fixing the disk and network bottleneck\n\nMoving a 32B model out of VRAM in 1 second doesn't help much if you then spend 30 seconds uploading it to S3 over HTTP. On a standard cloud, a server might only have a 10 Gb/s network link, which quickly becomes the bottleneck when migrating between machines.\n\nTo solve this across the cluster, we built our storage tier on CoreWeave Dedicated VAST Storage. Instead of sitting behind high-latency HTTP object endpoints, VAST exposes a co-located, all-flash NVMe-over-Fabrics (NVMe-oF) shared filesystem directly to our GPU nodes over a 400 Gb/s RDMA fabric.\n\nWe combine local memory and VAST into a three-tier cache (VRAM ↔ Host RAM ↔ VAST):\n\nTier 1 (Local Host RAM in ~1s): When we checkpoint a model, we stream it straight from VRAM into local Host RAM. As soon as that 1-second copy finishes, the GPU is immediately free to serve the next request.\n\nTier 2 (Asynchronous flush to VAST): In the background, worker threads flush the 64 MiB BLAKE3 chunks from Host RAM to VAST while keeping a local LRU cache on the machine. Our scheduler tracks which machines hold warm chunks so returning requests can be routed back to the same node and restore from RAM at light speed.\n\nTier 3 (Line-rate cross-node reads over RDMA): When a workload needs to cold start or live-migrate onto a completely different machine, 8 parallel workers pull the 64 MiB chunks straight off VAST over the 400 Gb/s RDMA fabric, saturating the network link without S3 or HTTP overhead.\n\n“But wait, isn’t the line rate of VAST 400 Gb/s? How’d you exceed 50 GB/s?”\n\nWell, eagle-eyed Hacker News reader, we compress the model on the GPU before storage using NVIDIA’s nvCOMP!\n\nRight after computing the raw BLAKE3 digest of each 64 MiB chunk on the GPU, we run nvCOMP’s batched ANS (Asymmetric Numeral Systems) codec in its 2-byte float (FLOAT16) mode to entropy-code the tightly clustered BF16 exponents. On real BF16 model weights, nvCOMP shrinks every chunk to 0.667x its original size (a full one-third smaller) at 1.19 TB/s compression and 1.87 TB/s decompression.\n\nEach compressed frame is still named by the BLAKE3 hash of its raw bytes so content deduplication keeps working unchanged. Because every checkpoint and VAST transfer moves a third fewer bytes over the wire (42.7 GB instead of 64 GB for a 32B model), and decompression runs at nearly 1.9 TB/s straight into our exact-address VRAM mappings, our effective model throughput blows right past the 50 GB/s physical wire limit to 60+ GB/s.\n\nOnce you can move live GPU state in and out of VRAM in under a second, there's some neato stuff you can do that wasn't possible before:\n\nTrue scale to zero: Most \"serverless\" GPU deployments still keep warm GPUs burning money 24/7 because a 30-second cold start ruins user experience. When cold starts take under a second, you can scale idle endpoints all the way down to zero and spin them back up as requests arrive, saving you money by making cold starts affordable.\n\nTurn spot instances into reserved instances: Because your CPU instance lives separately from the GPU, we can put your CPU on a cheap reserved instance and your GPU on a spot instance. If the spot instance goes down, we checkpoint your VRAM to VAST and live-migrate you to a new GPU within a second. The CPU doesn't know any different, aside from a small blip in latency. Pay for spot instances, get reserved instances. Neat!\n\nFork/join training at will: Tired of pausing your training run for torch.save() and evals? Just snapshot your GPU in a second and send the snapshot to another GPU to run the eval while your training cluster keeps running.\n\nSub-second deploys without green/blue complexity: Instead of allocating double the GPUs, waiting minutes for weights to load and CUDA graphs to warm up, and draining traffic over, checkpoint/restore lets you resume state on the fly with zero code changes.\n\nAnd there we go, paid for the whole bus, use the whole bus. Now we can just dream about the day when someone lets us bolt a network card directly onto the HBM stacks.\n\nThat's it, there's no call to action, hope you enjoyed the post!", "url": "https://wpnews.pro/news/cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore", "canonical_source": "https://numerata.com/blog/post-gpu-cold-start-checkpoint-restore", "published_at": "2026-10-20 19:03:19+00:00", "updated_at": "2026-10-20 19:16:45.581410+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["Numerata Engineering", "Qwen3-0.6B", "Qwen3-32B", "vLLM", "NVIDIA", "cuda-checkpoint", "CRIU", "Lupine"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore", "markdown": "https://wpnews.pro/news/cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore.md", "text": "https://wpnews.pro/news/cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore.txt", "jsonld": "https://wpnews.pro/news/cold-starting-a-32b-model-in-1-1s-with-gpu-checkpoint-restore.jsonld"}}