{"slug": "making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog", "title": "Making CUDA Checkpointing 15x Faster on a B300: A Worklog", "summary": "A worklog benchmarking NVIDIA's cuda-checkpoint on a B300 GPU found that tuning thread and buffer counts, NUMA placement and host-to-PCIe copy matching can make CUDA checkpointing and restore up to 15x faster, with NUMA alone accounting for up to 2x and some copies running 3x faster on one platform than another. The author identifies memory allocation and the number of copy buffers and threads as the two application-specific bottlenecks in cuda-checkpoint, and reports a possible 2x speedup from a version run under gVisor. The worklog concludes that performance is hard to predict without benchmarks and traces, and recommends parallelizing operations.", "body_md": "# Making CUDA Checkpointing 15x Faster on a B300: A Worklog\n\n# GPU Checkpointing\n\nGPU checkpointing lets an application save its GPU state and later pick up where it left off. This gives us more flexibility in how we run application on top of GPUs. An inference service can release them during quiet periods and resume when traffic returns [Modal: Serverless GPU scaling](https://modal.com/blog/truly-serverless-gpus)\n\n [Doubleword: Faster SGLang cold starts](https://blog.doubleword.ai/fast-sglang-starts)\n\n [Cedana: GPU checkpointing and migration with CRIU](https://cedana.com/blog/criu-gpu-checkpointing/) .\n\nWe wanted to see how far we could push GPU checkpointing and restore out of curiousity and to learn a couple things along the way. We began with the `cuda-checkpoint` implementation on a B300 and spend some time understanding how it works and to understand where the bottlenecks lay. Then we’ll try different ideas to make it faster and see what did and didn’t work.\n\n# gVisor\n\n`cuda-checkpoint` is often paired with another program such as `CRIU`[CRIU: GPU checkpointing](https://www.criu.org/GPU_Checkpointing) or `gVisor`[gVisor: GPU checkpoint/restore](https://gvisor.dev/docs/user_guide/checkpoint_restore/#gpu-checkpointrestore)\n\n [Modal: Memory snapshots with gVisor](https://modal.com/blog/mem-snapshots)\n\n [Modal: GPU memory snapshots](https://modal.com/blog/gpu-mem-snapshots) to fully serialize the gpu resource and the application itself to disk. Often, one will hear about `sandboxing` or `snapshotting` with these types of applications.\n\nWe observe a possible 2x speedup by running this version of cuda-checkpoint.\n\n*Takeaways/TLDR*\n\n- Tuning is difficult. There’s many types of hardware and system configurations. \n  - Hardware makes a difference for application performance where sometimes copies can be 3x faster on one platform.\n  - The configuration of the number of threads and buffers difficult as host copy speeds must match up to the PCIE copy speeds.\n- It’s hard to predict performance without tools. Always have some benchmarks and traces to understand what is going on.\n- NUMA affects performance (up to 2x the performance) due to memory being allocated on a different socket\n- The two application specific bottlenecks of `cuda-checkpoint` are memory allocation and the number of buffers and threads used for copying data to and from the GPU.\n- Find opporunities in parallelize operations\n\n# How to use cuda-checkpoint\n\nCheckpointing and restoring a CUDA process each take two steps:\n\nTo checkpoint, we first **lock** the process’s CUDA activity, preventing new work and waiting for existing work to finish. We then **checkpoint** its GPU state into host memory and release its GPU resourcesAlthough releasing the GPU resources is not necessary for checkpointing, it allows other resource-hungry applications to run on the GPU\n\nIn between, another GPU application can utilized the free’d up resources.\n\nTo bring it back, we **restore** the GPU resources and copy the saved data back to GPU memory. Finally, we **unlock** the process’s CUDA activity so it can submit work again.\n\nNVIDIA’s `cuda-checkpoint` tool exposes these four operations through a simple command-line interfaceThis cli communicates using a custom protocol to a cuda process over a pipe. It is a thin wrapper, but impacts latency as shown later:\n\n```\nps aux | grep (your gpu application)\n\ncuda-checkpoint --action lock --pid $PID\ncuda-checkpoint --action checkpoint --pid $PID\ncuda-checkpoint --action restore --pid $PID\ncuda-checkpoint --action unlock --pid $PID\n```\n\nAnd this is how it interacts in action:\n\nThe API is very similar:\n\n```\npid_t pid = getpid();\n\nCUcheckpointLockArgs lock_args{0};\ncuCheckpointProcessLock(pid, &lock_args);\n\nCUcheckpointCheckpointArgs ckpt_args{0};\ncuCheckpointProcessCheckpoint(pid, &ckpt_args);\n\nCUcheckpointRestoreArgs rest_args{0};\ncuCheckpointProcessRestore(pid, &rest_args);\n\nCUcheckpointUnlockArgs unlock_args{0};\ncuCheckpointProcessUnlock(pid, &unlock_args);\n\nCUprocessState st;\ncuCheckpointProcessGetState(pid, &st);\nif (st == CU_PROCESS_STATE_CHECKPOINTED) {\n    // ...\n}\n```\n\nThis tool only stores the checkpoint image to CPU memory.\n\n## Existing work\n\nFergus Finn at Doubleword made a ~9 GB checkpoint-and-restore benchmark on an RTX 4090 nearly 4× faster, cutting the time from 4.5 seconds to 1.2 seconds. The improvements came from avoiding repeated CLI initialization, enabling huge pages, preallocating the host buffer, and deferring its cleanup[Reverse-engineering NVIDIA’s cuda-checkpoint for faster cold starts](https://blog.doubleword.ai/what-happens-when-you-checkpoint-a-cuda-process). We’ll explore these techniques in more detail, tracing the driver’s internals to explain why they help and which bottlenecks they address. We’ll also apply some of them ourselves and see that the optimizations with CLI, huge pages and preallocating host buffers are useful.\n\n## What’s happening in cuda-checkpoint?\n\nWe’ll dig into libcudaNVIDIA’s user-space CUDA driver to see what’s happening underneath the hood during each step. We want to observe where the time goes through tracesEach perfetto trace shows a vLLM instance running on two B300s running GLM5-3 flash (~347GB in total). Each GPU instance holds around half the memory (~172GB) and we are checkpointing both gpus, each referred to as a rank.\n\n## Setup\n\n| Setup | Component | Specification | \n|---|---|---|\n| Host | Number of nodes | 1 | \n|  | GPU | 8 × NVIDIA B300 SXM6 AC | \n|  | PCIe | Gen6 x16; one link per GPU | \n|  | NUMA | GPUs 0–3 on node 0; GPUs 4–7 on node 1 | \n|  | CPU | 2 × Intel Xeon Platinum 8559C; 48 cores per socket | \n|  | Memory | 4 TB; 1 TiB container limit; no swap | \n|  | Driver | 580.95.05 | \n| Workload | Model | GLM-5.3-flash | \n|  | Runtime | vLLM; tensor parallelism 2 | \n|  | GPUs | 0 and 1 for every run | \n| Checkpoint | Per rank | 172.74 GB through the capture ring | \n|  | Driver image per rank | 173.85 GB including about 1.1 GB on the native driver path | \n| Restore | Per rank | 173.8 GB | \n\n**Note** Every optimization arm ran as an untraced / traced, although it didn’t make much of a difference with tracing on.\n\nWe run GLM-5.3-flash with vLLM 0.30.0 on GPUs 0 and 1 of one B300 host. Tensor parallelism is 2, so each GPU has a worker process, called a rank. We use eager execution, an 8,192-token context limit, and an 8 GiB KV cache per GPU.\n\n```\nCUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \\\n  --model /path/to/GLM-5.3-flash \\\n  --tensor-parallel-size 2 \\\n  --enforce-eager \\\n  --max-model-len 8192 \\\n  --kv-cache-memory-bytes 8589934592\n\nGeneration settings go in each request:\n\n{\"temperature\": 0, \"seed\": 1234, \"max_tokens\": 128}\n```\n\nWe run some requests to see if vLLM is running. The benchmark controller waits until vLLM reports no running or queued requests. The controller starts `cuda-checkpoint --action lock --pid <worker-pid>` for both ranks in parallel and waits for both calls to finish. It does the same for `checkpoint`, `restore`, and `unlock`, in that order. After each cycle, it sends the same three requests and checks that their answers and token IDs match those recorded before checkpointing.\n\nOur tracing tool uses Frida to attach function-entry and function-return hooks inside both vLLM workers. It also instruments each `cuda-checkpoint` process, so we can see the CLI’s setup and request alongside the work inside the target worker. The hooks record driver operations and CPU calls such as `mmap` and `memcpy`. We add GPU counters from NVML, sampled at a target interval of 100 ms, and CPU, memory, and page-fault counters from `/proc`.\n\nThese traces and counters get aggregated into a single perfetto trace output.\n\nWe can see how much time spent locking, checkpointing, restoring and unlocking. Clearly, checkpointing and restoring take up most of the time, so let’s zoom into checkpointing.\n\nThe checkpointing profile has three main phases: setup, copying, and cleanup. Copying takes up the most of the time. Even with the theorical PCIE max bandwidth of 64GB/s on the machine, that would take at least ~2.5s to checkpoint 172GB\n\nIt’s interesting tha `mmap` is taking longer than copying. We’ll revisit this later once we piece everything together.\n\nThe copying path allocates a large host buffer with `mmap` and two pinned staging buffersEach staging buffer is 2 MB with small pushbuffers for executing the copies. This large buffer holds the final checkpoint image.\n\nThe staging buffers appears to alternate in a double-buffering scheme: while the GPU copies data into one, the CPU copies data from the other into the final checkpoint buffer:\n\nThe GPU’s DMA engine transfers data from VRAM into one staging buffer while the CPU copies data from the other into the final checkpoint buffer allocated with `mmap`.\n\nSo now that we understand the individual parts, taking a step back, we can stitch a more complete pictureI haven’t explained the restore, but intuitively, it should reverse operations of checkpointing:\n\n## How does the engine know what to copy?\n\nAnd how does the engine know what to copy? By walking memory manager’s blocks and copying its contents from GPU memory to CPU memory.\n\nEach block tracks its checkpoint state. During checkpointing, its physical GPU memory is freed while its virtual address is preserved. On restore, newly allocated GPU memory is mapped back to that addressOne possible reason for keeping the checkpoint in host memory is to leave full-process serialization to CRIU, which can save the driver’s bookkeeping alongside the rest of the process’s CPU-side state.\n\n## Reasoning cuda-checkpoint’s design\n\nLet’s dig into why `cuda-checkpoint` is slow. We have two areas of the checkpoint taking up a good chunk of time.\n\n`mmap` is called with `mmap(address, total, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED | MAP_POPULATE, -1, 0);`.\n\nThe fourth argument includes `MAP_POPULATE`\nPopulate (prefault) page tables for a mapping. For a file mapping, this causes read-ahead on the file. This will help to reduce blocking on page faults later\n\n - \n\n[man pages](https://man7.org/linux/man-pages/man2/mmap.2.html)</span> will go ahead and tell the OS to touch every page to page fault. Allocating 173GB of memory twice takes a while.\n\nOn this B300 node, the theoretical maximum bandwidth of a one way PCIE transfer is 64GB/s. Checkpointing hovers around 6GB/s and restore 3GB/s, an almost 10x and 20x decrease.\n\nTwo questions arise here:\n\n- What is causing the checkpointing to be slow? (Is somehow the DMA transfer slow? Is it the CPU copying? Is it the size of the staging buffers?)\n- What makes restore slower than checkpointing? We should expect the same bandwidth from both.\n\nThe first quesion can be answered by taking a look at the following graph how much time the copy spent calling `memcpy` to drain the staging buffers. If the total time of `memcpy` is small, then the staging buffers are taking too long to fill up. Otherwise, the CPU is taking too long to copy the buffers from the staging buffers into the checkpoint memory. We can see that this is the case belowYou might be wondering why I don’t show the `memcpy` in the cuda track. There are hundred of thousands of `memcpy` s in the trace and I wanted to clearly separate it out:\n\nNow we see that bottleneck is the CPU, being a single thread that copies from the buffer.\n\nOn to the next question: Why is restore then so much slower? If we take a look at the number of page faults in the restore phase, the trace reports around 800k faults per second for each GPU. If each 4 KiB page copied incurs one fault, that corresponds to:\n\n`1.6 million faults/s × 4,096 bytes/page ÷ 2 GPUs ≈ 3.3 GB/s per GPU`\n\nWhich matches the reported PCIE RX GB/s.\n\nWhy is the number page faults higher? We shouldn’t really be page faulting here. We have enough memory ad we’re not allocating any memory because we’re reusing the checkpoint buffer. I suspect it might be dealing with numa because the system is setup with two sockets:\n\nIf we rerun the baseline with some NUMA tracing (reading `/proc/vmstat`), we can see there’s local and total NUMA page faultsNUMA page faults are categorized into local and remote. Local means the page exists on the same CPU node. Remote means that the CPU has to go read the memory through another CPU, incuring cost:\n\nLocal page faults don’t incur much cost, but remote page faults indicate that the application is reading memory through another CPU socket. With half of the reads going to through another socket, we can see why the throughput is halved.\n\nIn this example, both GPUs connect to NUMA node 0. If the buffers are on node 1, GPU transfers must cross the link between sockets. Placing the buffers and workers on node 0 avoids that extra path.\n\nLet’s see a quick fix by setting mpol to localGives a hint to the kernel to prefer to [allocate memory from the local CPU node](https://www.kernel.org/doc/html/latest/filesystems/tmpfs.html).\n\nNo more remote reads and we see now see that the restore is now equal to the checkpoint (~6Gb/s)!\n\n## Why was cuda-checkpoint designed this way?\n\nThe driver cannot assume a particular CPU or pinned-memory budget, keeping compatibility and usability a priority:\n\n- A large worker pool could compete with the application or other checkpointing processes, especially when several GPU processes checkpoint simultaneously. P- inning the entire checkpoint could demand more pinnable memory than the system can support.\n- Two small staging buffers and a single copying thread keep those requirements modestEach 2 MiB staging buffer matches a common huge-page size. I’m not sure if this is a concidence.\n- The newer drivers (as of writing, 2026), allow users to specify what to do with the gpu memory. See [custom storage checkpointing](#new-kid-on-the-block-custom-storage-checkpointing) .\n\n# Optimizations for cuda-checkpoint\n\nEach optimization below will break down what the optimization is and have a perfetto trace associated with itEach trace shows a GPU checkpoint and restore of a vLLM instance running glm5.3 on two B300s. Each checkpoint and restore is ran multiple times (at least 3). A cycle is locking + checkpointing + restoring + unlocking. The run measurement is the median time taken for a cycle after the first cycle.\n\n## Rewriting the CLI\n\nWhy is the CLI slow in the first placeI was curious from the DoubleWord blog, but it didn’t have an explanation? If we take a profile, it becomes immediately obvious:\n\nThe actual work done locking is miniscule compared to time to call the cli to setup and then send the requestThis CLI time is constant. You can observe it for every command. So, every invocation of cuda-checkpoint will have an overhead of at least one second.\n\nEach CLI call initializes CUDA before it sends a request. We can keep a controller running and reuse it for lock, checkpoint, restore, and unlock. This avoids repeating initialization for each action.\n\nThis startup cost matters more as the rest of the cycle gets shorter because this is a decrease in ~4 seconds, one second per action commnad. When we reduce the checkpoint and restore to a couple of seconds, the cli optimization matters more.\n\n## Huge pages\n\nLet’s try out huge pages. It should speed up the `mmap` call to allocate the checkpoint buffer as well as speed up copying a bit.\n\nThe checkpoint buffer holds the same amount of data. Larger pages mean fewer pages for the kernel to manage. This can reduce the time needed to prepare the buffer before copying starts.\n\nThis optimization reduces the time from 118s with the cli optimization to 98s (~1.4x speedup compared to baseline).\n\n## Copying differently\n\nHow can we speed up the copying?\n\n### Minimizing page faults\n\n`mmap` consumes most the time for copying. Each time a page is allocated, the system page faults to bring in the physical page. We see that for every checkpoint, this consumes ~40s.\n\nInstead, we can just preallocate the page ahead of time and remove that bottleneck.\n\nWe allocate and populate the host buffer once, then keep it after restore. The next checkpoint reuses this memory, avoiding repeated allocation and cleanup. We still copy the GPU memory into the buffer each time.\n\nThis optimization reduces the time per checkpoint + restore cycle from 98 s to 69 s (about ~1.4× faster, and ~1.9x compared to the baseline)\n\nAt this point, we’ve reached about the same result as what’s descrbed in the Doubleword blog. We get a ~2x speedup compared to the baseline, which is great! Although the Doubleword blog describes a 4x speedup, we will try to match that and even more so on this node.\n\n### Parallelizing the copy\n\nNow that preallocation solves the time spent in the allocation, we can tackle the ~30s spent in copying.\n\n`cuda-checkpoint` uses a single CPU thread to copy data from the staging buffers into the final checkpoint buffer. Can that thread keep up with data arriving over PCIe? We’ll first compare single-threaded and multithreaded CPU-copy bandwidth against GPU-to-host PCIe bandwidth on the same machine.\n\nWe can start by keeping the staging buffers and trying to drain them faster. Multiple CPU workers can each copy part of a completed staging buffer into the final checkpoint buffer, while the GPU fills the other staging buffer.\n\nEach worker copies a separate part of the completed staging buffer into the checkpoint image. The GPU can fill another staging buffer while the workers finish this copy.\n\nWith sixteen staging slots, eight workers take 11.3 s and sixteen workers take 17.0 s. More workers do not always help. This comparison uses warm cycle 2 for eight workers and warm cycle 1 for sixteen workers.\n\nWe can also vary the number of staging buffers.\n\nMore slots let GPU and CPU copies overlap. With eight workers, increasing the slot count from two to sixteen reduces the full cycle from 35.9 s to 11.3 s (3.2× faster). Increasing it to sixty-four slots raises the cycle time to 12.4 s.\n\n## Worker and staging-slot traces\n\n**Vary staging slots — 8 workers**\n\n**Vary workers — compare with the 8-worker, 16-slot run above**\n\n**Ring2-spin — tuned ring settings**\n\n### Removing the staging buffers\n\nBut can we avoid the CPU copy altogether? Instead of copying through staging buffers, we could copy GPU memory directly into the final checkpoint buffer.\n\nThe catch is that the final buffer is pageable host memory. To use it directly for DMA, we need to pin its pagesPinning keeps the pages resident at stable physical addresses while the GPU accesses them.\n\nThis removes the extra CPU copy, but requires pinning the entire checkpoint buffer rather than two small staging buffersFor a large checkpoint, allocating or registering that much pinned memory may fail.\n\nDirect copying removes the extra CPU copy, but it is slower than using staging buffers. An increase from 8.7s to 11.5s.\n\nWhy does should this happen? Wouldn’t copying the GPU memory to the final checkpoint buffer be faster than having an intermediate buffer?\n\nThe answer lies in how the page tables are setup on the GPU. One calls map the pinned CPU memory by calling `cuMemHostRegister`. Each checkpoint call must register this pinned memory and deregister the pinned memory with `cuMemHostUnregister` during restorewe must enforce the policy that we release all GPU resources during a checkpoint which includes unmapping the pinned cpu memory.\n\nHowever, this page table setup is slow. The GPU creates 4KB pages for its PTEs to be mapped to the CPU pages and does this serially through the kernel. It’s not that slow, but for 174 GB, this takes up quite a bit of time.\n\n`174 GiB ÷ 4 KiB / 512 entries per page = ~90k calls to kernel to allocate PTEs`\n\nYou might ask: why not use 2MB pages on the GPU? That isn’t supported through `cuMemHostRegister` yet.\n\n## Becoming NUMA-aware\n\nAs we’ve seen from analyzing the NUMA bottleneck in the baseline, it can decrease bandwidth by quite a bit if not configured correctly.\n\nTo reiterate, GPUs may be connected to one host chip. This chip may be copying memory from this CPU node to another CPU node:\n\nI want to show the results with no options (letting the kernel figure it out for us), forcing all allocations to be on a remote node (the worst case), and forcing the allocations to be only local. Here is that graph with the results:\n\nHere I’ve included two different runs to demonstrate the difference between using the memory allocations on a local node versus a remote node.\n\nHoweever, this didn’t show an improvemnt for our system, probably due to MPOL NUMA policy working properly.\n\n## Becoming PCIe link aware\n\nGPUs may share PCIE bandwidth by sharing a root complex[Tander poder’s blog](https://tanelpoder.com/posts/11m-iops-with-10-ssds-on-amd-threadripper-pro-workstation/#pcie-root-complex-bottleneck) is first how I learned about this type of bottleneck.\n\nThis occurs for the B200 for example, unlike the B300.\n\nWhen two GPUs share one uplink, their transfers compete for its bandwidth. Choosing GPUs with separate uplinks removes that shared-link limit. Both transfers still use the host’s memory bandwidth.\n\nHere is a microbenchmark testing the bandwidth with `cudaMemcpyDtoH` and `cudaMemcpyHtoD` with 1GB copies:\n\nWe see that in the first two bars, we’re running with the workload on a single GPU and that hits the PCIE limit. However, in the next two bars, we have two copies happening on two GPUs sharing a root GPU. This results in halving the bandwidth for both GPUs. In the 5th bar, the bandwidth is again at max because those two gpus don’t share the same PCIE bandwith.\n\nSeparate PCIe links avoid a shared bandwidth limit. Copy bandwidth rises from about 30 GB/s to 55 GB/s per GPU (about 1.8× the bandwidth). These are transfer measurements.\n\nHere’s a small bandwidth benchmark measuring the copy throughput at different sizes with inference (either from another process on another GPU that sharing the PCIE bandwidth and another process on the same GPU):\n\n## B200 PCIe microbenchmarks\n\n## B200 workload traces\n\nBut I had an idea: what if we can also utilize the other link if it’s not being used for extra bandwidth on a B300 (placing half the weights on GPU 0 and the other half on GPU 4, which are connected to separate NUMAs):\n\nThis increases the performance slightly: from 9.8s to 9.5s.\n\n## Streaming Stores\n\nSmall idea with copies: If we know that the memory being copied to is not going to be reused (cough cough, the staging buffers), then we can optimize the CPU copies even more. By using CPU specific intrinstics, `_mm512_stream_si512`[Intel Intrinsics Guide: _mm512_stream_si512](https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#text=_mm512_stream_si512) for AVX512 (the CPU we test supports AVX512) or `_mm256_stream_si256`[Intel Intrinsics Guide: _mm256_stream_si256](https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#text=_mm256_stream_si256) for AVX2, we can speed up the copy. This gives a hint to the CPU to write directly to memory rather than populating the CPU caches.\n\nThis optimzation reduces the last run from 9.5s to 8.7s, almost a 1s decrease!\n\n## Not-so-generic optimizations\n\n### Blackwell decompression\n\nNVIDIA introduced a dedicated Decompression Engine with Blackwell, announced in March 2024. It is integrated with the copy engine and is supported on B200, B300, GB200 and GB300[Blackwell announcement, March 18, 2024](https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing/)\n\n [NVIDIA: Decompression Engine and supported GPUs](https://docs.nvidia.com/cuda/nvcomp/decompression_engine_faq.html)\n\n Software support arrived in nvCOMP 4.2 in February 2025: [release notes](https://docs.nvidia.com/cuda/nvcomp/release_notes.html#nvcomp-4-2-0), [release archive](https://developer.nvidia.com/nvcomp-archive)..\n\nA microbenchmark compares the decompression engine bandwidth against nvCOMP kernels running on SMs[nvCOMP: NVIDIA’s GPU compression and decompression library](https://developer.nvidia.com/nvcomp).\n\nThe only compression that had gains was `Deflate` (1.28 compression) as well as a huge throughput gain. GFlate and Zstd aren’t supported on the B300 hardware (yet).\n\nIf we now limit it across the PCIE, we see that the decompression engine has gains beyond the raw transfer bandwidth because the hardware can handle both the copying and decompression.\n\nHowever, running with nvCOMP kernels to compress the data and using the decompression engine to decompress the data on restore doesn’t result in an improvement.\n\nThe time per cycle goes from about 9.4 seconds to 12.5 seconds.\n\n### Skipping zero-filled chunks\n\nThis is an experiment where zero pages are marked in as a byte in an array instead of sending the entire chunkThe chunk here is the size of the staging buffer, which I just chose as 64KB. This could potentially speed up the checkpointing and restore as less bytes are sent. However, it seems that the image did not have many chunks containing many zeroes. I was motivated by some older experiments where pytorch or vLLM would manage its own memory allocator by allocating all of its memory up front.\n\n### Incremental checkpointing\n\nIncremental checkpointing is an optimization that takes a snapshot that records only the pages that have been modified. We can do this by running a kernel before hand and then guiding the copy to only copy pages that have been modified.\n\nThe red pages have changed since the saved checkpoint. We copy these pages and keep the earlier checkpoint for the unchanged pages. Restore needs both. Finding which pages changed also takes time.\n\nWe copy only changed data during checkpointing. This experiment was ran with an AMD CPU.\n\n## Another platform to run on!\n\nI’ve ran on AMD to see how it behaves and results in this graph:\n\nWe can see that out of the box, the AMD machine behaves with much higher performance. Here is a breakdown of the checkpoint/restore times\n\n## Expand to see the EPYC traces for each step\n\nAlthough we get 4x, we can see why we get ~3x because copies are faster on these chips:\n\nWhen we run these, it makes it so that you have to configure the number of threads on the intel to be higher to achieve the same bandwidth. So the takeaway is that because hosts hardware perform differently, you have to tune the copy to be the most performant.\n\n## To conclude\n\nAt this point, we’ve reached at least a 15x speedup, which is great. Each technique mostly builds on top of each other:\n\nThe cli optimization + transparent huge pages + preallocation + NUMA awareness are pretty generic optimizations that any application can use, resulting in a 2x speedup.\n\nBy digging into the cuda stack and changing the number of workers and staging buffers, we can achieve a greater speedup.\n\n# Optimizations for sandboxes\n\nBut checkpointing cuda applications from GPU memory is one slice of the bigger pie. The final goal is to serialize the entire application state to some medium. Typically a disk is preferred because it can store a whole lot more than CPU memory at significantly less cost.\n\nSo let’s see how well the `cuda-checkpoint` optimizations play in these applications. We can use [gVisor](https://gvisor.dev/), a popular sandboxing tool built and maintained by google. Modal uses gVisor’s `runsc` runtime for container isolation and memory snapshots[Modal: Memory snapshots with gVisor](https://modal.com/blog/mem-snapshots)\n\n [Modal: GPU memory snapshots](https://modal.com/blog/gpu-mem-snapshots).\n\nIn the following sections, I’ll compare runnning gVisor running `cuda-checkpoint` with different flags used by gVisor with the goal in mind to minimize the CPU bottlenecks as much possible so we can see the `cuda-checkpoint` optimizations shine. We ran with commit `309599c1ec020ad52c82b7b6657acba73888e367`.\n\n### Where gVisor spends the time\n\nLet’s start with stock gVisor running the same GLM workload. I’ll ask the same question again: where does the time go?\n\n`cuda-checkpoint` leaves the GPU checkpoint in host memory. gVisor then pauses the sandbox and saves its memory and kernel state. Then it serializes checkpoints as images on disk.\n\nHere’s a breakdown of that timeline at a high level:\n\nSerialiazing the GPU memory takes ~64s, but then serializing all of the CPU memory takes 228s (almost 3x more)? Almost all of that second part is spent saving about ~333GB of memory pages.\n\nUnexpectedly restore does the reverse. Load the pages from the file, restore the GPU state, then let the application run. Loading pages takes less time at 42 seconds and GPU restore takes 27 secondsYou might wonder why restore is quite fast? Looks like the NUMA memory placement was correct for restore.\n\nSo making CUDA faster can help, but it won’t remove those 228 seconds. We need to look at how gVisor saves memory too.\n\n## Seeing cuda-checkpoint optimizations in action\n\nFirst, let’s use our optimized CUDA path without changing gVisor’s page writer.\n\nThe gpu checkpointing/restore swap in reduces the end to end time by almost 100s, which is pretty neat.\n\n### Getting the checkpoint onto storage faster\n\nMost of the time is spent in gVisor trying to serialize the pages to disk. Why is this so slow? It’s because the gVisor sends 256 KiB writes through one serial worker. We can skip this with `--direct` flag turned on.\n\nBy opening the pages file with `O_DIRECT` to bypass the page cache. gVisor then uses Linux asynchronous I/O, allowing several requests to be in flight instead of finishing one buffered write at a time, which it configures to have a depth of 128 instead of 1. We can also make the buffers bigger.\n\nFirst, let’s measure file I/O by itself. This fio test uses one job and moves 16 GiB per pass. Writes use 256 KiB requests; reads use 4 MiB requestsThe requested I/O depth is 128 for writes and 12 for reads. These match stock gVisor’s request sizes, but fio’s queueing differs from gVisor’s workers. The runs are short and may use backend caches, so they do not establish sustained storage limits. [Measurements and achieved I/O depth](https://maknee.github.io/assets/images/posts/2026-09-21/traces/blog/gvisor/microbenchmarks/fio-nvme-fuse-gvisor-default-256kib-writes-4mib-reads.json)..\n\nThis change with larger requests decreases the time taken by almost 90%, cutting down the time spent copying CPU memory from about 228 seconds to 23 secondsYou might wonder why Stock + direct I/O reduces CUDA checkpoint. Again NUMA allocated memory correctly this time on the right node. Running with optimized `cuda-checkpoint` reduces the time by 50%.\n\nHere’s the perfetto traces for this\n\n### Parallel CPU/GPU streams\n\nWe are still waiting for CUDA to finish before starting most of the page writes. Can we use that time to write pages we already have?\n\nAn optimization that be done is to write the gvisor memory and cuda-checkpoint both to disk as images at the same timegVisor has different on disk images for the pages and the gpu checkpoint, which makes this even more promising. However one problem is that the cuda-checkpoint pages and cuda state won’t be synchronized with the rest of the process. How can we fix this?\n\nWe record a hash when we copy the bytes, then hash the same memory again after CUDA finishes and the sandbox is paused. If the hashes match, we keep the data we’ve already written. If they differ, we know that the contents has changed and can save the updated contents. We still have to read the every piece of memory again, but we don’t have to write every page again to disk. This optimization is not a flag in gVisor.\n\nHere is what that overlap looks like in the trace:\n\nWhile the gpu checkpointing is running, gVisor starts checkpointing the process at the same time. The GPU checkpoint finishes at 14.4s, so it rechecks the hashes and then serailizes any difference to disk, which would be in this case, the cuda-checkpoint image and and cuda context state.\n\n### Starting GPU restore before all pages arrive\n\nRestore has a similar wait. The stock path loads the memory pages, then starts CUDA restore. It can instead load pages in the background and start CUDA restore earlier, waiting when it needs a page that hasn’t arrived yet. This is enabled by setting `--background`\n\nAfter gVisor restores the process state and memory mappings, it can start CUDA restore while host pages continue loading. If restore needs memory that has not arrived, gVisor blocks that access until the saved data is availableSee [`HandleUserFault` in `syscalls.go`](https://github.com/google/gvisor/blob/309599c1ec020ad52c82b7b6657acba73888e367/pkg/sentry/mm/syscalls.go#L38-L77).. CUDA restore then rebuilds the GPU resources from those bytes.\n\nThis makes no difference in speeding up the host pages, but allows `cuda-checkpoint` to work while the host pages are being loaded.\n\nCUDA restore itself seems to get slower: about 21 seconds instead of 11 seconds. But that’s because it’s waiting for the pages to load in from gVisor. Time reduces from 28.5s to 21.5, a nice 7 seconds in reduction.\n\n### Putting the changes together\n\nFor GLM on NVMe, we brought checkpoint time from 296 seconds to about 20 seconds, roughly 15× faster.\n\n## What changes on an object store vs running on NVMe?\n\nThese results use local NVMe. What if the checkpoint file is on something like FUSE on S3?\n\nRepeating the same fio test on FUSE gives a different result: direct I/O lowers both write and read bandwidth in these runs.\n\nThe same choices don’t work as well here.\n\nThe reason is that there is contention in the fuse layer. The seq write speed caps out at 1.5 GB/s already.\n\n## FUSE traces and storage microbenchmarks\n\n## Does this help a smaller model?\n\nGLM5.3 is a fairly large model at doesn’t fit on a single GPU. Let’s try Qwen3-4B-FP8 on one GPU (~5GB vs ~344GB).\n\nStock takes 17.3 seconds for checkpoint and restore. Direct streaming with optimized CUDA takes 8.0 seconds, about 2.2× faster. Even with such a small model, we still can see a 2x improvement.\n\n## Qwen traces\n\n## Other tidbits\n\nI see that gVisor also calls `cuda-checkpoint` on contexts with very small memory (like a vLLM monitoring processed that forked from the original). There could be ways to optimize for this if cuda isn’t being used in the process.\n\nI think gVisor could greatly benefit from hardware decompression on blackwell. Even for weights, a ~1.2x speedup while compressing 1.3x for model weights is pretty good. When there’s a lull in activity, a couple of cores can be used for compressing the gpu checkpoint image.\n\n# Future thoughts\n\n## New kid on the block: custom storage checkpointing\n\nCUDA 13.4 added a custom storage api to the checkpoint API that allows skipping the host buffer entirely [CUDA 13.4 release notes](https://docs.nvidia.com/cuda/archive/13.4.1/pdf/CUDA_Toolkit_Release_Notes.pdf): cuda-checkpoint can now write checkpoint state to persistent storage instead of retaining it only in process memory\n\n [CUDA Checkpointing API reference](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__CHECKPOINT.html) . Instead of the driver copying VRAM into its own staging buffers, it hands us a device pointer per GPU and lets us decide where the bytes go.\n\n```\nCUresult do_checkpoint(int pid, void **dst /* one host buffer per GPU */)\n{\n    cuInit(0);\n\n    CUcheckpointLockArgs lock = {0};\n    lock.timeoutMs = 30000;\n    CUresult rc = cuCheckpointProcessLock(pid, &lock);\n    if (rc != CUDA_SUCCESS) return rc;\n\n    // Ask the driver to map the target's VRAM instead of copying it to host\n    CUcheckpointCustomStorageInfo *info = NULL;\n    CUcheckpointCheckpointArgs ckpt = {0};\n    ckpt.customStorageInfo_out = &info;\n    rc = cuCheckpointProcessCheckpoint(pid, &ckpt);\n\n    if (rc == CUDA_SUCCESS) {\n        // One contiguous mapping per GPU. Copy it wherever we want.\n        for (unsigned i = 0; i < info->deviceCount; i++) {\n            CUcheckpointCustomStoragePerDeviceData *d = &info->perDeviceData[i];\n            cuMemcpyDtoHAsync(dst[i], d->devPtr, d->size, d->stream);\n        }\n        // Driver syncs every stream above, then unmaps and releases VRAM\n        rc = cuCheckpointOperationComplete(info->handle);\n    }\n\n    CUcheckpointUnlockArgs unlock = {0};\n    cuCheckpointProcessUnlock(pid, &unlock);\n    return rc;\n}\n```\n\nThe flow is the same lock, checkpoint, unlock as before, with two differences. Checkpointing takes a `customStorageInfo_out` pointer, and we have to call `cuCheckpointOperationComplete` once our copies are enqueued[cuCheckpointOperationComplete](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__CHECKPOINT.html): synchronizes the returned streams, then unmaps the GPU memory and moves the target to `CU_PROCESS_STATE_CHECKPOINTED`. Restore is symmetric: `cuCheckpointProcessRestore` takes the same output pointer, we copy our bytes back into the mapped device pointers, and complete the operation again.\n\nYou might wonder how a plain `cuMemcpyDtoHAsync` from our process work on another process’s memory? Because the driver maps it into our address space for us as described from the docs[cuCheckpointProcessCheckpoint](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__CHECKPOINT.html), [CUcheckpointCustomStoragePerDeviceData](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/structCUcheckpointCustomStoragePerDeviceData.html)> all the GPU memory allocated by the process with `pid` will be mapped onto the calling process’s GPUs so the application can copy the memory to custom storage:\n\nThis gives us the previous section should capture similar bandwidthProbably won’t work out of box, but at least when I tried on a spare RTX3090, it had reached PCIE bandwidth calling `cuMemcpyDtoHAsync`. Also, why won’t it work with older cuda versions? The kernel driver doesn’t expose a clear way to map the same gpu virtual memory between two processes to the same cuda context.\n\n## Grace Hopper\n\nGrace Hopper and Grace Blackwell use NVLink-C2C instead of PCIe for transfers between the GPU and its paired CPU’s memory. GH200 provides up to 450 GB/s per direction for one GPU. GB200 splits the same total bandwidth between two GPUs, giving each up to 225 GB/s per direction [NVIDIA: Grace Hopper architecture](https://developer.nvidia.com/blog/nvidia-grace-hopper-superchip-architecture-in-depth)\n\n [NVIDIA: GB200 bandwidth clarification](https://forums.developer.nvidia.com/t/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/286460/7)\n\n The advertised 900 GB/s includes both directions: 900 ÷ 2 = 450 GB/s for GH200; 900 ÷ 2 ÷ 2 GPUs = 225 GB/s per GPU for GB200. .\n\nWe should expect much higher bandwidth bottlenecks from staging buffers and should much prefer direct copy instead. Many more cores are needed to reach the bandwidth.\n\n## Batching setup and teartown commands\n\nSince we know we’re going to setup the state and all of the state for the GPU down, we can batch the commands into one syscall call to the kernel module instead of spreading it out.\n\n## Serializing to other mediums\n\nOther mediums will mostly be the bottleneck compared to PCIE (lower cost!). Could be HDDs, SSDs, Object Stores, memory pools, distributed filesystems. We should try to aim to parallelize the work of these devices to decrease latency\n\n## Thinking about running these optimizations in production\n\nSince we are focused on optimizing a single microbenchmark, some questions are left for systems that have multiple concurrent checkpointing and restores that could be happening at once. There could be a burst of requests and then a lull, for example.\n\n### Preallocation optimization\n\nI believe that it’s pretty hard to predict the future (and especially if that future is very soon). If we preallocate memory for an image that in the future be stored or restored, how big should that allocation be?\n\nThe simplest way is to statically allocate a big pool of memory and use it for checkpointing.\n\nAnother hacky way is to throw all the metrics into a machine learning model and tell it to optimize for say checkpointing/restore hits with the mimimal amount of memory by observing a history of requestsUnderstability is a bit harder to get here.\n\nOr maybe just not preallocate at all. Or maybe preallocate only if memory limits never get hit with other workloads. Or maybe just isolate the machine to only allocate up to a certain amount using cgroups and you know that the rest of memory has to be used for checkpointing.\n\n### NCCL and other state\n\nNCCL and other custom protocols require serializaton as well, which `cuda-checkpoint` is unaware of. `cuda-checkpoint` only checkpoints at the awareness of a single GPU. Most systems running in data centers utilize more than a single GPU.\n\n### Priority\n\nTo achieve the lowest latency, the gpu sandbox should be checkpointed or restored immediately. But one could imagine a huge queue of sandboxes ready to be checkpointed or restored at any given moment. The GPU process might hold a huge amount of memory or very little. What does one decide to checkpoint? We know that the speed is bounded by PCIE, so that’s the limit. But we could run the workload on any GPU. Perhaps there could be a scheduler that tries to maximize the PCIE bandwidth used while minimizing the contention between applications trying to checkpoint or restore.\n\n# Acknowledgments\n\nThanks to [Chirag Shetty](https://www.chiragshetty.com/) and [Ayush Ranjan](https://ayushr.com/) for giving valuable feedback.\n\n# Citation\n\nTo cite this article:\n\n```\n@article{zhu2026cudacheckpointing,\n  title = {Making CUDA Checkpointing 15x and gVisor 2x Faster on a B300: A Worklog},\n  author = {Zhu, Henry and Sathia, Vimarsh},\n  journal = {maknee.github.io},\n  year = {2026},\n  month = {October},\n  url = \"https://maknee.github.io/blog/2026/Cuda-Checkpointing/\"\n}\n```\n\n## Enjoy Reading This Article?\n\nHere are some more articles you might like to read next:\n\n[NVIDIA TileIR Internals: from CuTile to MLIR/LLVM to SASS](https://maknee.github.io/blog/2026/NVIDIA-TileIR-Internals-from-CuTile-to-MLIR-LLVM-to-SASS/)\n\n[Performance Hints](https://maknee.github.io/blog/2026/Performance-Hints/)\n\n[Maybe consider putting \"cutlass\" in your CUDA/Triton kernels](https://maknee.github.io/blog/2025/Maybe-Consider-Putting-Cutlass-In-Your-CUDA-Kernels/)\n\n[Network Storage and Scaling Characteristics of a Distributed Filesystem](https://maknee.github.io/blog/2025/3FS-Performance-Journal-3/)\n\n[Network and Storage Benchmarks for LLM Training on the Cloud](https://maknee.github.io/blog/2025/Network-And-Storage-Training-Skypilot/)", "url": "https://wpnews.pro/news/making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog", "canonical_source": "https://maknee.github.io/blog/2026/Cuda-Checkpointing/", "published_at": "2026-11-02 22:19:16+00:00", "updated_at": "2026-11-02 22:51:29.515315+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "mlops"], "entities": ["NVIDIA", "cuda-checkpoint", "B300", "gVisor", "CRIU", "Modal", "Doubleword", "Cedana"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog", "markdown": "https://wpnews.pro/news/making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog.md", "text": "https://wpnews.pro/news/making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog.txt", "jsonld": "https://wpnews.pro/news/making-cuda-checkpointing-15x-faster-on-a-b300-a-worklog.jsonld"}}