{"slug": "reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts", "title": "Reverse-engineering NVIDIA's cuda-checkpoint for faster cold starts", "summary": "NVIDIA's closed-source cuda-checkpoint tool, which freezes and restores CUDA processes, suffers from slow PCIe transfers. Engineers reverse-engineered the tool to understand why checkpoint transfers fail to saturate PCIe bandwidth and developed a method to speed them up without modifying the application or driver.", "body_md": "# Reverse-engineering NVIDIA's cuda-checkpoint for faster cold starts\n\nThere’s a little known feature in the closed-source NVIDIA driver that lets you\nfreeze a running CUDA process, serialize its GPU state into host memory, and\nlater restore it to the GPU exactly as it was. We used it in an earlier post to\nspeed up [SGLang server\nstartup](/fast-sglang-starts) by up to 70x.\n\nThe utility is called `cuda-checkpoint`\n\n. The feature is documented, but how it\nworks isn’t. One very frustrating aspect, that dogs anyone trying to use it to\ncheckpoint complex GPU processes, is that the checkpoint transfers come nowhere\nclose to saturating PCIe bandwidth. We left off our investigation in the\nearlier post without a good answer for why that was the caseIn the end, we just used cooperation from the application side to work\naround it..\n\nNow we do: why it costs so much, and how to make it faster, without modifying the application, or the driver.\n\n[How to checkpoint a CUDA process](#how-to-checkpoint-a-cuda-process)\n\nHere’s a small CUDA program:\n\n```\n__device__ int counter = 100;\n__global__ void increment() { counter++; }\n\nint main(void) {\n    cudaFree(0);                                   // force context creation\n    int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);\n    sockaddr_in addr = {AF_INET, htons(10000), inet_addr(\"127.0.0.1\")};\n    bind(sock, (sockaddr *)&addr, sizeof addr);\n\n    while (true) {\n        char buffer[16] = {0};\n        sockaddr_in peer = {0}; socklen_t n = sizeof peer;\n        recvfrom(sock, buffer, sizeof buffer, 0, (sockaddr *)&peer, &n);\n\n        increment<<<1,1>>>();                       // one thread, counter++\n        int h = 0;\n        cudaMemcpyFromSymbol(&h, counter, sizeof counter);\n\n        size_t bytes = sprintf(buffer, \"%d\\n\", h);\n        sendto(sock, buffer, bytes, 0, (sockaddr *)&peer, n);\n    }\n}\n```\n\nIt binds a UDP socket, and every time a packet arrives it launches a\none-thread kernel that increments a `__device__ int`\n\n, reads it back, and\nreplies with the valueThis is NVIDIA's demo, shipped alongside the\n[ cuda-checkpoint](https://github.com/NVIDIA/cuda-checkpoint) tool. I've\ntrimmed the error handling. Same 4090 and driver 590.48.01 as the\n\n[kernel-launch post](/what-happens-when-you-run-a-cuda-kernel).. The counter lives in GPU memory and starts at 100. Ping it, and it says 101.\n\n``` bash\n$ ./counter &\n$ echo -n ping | nc -u -w1 127.0.0.1 10000 # send the packet, and view the response\n101\n```\n\nWe can freeze this process — copy its GPU state out, tear its CUDA context down to nothing, remove it from the GPU entirely — and then, some time later, bring it back exactly where it was:\n\n``` bash\n$ P=$(pgrep -xn counter)\n$ cuda-checkpoint --action checkpoint --pid $P     # (lock first; see below)\n$ cuda-checkpoint --action restore   --pid $P\n$ echo -n ping | nc -u -w1 127.0.0.1 10000\n102\n```\n\nIn between those two commands the process holds no GPU memory, has no CUDA\ncontext, and does not appear in `nvidia-smi`\n\n. The counter, which lived only on\nthe device, survives anyway. This is the mechanism that a [previous\npost](/fast-sglang-starts) leaned on to restore a 122B-parameter server in\na few seconds — there, `cuda-checkpoint`\n\nwas a black box called by CRIU. This\npost is about how we can find out what’s inside the box.\n\n[Watching the process disappear](#watching-the-process-disappear)\n\nLet’s watch the process disappear from the device. `cuda-checkpoint`\n\ndrives a\nsmall state machine over the target process; `--action lock`\n\nmoves it from\n`running`\n\nto `locked`\n\n, and then `--action checkpoint`\n\nmoves it from `locked`\n\nto\n`checkpointed`\n\n.\n\n```\ncuda-checkpoint --action lock       --pid $P\ncuda-checkpoint --action checkpoint --pid $P\n```\n\nWatching the process across the two calls, with `nvidia-smi`\n\n, its\n`/proc/$P/maps`\n\n, its open file descriptors, and the `RssAnon`\n\nline of\n`/proc/$P/status`\n\n:\n\n| running | locked | checkpointed | |\n|---|---|---|---|\n`RssAnon` | 12,860 kB | 12,860 kB | 420,812 kB |\nNVIDIA VMAs in `/proc/$P/maps` | 26 | 26 | 0 |\nNVIDIA fds in `/proc/$P/fd/` | 24 | 24 | 0 |\nvisible in `nvidia-smi` | yes | yes | no |\n\n`lock`\n\ndoesn’t change anything observable. But checkpoint does: every mapping\nof a `/dev/nvidia*`\n\nfile is gone, every file descriptor pointing at the driver\nis closed, and the process vanishes from `nvidia-smi`\n\n. As far as the kernel\ndriver is concerned, this process is no longer using a GPU.\n\nThe resident anonymous memory jumped by 407,952 kB at the same moment, as the GPU state moves into ordinary host memory, into the process’s own address space. Can we find it?\n\n[Poking the checkpoint](#poking-the-checkpoint)\n\nThe jump in `RssAnon`\n\nis almost exactly the size of one new anonymous mapping\nthat appears in `/proc/$P/maps`\n\nat the checkpoint. If we attach `strace`\n\nto the\ntarget across the checkpoint we can\ncatch it being allocated:\n\n``` bash\n$ strace -f -p $P -e trace=mmap\n...\nmmap(NULL, 417739792, PROT_READ|PROT_WRITE,\n     MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0) = 0x...\n```\n\n417,739,792 bytes is about 398 MiB, against the 388 MiB of device memory\n`nvidia-smi`\n\nhad attributed to the process. So the inference is the counter’s\ndevice footprint has been serialized into this buffer, plus about\nten megabytes of something else. `MAP_POPULATE`\n\nasks the kernel to fault\nthe whole thing in immediately rather than lazily, which matters later.\n\nWhat’s in there? We know what the `increment`\n\nkernel compiles to — `cuobjdump -sass counter`\n\ngives us its SASS — so we can prove that it’s in the mapping by\ntaking the first few instruction words as a needle and searching the anonymous\nmapping for them. We can also find the counter. Nearby, in a page that is\notherwise entirely zeros, is a single non-zero integer holding `0x67`\n\n— 103,\nbecause I’d pinged it a couple of times before checkpointing.\n\nWith a few more of these kinds of tricks, we can see that the checkpoint buffer structure is pretty simple:\n\nThe counter demo's checkpoint image, mapped by classifying every 4 KiB\npage and locating the driver's GPU-mapped surfaces inside it. The driver state\nis the constant ~10 MiB diff between the image and the `nvidia-smi`\n\nfootprint.\nIt seems to be GPU-mapped *host* memory (the `increment()`\n\nSASS is in there, as\nare kernel-launch parameter banks, and the channels' notifier pages).\n\nWhat’s more, the checkpoint image is plain anonymous memory in a process we\nown. Let’s mess with it. Rather than increment the counter through the GPU, we\ncan reach into the frozen image and tweak it there. `/proc/$P/mem`\n\nlets us\nwrite the process’s memory directlyNeeds `CAP_SYS_PTRACE`\n\nor `ptrace_scope=0`\n\n., so we seek to the offset where we\nfound the counter and write `424242`\n\n. Then:\n\n``` bash\n$ cuda-checkpoint --action restore --pid $P\n$ cuda-checkpoint --action unlock  --pid $P\n$ echo -n ping | nc -u -w1 127.0.0.1 10000\n424243\n```\n\nThe restore uploads the number back onto the GPU, the next packet runs\n`counter++`\n\non it, and the process replies 424243, incrementing our injected\nvalue.\n\nSo we know that the host-side anonymous buffer *is* the device memory across a\ncheckpoint. But how did that memory get there?\n\n[Who does the work](#who-does-the-work)\n\n`cuda-checkpoint`\n\n, the process we invoked, cannot read the target’s device\nmemory. The tooling that does lives inside the target process and worse, inside\nthe closed-source userspace driver.\n\nIf you `strace`\n\nthe utility, essentially everything it does to the target process is\nthis:\n\n``` bash\n$ strace -f -e trace=openat,read,write cuda-checkpoint --action checkpoint --pid $P\n...\nopenat(AT_FDCWD, \"/proc/$P/task\", O_RDONLY|O_DIRECTORY) = 35\nopenat(AT_FDCWD, \"/proc/$P/task/2863110/comm\", O_RDONLY) = 36\nread(36, \"cuda00001400006\\n\", 1024)          = 16    # the CUDA service thread\nopenat(AT_FDCWD, \"/proc/$P/fd/6\",  O_RDONLY) = 35    # its reply pipe\nopenat(AT_FDCWD, \"/proc/$P/fd/5\",  O_WRONLY) = 36    # its command pipe\nwrite(36, \"\\5\\0\\0\\0\", 4)                     = 4     # \"where do I talk to you?\"\nread(35, \"-\\0\\0\\0\\20\\0\\0\\0\", 8)              = 8     # -> use fds 45 and 16\nopenat(AT_FDCWD, \"/proc/$P/fd/45\", O_RDONLY) = 37\nopenat(AT_FDCWD, \"/proc/$P/fd/16\", O_WRONLY) = 38\nwrite(38, \"\\6\\0\\0\\0\\0\\0\\0\\0\"..., 2064)       = 2064  # handshake\nread(37, \"\\0\\0\\0\\0\\1\\0\\0\\0\", 8)              = 8\nwrite(38, \"\\2\\0\\0\\0\\1\\0\\0\\0\"..., 2064)       = 2064  # opcode 2, action 1: checkpoint\nread(37, \"\\0\\0\\0\\0\\2\\0\\0\\0\", 8)              = 8     # status\n...\n```\n\nIn words, it walks `/proc/$P/fd/`\n\n, finds a pipe that `libcuda`\n\nopened inside\nthe target process when the CUDA context was first created, and writes a\ncommand word into it. The action lives in a single word, in exactly the order\nthe CLI lists them:\n\n`word1` | action |\n|---|---|\n| 0 | `lock` |\n| 1 | `checkpoint` |\n| 2 | `restore` |\n| 3 | `unlock` |\n\nOn the other end of that pipe, inside the target, a thread named\n`cuda00001400006`\n\n`cuda-checkpoint --get-restore-tid`\n\nreturns a tid that matches this thread. has been sitting in `poll()`\n\nsince the process started.\nIts kernel stack, the entire time the process is running, shows it waiting for\nwork:\n\n``` bash\n$ sudo cat /proc/$P/task/<tid>/stack\n[<0>] do_poll.constprop.0+0x315/0x3c0\n[<0>] do_sys_poll+0x1ef/0x290\n[<0>] __x64_sys_poll+0x4e/0x150\n```\n\nThat thread does the entire checkpoint, from inside the process that is being checkpointed.\n\n[What the driver sees](#what-the-driver-sees)\n\nIf the service thread is doing the checkpoint, then it must be making driver\ncalls. So let’s try to watch those. An `LD_PRELOAD`\n\nshim on the target that decodes\neach `ioctl`\n\n’s command number and parameter structThe shim wraps `ioctl`\n\n, matches NVIDIA's `'F'`\n\nmagic, and decodes the\n`NVOS54`\n\n(RM_CONTROL) and `NVOS21`\n\n/`NVOS64`\n\n(RM_ALLOC) parameter structs\nagainst the [open kernel\nmodules](https://github.com/NVIDIA/open-gpu-kernel-modules). The\n[appendix](#appendix-the-diagnostic-hooks) has more details. gives us a per-phase\nhistogram:\n\n| phase | RM_CONTROL | RM_ALLOC | other `NV_ESC_*` | UVM | total ioctls |\n|---|---|---|---|---|---|\n`lock` | — | — | — | — | 2 |\n`checkpoint` | 133 | 25 | 97 | 56 | 793 |\n`restore` | 213 | 124 | 117 | 108 | 1197 |\n`unlock` | — | — | — | — | 2 |\n\nThere is no checkpoint ioctl. All the commands that the driver sees when a\ncheckpoint is performed are ordinary resource-manager ioctls, the same ones\nthat `libcuda`\n\nuses to create and destroy contexts, allocate and free memory,\nand so on. It’s [Checkpoint and Restore in\nUserspace](https://criu.org/Main_Page), but for GPU contexts.\n\n[Coming back](#coming-back)\n\nAt checkpoint the context was torn down completely, so at restore time we start from host buffer and a process with no GPU context.\n\nDecoding the classes restore passes to `NV_ESC_RM_ALLOC`\n\nmakes it clear that\nthe result is basically just context creation (see the previous post for some\nof the\n[details](/what-happens-when-you-run-a-cuda-kernel)),\nrun again from scratch, followed by refilling the fresh allocations from the\nhost image. The memory re-allocations track the anatomy above: restore\nwalks the buffer front to back, re-creating each allocation in the order it was\nserialized.\n\n[Forcing ](#forcing-cuda-checkpoint-to-be-faster)`cuda-checkpoint`\n\nto be faster\n\n`cuda-checkpoint`\n\nto be fasterSo that’s the mechanism. When we first were doing checkpoint and restore\nexperiments to build out our [infrastructure for fast starting inference\nengines](/fast-sglang-starts), this `cuda-checkpoint`\n\nblob frustrated me\nin its opacity: 3s of driver time, that we couldn’t explain or explore. But\nnow we know what it’s doing, maybe we can force it to be fast.\n\nHere’s the benchmark: one process, with 8578 MiB of allocations, run through a full checkpoint cycle:\n\nLocking and unlocking are doing nothing here, because the process doesn’t have\nanything running when it’s being lockedIn general, it looks like `lock`\n\nis the quiesce barrier the checkpoint\nneeds to see a still GPU. You can see it by running it against a process\nrunning a kernel that spins for a known duration, launched without a sync,\n`lock`\n\nblocks for exactly the remaining kernel time. While the process is\nlocked, new launches queue in `libcuda`\n\nand only land after `unlock`\n\n.. But they still take 200ms. We\ncould skip them, because we know that the process is quiesced. But that’s not\nreally sound in general. The reason they take so long is because they’re\nindependent invocations of `cuda-checkpoint`\n\n, which means they spin up contexts\nand open all the device files, etc. But we know they’re actually only writing a\nsingle command word into a pipe.\n\nSo let’s speak their language directlyWe do this with a little client that speaks the pipe protocol. Realistically, we're probably just rebuilding what the C API does., and skip them:\n\nWith the utility out of the way, the remaining time is all driver. Swept across device footprint with a program that just allocates N MiB and idles:\n\n| GPU MiB | lock | checkpoint | restore | unlock |\n|---|---|---|---|---|\n| 450 | 0.3 ms | 231 ms | 197 ms | 1 ms |\n| 642 | 0.3 ms | 287 ms | 203 ms | 1 ms |\n| 1410 | 0.4 ms | 517 ms | 284 ms | 2 ms |\n| 4482 | 0.3 ms | 1407 ms | 571 ms | 4 ms |\n| 8578 | 0.3 ms | 2749 ms | 1056 ms | 4 ms |\n\n`checkpoint`\n\nand `restore`\n\nare both linear in the amount of device memory,\nwhich makes sense since they’re copying it. They run at very different rates\nthough: checkpoint at about 3 GiB/s, restore at about 8. Restore, which has all\nthat context to rebuild, is well over twice as fast as checkpoint, which mostly\njust copies memory out. Neither rate is anywhere near the PCIe ceiling: this\nlink moves pageable host memory at 20.6 GiB/s host-to-device and\n17.4 GiB/s device-to-host, and pinned memory at about 25 either wayPCIe 4.0 x16 here, in principle 31.5 GB/s per direction..\n\nIt turns out almost all the time is going into allocating and deallocating the\nstaging buffer. Timing the individual syscalls inside the 8,578 MiB checkpoint,\nthe `mmap(..., MAP_POPULATE)`\n\nof the staging region alone accounts for 2.08 s\nof the 2.8 s checkpoint, because `MAP_POPULATE`\n\nfaults in every page up front\nand the kernel has to zero each fresh page before handing it over. On restore,\nthe mirror operation is the `munmap`\n\nthat frees those pages (at 435 ms) and\nfreeing is much cheaper than zeroing.\n\nSo the real story looks like this:\n\nHow can we do better?\n\nWithout going too crazy, we can just turn on [Transparent Huge\nPages](https://docs.kernel.org/admin-guide/mm/transhuge.html) (THP). THP lets\nthe kernel automatically choose to back the anonymous mapping with 2 MiB pages\ninstead of 4 KiB ones, and zeroing a 2 MiB page is enormously cheaper per byte\nthan doing it 512 timesA small victory for humanity here, Claude can't launch processes with THP\nenabled. The problem is in bun's `mimalloc`\n\nbuild, which sets\n`prctl(PR_SET_THP_DISABLE)`\n\nfor some kind of internal allocation reason, but\nthen (presumably accidentally) the result percolates out into all of Claude\ncode's subprocesses..\n\n``` bash\n$ echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled\n```\n\nWith huge pages, the registration and unregistration costs collapse:\n\nWhat if we don’t have or want huge pages? or what if we really want to get rid of the allocation completely?\n\n`cuda-checkpoint`\n\nallocates the staging buffer inside closed `libcuda`\n\n, so we\ncan’t work with it nicely. But, the allocation is an ordinary `mmap`\n\nthrough\nlibc, and it’s the only large `MAP_ANONYMOUS | MAP_POPULATE`\n\nmapping the\nprocess makes, so an `LD_PRELOAD`\n\nshim can recognise it, and we can replace the\nresult with whatever we want.\n\nIf we know we’re about to perform a checkpoint, we can create a faulted, zeroed buffer, wait for the mmap, and give it to cuda-checkpoint immediately. And if we know our process is going to keep running after restore, we can unmap the buffer later, when we feel like it. Putting everything together:\n\nCheckpoint ends at 540 ms, at about 15.5 GiB/s — the pageable PCIe copy. Restore’s floor is a little higher (12.7 GiB/s effective); it still has a context to rebuild. The context is of fixed size, so it amortizes over larger transfers.\n\nCan we hit the PCIe ceiling? Not by handing libcuda a pinned buffer: when we\ndo, it doesn’t get used in pinned form. We could try to patch the binary, or do\nsome kind of `ioctl`\n\nhook. But at some point, you’ve got to choose to be\ndone, and this is a reasonable place: by poking and prodding at how\n`cuda-checkpoint`\n\ndoes its work, we’ve sped up the checkpoint/restore cycle 4x.\n\nCan you productionise any of this stuff? THP is a kernel setting, so that’s\neasy enough. `cuCheckpointProcess`\n\nin the C API plays our pipe tricks in a\nsupported C API, so you can pay the 200ms context setup cost once at startup\ninstead of for each verb. In exchange you get stability guarantees; not so much\nwhen, like we do here, you’re writing bytes to a pipe. But our client doesn’t\nneed a CUDA context at all, which is pretty nice. The staging-buffer swap is\nmore fragile: it depends on catching an mmap that closed libcuda happens to\nmake, and there are no ABI guarantees of any kind there.\n\nBut before, this was NVIDIA’s playground. Now we can play too.\n\n[Appendix: the diagnostic hooks](#appendix-the-diagnostic-hooks)\n\nAlmost everything here was read off two `LD_PRELOAD`\n\nshims and `/proc`\n\n, because\n`libcuda`\n\nis closed and the interesting work happens inside a process we can\nwatch but not read the source of. As in the [kernel-launch\npost](/what-happens-when-you-run-a-cuda-kernel), the trick is to interpose\non the ordinary libc calls the driver makes and decode them against the open\nkernel modules.\n\n[Watching the driver’s ioctls](#watching-the-drivers-ioctls)\n\nThe per-phase histogram comes from wrapping `ioctl`\n\n, filtering for NVIDIA’s\n`'F'`\n\nmagic, and decoding the command number and parameter struct. The command\nnumbers live in\n[ nv_escape.h](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/590.48.01/src/nvidia/arch/nvalloc/unix/include/nv_escape.h);\nthe RM_CONTROL and RM_ALLOC structs (\n\n`NVOS54`\n\n, `NVOS21`\n\n) in `nvos.h`\n\n; and the\nallocation class numbers in `src/common/sdk/nvidia/inc/class/`\n\n. To split the\nhistogram by phase, mark the shim’s log at each `cuda-checkpoint`\n\ninvocation and\ndiff the offsets.[Finding the staging buffer, and the counter in it](#finding-the-staging-buffer-and-the-counter-in-it)\n\nTo catch the allocation, wrap `mmap`\n\nand log any large `MAP_ANONYMOUS`\n\nmapping.\nThe staging buffer is the only big one with `MAP_POPULATE`\n\nset. There’s\nprobably something more selective here. To find the contents afterwards, scan\nthe process’s anonymous VMAs (from `/proc/$P/maps`\n\n) for a needle: the first\nfour 128-bit instruction words of the `increment`\n\nkernel works. The device\nglobal turns up as an isolated non-zero integer in a page of zeros near the\nSASS. Reading and writing the image itself is `pread`\n\n/`pwrite`\n\non\n`/proc/$P/mem`\n\n.\n\n[The control protocol](#the-control-protocol)\n\nTo see the command words, wrap `write`\n\nin the *utility* rather than the target\nand dump any write to a pipe fd: you’ll get the opcode-5 rendezvous, the\nhandshake, and the 2064-byte command struct whose first two words are the opcode\nand the action index. It’s almost entirely zeros; the only non-zero fields for a\nplain checkpoint are `word0 = 2`\n\nand `word1 = action`\n\n. Passing\n`--device-map <uuid>=<uuid>`\n\nto a restore fills a few more words — a mapping\ncount and inline source/destination GPU UUIDs — which is the machinery for\nrestoring a checkpoint onto a different physical GPU than it was taken on.\n\n[A minimal client](#a-minimal-client)\n\nPutting the protocol together: the client below does the rendezvous, then\ndrives the state machine one 2064-byte command at a time. It never links\nlibcuda, so there is no init to pay; even invoked cold, once per action,\n`lock`\n\nlands in a few milliseconds. Error handling trimmed.\n\n```\n// Speak cuda-checkpoint's pipe protocol directly, no libcuda needed.\n// usage: ckpt_ctl <pid> <action>...    (lock | checkpoint | restore | unlock)\n#define _GNU_SOURCE\n#include <stdio.h>\n#include <stdlib.h>\n#include <string.h>\n#include <stdint.h>\n#include <fcntl.h>\n#include <unistd.h>\n#include <poll.h>\n#include <sys/ioctl.h>\n#include <dirent.h>\n\nstatic int openfd(int pid, int fd, int flags) {\n    char p[64];\n    snprintf(p, sizeof p, \"/proc/%d/fd/%d\", pid, fd);\n    return open(p, flags);\n}\n\n// The channel is a loopback pipe: the target reads commands and writes replies\n// on the same pipe, so never read until the whole reply has been queued.\nstatic void reply(int fd, uint32_t status[2]) {\n    int avail = 0;\n    while (ioctl(fd, FIONREAD, &avail), avail < 8)\n        poll(&(struct pollfd){fd, POLLIN, 0}, 1, 1);\n    read(fd, status, 8);\n}\n\nint main(int argc, char **argv) {\n    int pid = atoi(argv[1]);\n\n    // find the target's pipe fds\n    char path[64];\n    snprintf(path, sizeof path, \"/proc/%d/fd\", pid);\n    DIR *dir = opendir(path);\n    int pipes[64], np = 0;\n    for (struct dirent *de; (de = readdir(dir)) && np < 64;) {\n        char lnk[64], tgt[64] = {0};\n        snprintf(lnk, sizeof lnk, \"/proc/%d/fd/%s\", pid, de->d_name);\n        if (readlink(lnk, tgt, sizeof tgt) > 4 && !strncmp(tgt, \"pipe:\", 5))\n            pipes[np++] = atoi(de->d_name);\n    }\n\n    // rendezvous: opcode 5 into the lowest pipe; some pipe answers with\n    // 8 bytes naming the command channel's fds\n    uint32_t op5 = 5, chan[2];\n    write(openfd(pid, pipes[0], O_WRONLY), &op5, 4);\n    for (int i = 0, avail = 0; ; i = (i + 1) % np) {\n        int p = openfd(pid, pipes[i], O_RDONLY);\n        if (ioctl(p, FIONREAD, &avail), avail >= 8) { read(p, chan, 8); break; }\n        close(p);\n        usleep(1000);\n    }\n    int rep = openfd(pid, chan[0], O_RDONLY);\n    int cmd = openfd(pid, chan[1], O_WRONLY);\n\n    // each action: 2064-byte handshake (word0=6), then the command itself\n    // (word0=2, word1=action), each acknowledged by an 8-byte status\n    const char *names[] = {\"lock\", \"checkpoint\", \"restore\", \"unlock\"};\n    for (int i = 2; i < argc; i++) {\n        uint32_t buf[516] = {6}, status[2];\n        write(cmd, buf, sizeof buf);\n        reply(rep, status);\n        memset(buf, 0, sizeof buf);\n        buf[0] = 2;\n        for (uint32_t a = 0; a < 4; a++) if (!strcmp(argv[i], names[a])) buf[1] = a;\n        write(cmd, buf, sizeof buf);\n        reply(rep, status);\n        printf(\"%s: status=%u state=%u\\n\", argv[i], status[0], status[1]);\n        if (status[0]) return 1;\n    }\n}\n@misc{doubleword-what-happens-when-you-checkpoint-a-cuda-process,\n  title        = {Reverse-engineering NVIDIA's cuda-checkpoint for faster cold starts},\n  author       = {Fergus Finn},\n  year         = {2026},\n  howpublished = {Doubleword Blog},\n  url          = {https://blog.doubleword.ai/what-happens-when-you-checkpoint-a-cuda-process},\n}\n```\n\n", "url": "https://wpnews.pro/news/reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts", "canonical_source": "https://blog.doubleword.ai/what-happens-when-you-checkpoint-a-cuda-process", "published_at": "2026-07-09 00:00:00+00:00", "updated_at": "2026-07-09 13:51:44.617297+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["NVIDIA", "cuda-checkpoint", "SGLang"], "alternates": {"html": "https://wpnews.pro/news/reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts", "markdown": "https://wpnews.pro/news/reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts.md", "text": "https://wpnews.pro/news/reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts.txt", "jsonld": "https://wpnews.pro/news/reverse-engineering-nvidia-s-cuda-checkpoint-for-faster-cold-starts.jsonld"}}