Semi-Persistence: Blazing-Fast Model Swapping for Complex Scheduling

Snowflake AI Research introduced Semi-Persistence, a model-serving approach that keeps model weights in persistent CPU memory to enable fast swapping of GPU copies, reducing end-to-end sleep and wake-up cycles by 5.6× to 19.9× across models from 2B to 397B parameters and achieving sub-second swapping for single-GPU models. The technique addresses the high cost of traditional vLLM sleep modes, which either copy weights from GPU to CPU or reload from storage on every cycle.

Open-source models are becoming increasingly compelling on both quality and cost. Specialization can push these economics even further: smaller models trained for focused tasks can match or exceed frontier-model quality at a fraction of the inference cost. Snowflake's Arctic Text-to-SQL models https://www.snowflake.com/en/blog/engineering/enterprise-text-to-sql-arctic-r2/ , for example, demonstrate frontier-level quality at about 25× lower inference cost per token than a closed-source frontier API.<sup 1</sup Snowflake AI Research is also working to reduce the cost and complexity of post-training open source models. This includes open source RL backends such as ZoRRo https://www.snowflake.com/en/blog/engineering/zorro-enterprise-rl-training/ and Arctic RL https://www.snowflake.com/en/blog/engineering/arctic-rl-open-source-backend/ , as well as Snowflake-managed services such as Snowflake Cortex Training https://www.snowflake.com/en/blog/snowflake-for-ai-enterprise-ai-platform/ . Serving presents a different system problem. A serving platform may need to host many models whose demand changes over time. Keeping every model permanently loaded wastes expensive GPU capacity; moving models in and out of GPU memory is useful only if the swap itself is fast enough to stay out of the critical path. Semi-Persistence makes model swapping practical. It keeps model weights in a persistent CPU-memory pool while allowing their GPU copies to come and go with demand. The lightweight model skeleton moves independently across disk, CPU, and GPU, while weights can be restored rapidly from CPU memory when the model is needed again. Across models from 2B to 397B parameters, Snowflake internal benchmarks show that Semi-Persistence reduces the end-to-end sleep and wake-up cycle by 5.6× to 19.9× , achieving sub-second model swapping for single-GPU models Figure 1 . Figure 1: End-to-end sleep and wake-up latency for vLLM and Semi-Persistence. Semi-Persistence persists the weights in CPU memory and optimizes their movement back to the GPUs. Why existing model swapping falls short Model swapping lets multiple models time-share the same GPUs: when a model becomes idle, its GPU memory can be reclaimed and used by another model; when demand returns, the model is restored and resumes serving. This makes it possible to increase GPU utilization without dedicating capacity to every model at all times. But model swapping only works as a serving primitive if the transition is fast. A slow eviction delays reuse of the GPU, while a slow restore adds directly to the time between request arrival and token generation. In practice, the swap path needs to avoid repeatedly moving or reloading the model's full weights whenever its GPU residency changes. The standard way to swap models in vLLM is through its two sleep modes https://docs.vllm.ai/en/latest/features/sleep mode/ , but each pays one of these expensive paths. Level 1 copies model weights from GPU memory to CPU memory before releasing the GPU. This supports a faster wake-up, but every sleep requires a full GPU-to-CPU transfer. Level 2 discards the GPU weights immediately, making sleep faster, but the weights must be loaded again from storage before the model can serve requests. Either way, every sleep/wake cycle pays a slow path — exactly what the swap budget cannot afford. Inference weights often do not change while a model is running. Instead of copying them back from the GPU as in Level 1 or repeatedly reading them from storage as in Level 2 , we can keep one long-lived CPU copy and treat the GPU copy as temporary. Semi-Persistence Semi-Persistence is a model-serving approach that makes model swapping fast — and manages many models across shared GPUs — by keeping model weights ready in CPU memory and moving models in and out of GPU memory as demand changes. Keep weights ready in CPU memory. Semi-Persistence keeps model weights in a persistent pinned CPU-memory pool. When a model becomes idle, its GPU copy can be discarded immediately because the weights remain available in CPU memory. The lightweight model skeleton, meanwhile, moves between disk, CPU, and GPU independently of the weights. Restore them quickly. When requests arrive, Semi-Persistence transfers the weights back to the GPUs. It shards the weights across the node, loads them through multiple devices in parallel and uses both PCIe and NVLink to maximize transfer bandwidth. Adapt as demand changes. Semi-Persistence also manages the models sharing the GPUs. It can load and evict models, migrate them between GPUs and nodes, consolidate smaller models to free capacity, and pause and resume requests during a move. These operations happen asynchronously, allowing model movement and request processing to overlap. We describe how each of these works in the Deep dive below. Performance Figure 1 shows benchmarks of the end-to-end sleep and wake-up cycle — the time to release a model's GPU memory plus the time to make it ready to serve again — across eight open-weight models, comparing vLLM's Level 2 sleep mode against Semi-Persistence under otherwise identical configuration. For single-GPU models, latency falls from 1.2–13.5 seconds to 214–801 milliseconds . For multi-GPU models, it falls from 10.5–40.6 seconds to 1.75–7 seconds . Overall, Semi-Persistence delivers a 5.6× to 19.9× speedup across models ranging from 2B to 397B parameters.<sup 2</sup The advantage grows for the largest frontier models. For trillion-parameter models such as DeepSeek-v4-pro and GLM-5.2-FP8, a cold start in vLLM takes 13.3–15.5 minutes, whereas Semi-Persistence restores them in 32.8 seconds Figure 8, detailed under Frontier models — roughly 24× to 28× faster . What are we open-sourcing today? We implemented Semi-Persistence around the vLLM backend that we use today, but the same instance-based design applies to other engines. We expose the semi-persistent capabilities to an instance as composable primitives. Anyone can use these primitives directly to add Semi-Persistence to their own serving stack or implement a new backend behind the same interface. On top of the Semi-Persistence wrapper , we are also open-sourcing our experimental complex-scheduling system: an orchestrator that drives many decentralized instances dynamically, and a dashboard for observing the live state of the system. Everything is available at ArcticInference/semi persistence https://github.com/snowflakedb/ArcticInference/tree/main/arctic inference/semi persistence . Deep dive Semi-Persistence is built around four ideas. First, it separates model skeletons from weights, allowing many specialized models to share the same runtime. Second, it restores weights at very high speed by saturating multiple interconnects in the communication hierarchy at the same time. Third, it orchestrates models asynchronously, enabling many instances to load, sleep, evict and migrate concurrently without a centralized bottleneck. Finally, it turns these primitives into complex, self-healing scheduling patterns that keep GPUs highly utilized as workloads change. Together, these techniques make dynamic model swapping practical for production serving. The Deep dive is organized around these four highlights: 1. Separating skeletons from weights: the reusable model skeleton is cached once while only the weights change per specialization, so many models share a single runtime. 2. Fast weight restoration: weights are streamed from a pinned CPU pool and assembled on the GPUs in parallel over PCIe and NVLink, restoring a model at full node bandwidth. 3. Asynchronous orchestration: each instance runs as an independent state machine coordinated by a lightweight reservation system, overlapping data movement without a central scheduler. 4. Complex scheduling: a few simple per-instance rules give rise to self-healing behaviors — eviction, migration and consolidation — that keep GPUs busy as traffic shifts. Separating skeletons from weights Semi-Persistence manages the skeleton and weights of a model independently. The skeleton contains the model structure and runtime state — typically a few gigabytes — while the weights define a particular specialization. Many specialized models share the same skeleton but differ only in their weights. Semi-Persistence caches each skeleton once and stores all weights in a persistent pinned CPU-memory pool. Because the weights are stored in vLLM's native format, any compatible skeleton can restore immediately without rebuilding or converting the model. The first time a configuration is seen, Semi-Persistence saves the skeleton image to disk and later restores it with CRIU Checkpoint/Restore In Userspace , so a single runtime can efficiently serve many specialized models. Figure 2: Semi-Persistence tracks each instance's state Saved, Checkpoint, Sleep, Up and manages weights and skeleton independently. Weights live in the pinned memory pool; the skeleton moves between disk image, CPU and GPU via CRIU and CUDA checkpoint/restore. Figure 2 defines the states an instance's skeleton moves through in the hardware hierarchy. Saved keeps the instance image ready on disk; Checkpoint holds the model entirely on CPU with no GPU context; Sleep keeps the skeleton on the GPU but consumes no compute and negligible GPU memory; and Up is fully deployed, serving generation with no added latency. We use different sequences of optimized primitives, shown in the figure, to move instances across the hierarchy. Fast weight restoration Persistent weights are only useful if they can be restored at GPU speed. The main bottleneck in weight restoration is the PCIe path between CPU and GPU. To saturate every switch, Semi-Persistence uses a pinned memory pool that 1 shards the weights across the pool, 2 transfers them in parallel with multiple GPUs over distinct PCIe switches, and 3 gathers them over NVLink. As a result, we use the aggregate bandwidth of the node full PCIe and NVLink rather than a single PCIe-only path. Figure 3 shows the weight restore for TP=1 and TP=2 models. Figure 3: Parallel weight restoration from pinned memory. Weights are sharded across the node, streamed from the pinned pool into 8 GB staging buffers over PCIe H2D and assembled on the target GPUs over NVLink D2D . The memory pool has a set of long-lived daemon processes that perform the host-to-device H2D transfers simultaneously, while the instance initiates the device-to-device D2D communication that gathers the shards. If instance and pool processes are not co-located, the shards are transferred using inter-process communication IPC over NVLink. We pipeline the hops in Figure 3 for overlapping H2D and D2D communications, keeping PCIe saturated throughout the restore. On our test platform an AWS p5en.48xlarge with 192 vCPUs, 2 TiB of memory and 8 H200 GPUs , PCIe bandwidth is highly sensitive to GPU placement: when multiple GPUs share a PCIe switch, bandwidth drops sharply Figure 4 . Even with the best PCIe-only placement, adding NVLink to gather the shards improves restore throughput by about 3.7× TP1 , 2× TP2 and 1.8× TP4 . Figure 4: GPU placement and the NUMA/PCIe topology top . Bandwidth suffers when GPUs share a switch left ; routing weights through staging GPUs and NVLink recovers throughput right . Asynchronous orchestration Fast restores alone are not enough. A production serving system must continuously load, evict, migrate and pause models as demand changes, and controlling every instance and interaction from a centralized scheduler causes significant software overhead. Therefore, Semi-Persistence treats each instance as an asynchronous state machine. Instances independently decide their own state transitions when a request arrives, which decentralizes scheduling and lets operations proceed concurrently, naturally overlapping disk I/O, CPU-GPU transfers and execution. To coordinate these independent decisions safely, Semi-Persistence uses a global reservation system that reserves a GPU slot the moment a request arrives — with low latency. Slots logically partition GPU memory: a quarter, half, or full GPU for small models, up to multiple GPUs for larger ones Figure 5 . A slot does not actually allocate memory on the hardware but blocks other instances from the same memory region; an instance that cannot obtain a slot of its size waits in the Checkpoint state, and a freed slot is handed to the next instance in the queue. This lets instances make independent moves without ever running out of memory. Figure 5: Orchestrator dashboard schematic . Each GPU is divided into quarter-, half-, or full-GPU slots — shown as stars in brackets, for example, GPU 3 | | — so several models can share one GPU here, three models on GPU 3 . The dashboard also surfaces GPU memory, instance state up vs. waiting , the CPU pinned pool, the live request log and the image cache. Complex scheduling The most powerful behaviors in Semi-Persistence are not programmed centrally — they emerge. From a few simple per-instance rules, complex scheduling behaviors such as eviction, migration, consolidation and waiting arise on their own, and the system effectively self-heals fragmentation with no central brain. Consider a concrete example of head-of-line HOL blocking Figure 6 . Small models run on GPU 1 and GPU 3, and a large model runs on GPU 2. Two requests then arrive back-to-back: one for a large model and one for a small model. In a naive FIFO queue, the large model head-of-line-blocks the smaller one — even though enough capacity for the small model already exists — so both wait here for 7.6 seconds. Semi-Persistence self-heals this situation by migrating and consolidating the small models onto GPU 1, freeing a contiguous slot for the large model and dropping the wait time to zero. Migration is driven by two primitives: pause saves the in-flight token IDs to CPU and releases the instance's GPU memory making it evictable and resume automatically moves the instance to its new GPU, re-prefills the KV cache and continues generation exactly where it left off — no additional commands required. Figure 6A. With HOL blocking: 7.6 s wait Figure 6B. Without HOL blocking: 0 s wait Figure 6: Self-healing scheduling. Migrating and consolidating small models frees a contiguous slot for the waiting large model, eliminating head-of-line blocking 7.6 s → 0 s . Migration involves more than moving weights. The skeleton is "sticky" to GPUs: it owns CUDA runtime state and distributed communication infrastructure, and it must leave no residual CUDA context behind when checkpointed. Semi-Persistence uses the CUDA checkpoint driver to capture the context — only the lightweight skeleton, so it stays fast — and CUDA restore to remap GPU addresses on the destination. For multi-GPU instances, it tears down vLLM's static communication infrastructure NCCL, IPC buffers and reinitializes it on the new GPUs; because reinitializing NCCL invalidates the captured CUDA graph, the graph is recaptured. As a result, migration latency is ultimately bounded by CUDA checkpoint and restore a driver operation , as seen in Figure 7. Figure 7: Migration latency breakdown. After optimization, cost is dominated by CUDA checkpoint/restore rather than restoring weights. Frontier models Semi-Persistence is not limited to small specialized models — it applies equally to the largest open source frontier models, such as DeepSeek, GLM, and Kimi. Teams can post-train and specialize these trillion-parameter models for specific tasks, then serve them at roughly 5×–10× lower inference cost per token than closed-source frontier APIs.<sup 3</sup Because such models occupy a full node, fast swapping matters even more: without it, simply bringing a model up can take minutes. Figure 8: left Large-model sleep/wake cycles and right vLLM cold-start vs. Semi-Persistence load-from-image times. Figure 8 makes the difference concrete. A cold start in vLLM initializes each model from a local cache and still takes several minutes — about 15.5 minutes for DeepSeek-v4-pro and 6.26 minutes for Kimi-K2.6. Semi-Persistence instead restores the same models directly from the image cache in roughly 33 seconds, more than an order of magnitude faster, so even the largest models can be swapped in and out on demand.<sup 4</sup Get started Getting started takes only a few steps: 1. Install vLLM https://github.com/vllm-project/vllm . 2. Install ArcticInference https://github.com/snowflakedb/ArcticInference 3. Explore the examples, then run the scripts below to reproduce our results. Run the server sudo python semi persistence/server.py --gpu 1,2,3 --image-cache /code/image-cache Run the client python from client import OrchestratorClient as cl cl.init "demo hol.jsonl" cl.generate "32b bird", "who are you?", 2000 cl.generate "model 8", "what is the capital of France?", 3000 cl.generate "model 10", "What is the meaning of life?", 6000 cl.pause "model 8" cl.generate "spec 30b", "who are you?", 2000 cl.generate "spec 8b", "who are you?", 3000 cl.resume "model 8" cl.wait Monitor with the dashboard python semi persistence/dashboard.py --interval 0.1 That's it. Semi-Persistence is available at https://github.com/snowflakedb/ArcticInference/tree/main/arctic inference/semi persistence https://github.com/snowflakedb/ArcticInference/tree/main/arctic inference/semi persistence — reproduce our results, or serve your own specialized models on shared GPUs. 1. Based on Opus 4.7 API $6.82 per million tokens vs. our cost of running Arctic-Text2SQL-R2 32B on H200 $0.27 per million tokens , assuming 20K input + 2K output tokens per request. 2. The Figure 1 results were measured on AWS p5en.48xlarge instances 192 vCPUs, 2 TiB of host memory and 8 NVIDIA H200 GPUs using vLLM v0.18.0. 3. Based on publicly listed per-token prices on OpenRouter as of August 2026. GLM-5.2 is listed at $1.19 per million input tokens and $3.74 per million output tokens, compared with $10.00 / $50.00 for Claude Fable 5 and $5.00 / $25.00 for Claude Opus 4.7. At a 3:1 input-to-output token mix that works out to about $1.83 per million tokens for GLM-5.2 versus $20.00 for Claude Fable 5 — roughly 10× lower — and roughly 5× lower than Claude Opus 4.7. Listed prices for open-weight models vary by provider and change frequently. Sources: GLM-5.2 https://openrouter.ai/z-ai/glm-5.2 providers , Claude Fable 5 https://openrouter.ai/anthropic/claude-fable-5 , Claude Opus 4.7 https://openrouter.ai/anthropic/claude-opus-4.7 , Qwen3-32B https://openrouter.ai/qwen/qwen3-32b . 4. Experiments with Frontier models Figure 8 were made on 8×B300 nodes using vLLM v0.24.0. All the other experiments were made on the 8×H200 nodes.