# How We Cut Inference Cold Starts from Minutes to Seconds > Source: > Published: 2026-08-12 15:42:56+00:00 *Co-authored with Netanel Kadosh* You've built a great inference service. Auto-scaling is configured. The model performs beautifully in testing. You ship it to production feeling good. Then a traffic spike hits at 2 AM. New pods start spinning up. Users start waiting. Your on-call phone lights up. You watch the dashboards- nodes are healthy, no errors, but the pods just... aren't ready. Eight minutes pass. Then nine. Then ten. Sometimes fourteen. Fourteen minutes for a service that responds in under 200 milliseconds once it's warm. So you start digging into the cold start timeline to understand where all that time is actually going. This is exactly what we did and when we profiled the startup sequence, the breakdown was humbling: | Stage | Time | Share of Total | |---|---|---| | Node provisioning | ~30s | ~5% | Container image pull & unpack | ~420s | ~70% | Model weight loading | ~120s | ~20% | | Service warmup | ~30s | ~5% | Total Cold Start | ~600s (10m) | 100% | The numbers made the problem clear: **90% of our cold start time was spent simply moving massive files around.** Our container image had bloated to a massive 25 GB. Pulling, unpacking and running an image that large was consuming 70% of our startup time. Inside, we found three different inference backends such as PyTorch, TensorRT-LLM and ONNX Runtime-bundled alongside gigabytes of static weight files. After that, we spent another two full minutes just transferring those weights into the GPU. We didn't need to compromise on a smaller, less capable model nor did we need to pay for more powerful nodes. We just needed to rethink how we packaged our environment, how we pulled it over the network and how we streamed the weights into memory. We had to make three engineering shifts that helped us cut our cold start time from minutes to seconds without changing a single line of application code. If you show a DevOps engineer a slow cold start, their immediate knee-jerk reaction is almost always: *"The image is too big, shrink it."* We are no different. Naturally, our very first instinct was to put this container on a strict diet. At 25 GB, there was simply too much bloat inside. We did a deep dive into our image layers using `dive` and identified several indirect dependency sinks. Deleting a few temp files wasn't going to fix this. We had to ask: *What does the inference service actually need at runtime?* We started with a heavy `nvidia/cuda` base image. In an EKS environment running Bottlerocket nodes, the NVIDIA device plugin handles driver exposure to workloads at runtime. Packaging full CUDA dev toolkits into the application image was redundant. We switched to `python:3.13-slim` , shedding several gigabytes immediately. We discovered TensorRT components were being installed twice: the C++ development package via `apt` and the Python runtime via `pip` . The C++ dev libraries weren't required for runtime inference, so removing them eliminated another 3 GB. Default PyTorch GPU wheels pull in massive libraries like NCCL, cuSPARSE and Triton. While crucial for distributed training, these added roughly 4.5 GB of unused weight for single-node inference. We considered mounting Python's `site-packages` externally via an S3 CSI driver to make the image smaller. However, Python imports touch thousands of small files during boot. Turning local disk reads into thousands of network requests replaced a slow image pull with an even slower import phase. We rejected this approach. For mixed-model workloads (like Silero VAD paired with LLM generation), `tensorrt_llm` forced heavy PyTorch dependencies. Even after our initial cleanup, the image was still stubbornly sitting around 15 GB. We refactored the runtime stack: we converted Silero VAD to ONNX (~50 MB) and adopted **CTranslate2** (~100 MB) for LLM generation. This gave us native tensor parallelism without PyTorch or MPI overhead. Combined, these changes stripped a massive 15 GB of bloat from our base image, bringing our final footprint down to 10 GB. We patted ourselves on the back, deployed the leaner container and checked the metrics. Reality hit us fast: a 10 GB image is still incredibly heavy and pulling it was still agonizingly slow. The classic "just shrink the image" DevOps reflex had taken us as far as it could. To go faster, we couldn't just change the size of the data- we had to change how the data was downloaded. When we looked under the hood to see why the pull was still taking so long the real culprit became clear: the default container runtime I/O. Runtimes fetch image layers sequentially over a single network stream and decompress them serially on a single CPU core. On modern cloud nodes with high-bandwidth interfaces and dozens of CPU cores, this approach leaves most host resources completely idle. To fix this, we enabled **Seekable OCI (SOCI) Parallel Pull Mode** on our EKS nodes. SOCI parallelizes both network downloads and extraction. It splits layers into smaller chunks, executes concurrent HTTP range requests to saturate network bandwidth and distributes decompression across all available CPU cores. You'd expect a fix this powerful to be complicated, but it was actually the easiest part of the project. All we had to do was pass this tiny configuration block into our Bottlerocket node user-data ``` [settings.container-runtime] snapshotter = "soci" [settings.container-runtime-plugins.soci-snapshotter] pull-mode = "parallel-pull-unpack" [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack] max-concurrent-downloads-per-image = 20 concurrent-download-chunk-size = "16mb" max-concurrent-unpacks-per-image = 10 discard-unpacked-layers = true ``` When you parallelize network fetches and CPU decompression, **disk I/O becomes your new bottleneck**. Because SOCI writes decompressed data directly to disk to maintain predictable memory usage, nodes must be backed by fast storage. We ensured our EKS nodes were backed by high-performance NVMe instance store disks or EBS volumes provisioned for at least **400 MiB/s throughput**. **Result:** Image pull times for our base containers dropped from nearly 5 minutes down to **55 seconds** an **87% reduction in pull latency**. Even with a smaller image and parallel pulls, packaging model weights inside a container image creates a fundamental architectural flaw: **it treats model data as code.** Every time a model updated, we had to rebuild, push and pull a massive new container image. The obvious solution was to decouple the model weights from the container image entirely and store them externally in Amazon S3. But getting those external weights into the GPU introduced a new problem. To get the endless flexibility of S3 without a massive latency penalty, we adopted the open-source **NVIDIA Run:ai Model Streamer**. Traditional weight loading from remote storage is agonizingly slow because it forces a sequential hop. The Run:ai streamer fixes this by executing concurrent HTTP range requests directly against S3 and bypassing the disk entirely. Here is how the workflow changes: **The Traditional load (Sequential):** `S3` ➔`[Local Disk]` ➔`[CPU RAM]` ➔`[GPU VRAM]` (Each step waits for the massive download to finish before moving to the next) **The Streamer Architecture (Concurrent):** ``` [ S3 Bucket ] │ │ (Parallel Network Pulls) ▼ [ CPU RAM ] (Acts only as a transit buffer, no disk staging) │ │ (Direct PCIe DMA Transfer) ▼ [ GPU VRAM ] ``` Instead of downloading a 15GB file to disk and moving it step-by-step, the streamer does everything at once. It uses your system RAM simply as a transit pipeline. As parallel requests pull data from S3, those chunks are instantly injected into the GPU using Direct Memory Access (DMA) over PCIe. Local disk staging is bypassed entirely and network downloads overlap perfectly with GPU ingestion. Integrating this into `vLLM` required zero application code changes. With S3 read permissions configured, we simply updated the vLLM execution command: ``` vllm serve s3://my-bucket-name/my-weights \ --load-format runai_streamer \ --model-loader-extra-config '{"concurrency": 32}' ``` By setting concurrency to 32, the streamer fully saturated our node's network bandwidth. **Result:** Loading 15 GB of model weights dropped from 120 seconds down to **4.9 seconds** a **96% reduction in load time**. By addressing each step, we transformed our start time from a 10+ minute liability into a responsive, scalable infrastructure: | Bottleneck | Architectural Solution | Impact | |---|---|---| Container Bloat (25 GB) | Removed PyTorch/CUDA bloat, adopted CTranslate2 + ONNX | ~60% image size reduction | Sequential Image Pull | SOCI Parallel Pull Mode (EKS) | ↓ 87% pull time (~5m → 55s) | Sequential Weight Load | Direct S3-to-VRAM Streaming (Run:ai) | ↓ 96% load time (~120s → 4.9s) | These optimizations build directly on top of each other. We stripped unnecessary dependencies from the image, parallelized the remaining layer pulls with SOCI and decoupled model weights entirely by streaming them from S3. When your GPU services take minutes to start, don't immediately assume you need pre-warmed fleets, smaller models or to change your business logic. First look at where data is moving and eliminate sequential I/O, you might just find that you can fix it all without changing a single line of application code.