Benchmarking Serverless GPUs: Modal vs RunPod vs Replicate Cold Starts (2026)

A developer benchmarked cold start latencies and costs across serverless GPU platforms, finding Modal fastest at 1.8 seconds on an A100, followed by RunPod at 4.2 seconds and Replicate at 6.5 seconds. The results highlight trade-offs between scale-to-zero convenience and latency for production LLM deployments.

Deploying open-source LLMs like Llama-3 or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 expensive or rely on serverless scale-to-zero cold start latency penalty . To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU platforms. | Provider | GPU | Median Cold Start | Equiv. Hourly Rate | Scale-To-Zero | |---|---|---|---|---| Modal | A100 40GB | 1.8s | ~$2.85 / hr | Yes | RunPod Serverless | A100 80GB | 4.2s | ~$2.59 / hr | Yes | Replicate | A100 80GB | 6.5s | ~$4.14 / hr | Yes | Together AI | H100 Cluster | Instant Pooled | Token-based | N/A | Lambda Labs | A100 80GB | VM Boot ~45s | $1.89 / hr | No | The full benchmark dataset, hardware configurations, and testing scripts are maintained at ServerlessGPUBench https://serverlessgpubench.com . Raw benchmark metrics are also open-sourced on GitHub: awesome-serverless-gpu-latency https://github.com/mrzitoun/awesome-serverless-gpu-latency .