Deploying open-source LLMs (like Llama-3) or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 (expensive) or rely on serverless scale-to-zero (cold start latency penalty).
To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU platforms.
| Provider | GPU | Median Cold Start | Equiv. Hourly Rate | Scale-To-Zero |
|---|---|---|---|---|
| Modal | ||||
| A100 (40GB) | 1.8s | ~$2.85 / hr | Yes | |
| RunPod Serverless | ||||
| A100 (80GB) | 4.2s | ~$2.59 / hr | Yes | |
| Replicate | ||||
| A100 (80GB) | 6.5s | ~$4.14 / hr | Yes | |
| Together AI | ||||
| H100 Cluster | Instant (Pooled) | Token-based | N/A | |
| Lambda Labs |
A100 (80GB) | VM Boot (~45s) | $1.89 / hr | No | The full benchmark dataset, hardware configurations, and testing scripts are maintained at ServerlessGPUBench.
Raw benchmark metrics are also open-sourced on GitHub: awesome-serverless-gpu-latency.
source & further reading
dev.to — original article
Benchmarking Real-Time Voice AI APIs: Cartesia vs Deepgram vs ElevenLabs (2026)
MCP Design Patterns: 7 Proven Patterns for Building Scalable AI Systems in Java
Stop Using One Frontier Model for Everything: My Role-Based AI Coding Stack