I built a failover daemon for Vast.ai spot GPUs, found 5 real bugs testing it

The real cost of a Spot GPU eviction was never the few minutes of downtime — it's the hours of training progress that vanish with it. SpotWarp is a lightweight, 100% local Python daemon that runs continuous automatic backups of your workspace in the background, so an eviction never costs you your work. Sub-minute cross-cloud failover (Vast.ai ⇄ RunPod) is what turns that protected workspace into a hands-off recovery, but the backup is the part that actually saves you.

Save up to 70% on GPU compute bills by safely using Spot pricing instead of on-demand — the eviction risk that normally makes that a gamble is exactly what SpotWarp removes.

• 🛡️ Continuous Automatic Backup — the real safety net: Runs high-speedrsync

/scp

incremental backups of your workspace to your local machine in the background, the whole time your instance is running — not just triggered after an eviction is detected. This is the feature that actually prevents loss; everything else below just makes recovering from it fast and hands-off. - 🏁 Parallel Candidate Racing: On eviction, SpotWarp rents several replacement candidates concurrently instead of trying them one at a time — a single slow or dead host no longer adds minutes to your downtime. Typical failover: well under a minute. - 🌐 Cross-Cloud Fallback (Vast.ai ⇄ RunPod): If your primary cloud has zero matching candidates at the moment of eviction, SpotWarp automatically bridges to RunPod — spot pricing first, retrying on-demand if RunPod has no spot capacity for that GPU model — so your workload stays protected instead of failing outright. - ↩️ Automatic Cost-Optimizing Failback: A bridge-cloud replacement is never left running indefinitely at the higher rate. SpotWarp keeps checking your original cloud in the background and migrates the workload back the instant a cheaper matching candidate reappears — verified end-to-end on a single live instance: rented on Vast.ai, evicted, bridged to RunPod, then automatically migrated back to Vast.ai once capacity returned. - 💸 CFO-Approved GPU Savings: Safely exploit cheap Spot pricing on Vast.ai. SpotWarp gives you the reliability of a Dedicated On-Demand GPU for the price of a Spot instance. - ✅ Real Connectivity Verification: Replacement hosts are confirmed reachable via an actual SSH handshake before they're trusted — not a proxy signal like a Jupyter API ping that can report false negatives on a perfectly healthy host. - 🔒 Zero-Trust Security (100% Local): Your cloud provider API keys (VAST_API_KEY

,RUNPOD_API_KEY

) stay on your local PC. Rented containers never see your cloud credentials. - 📦 Zero-Configuration: No need to install daemons, cron jobs, or synchronization tools inside the remote container. - ⚡ Training Auto-Resumer: Automatically restarts your training scripts (--resume-cmd

) in the background of the new container, pointing directly to your restored, backed-up checkpoints.

Install the official package via Pip:

pip install spotwarp

Set your Vast.ai API key on your local machine. Add a RunPod API key too if you want the cross-cloud bridge (optional, but recommended — it's what keeps a bad-inventory day on Vast.ai from becoming downtime):

export VAST_API_KEY="your_vast_api_key"
export RUNPOD_API_KEY="your_runpod_api_key"   # optional, enables cross-cloud fallback + auto-failback

$env:VAST_API_KEY="your_vast_api_key"
$env:RUNPOD_API_KEY="your_runpod_api_key"

Run the guard daemon on your local PC. Point it to your license key and define how your training should resume:

spotwarp start --license-key YOUR_SPOTWARP_KEY --resume-cmd "python /workspace/train.py --resume"
[Rented GPU Host]                     [Local PC (Client)]                    [New GPU Host]
  (Active Workload)
         │                                     │                                    │
         │ ─── (Sync: rsync/scp delta) ──────> │ (Cached Workspace)                 │
         │                                     │                                    │
    [🚨 Evicted!]                              │                                    │
         ❌ ─── (Detected within 5s) ────────> │                                    │
                                               │ ─── (Race 4 candidates) ──────────> │
                                               │      cheapest of the ready pool wins │
                                               │ ─── (Restore Workspace) ──────────> │
                                               │ ─── (Nohup Resume Command) ──────> [Run Workload]
                                                                                      (Continuing!)

  If Vast.ai has zero candidates:
                                               │ ─── (Bridge to RunPod) ────────────> [Temp Replacement]
                                               │ ⋯ keeps watching Vast.ai in the background ⋯
                                               │ ─── (Vast.ai capacity returns) ──> [Migrate back, cheaper]

Eviction Detection: SpotWarp polls the cloud API every 5 seconds. If eviction is detected, it triggers failover immediately.** Parallel Candidate Racing**: SpotWarp rents up to 4 matching-GPU candidates concurrently and picks the cheapest one that proves SSH-reachable, instead of trying offers one at a time.Cross-Cloud Bridge (if needed): If no Vast.ai candidate exists at that moment, SpotWarp automatically rents on RunPod instead — spot pricing first, on-demand as a second attempt — so the workload is never left completely unprotected.Delta Sync Restoration: SpotWarp transfers the cached workspace folder to the new container and confirms it's reachable over a real SSH connection.Nohup Handover: SpotWarp connects via SSH to trigger the--resume-cmd

in the background.Auto-Failback: If step 3 was needed, SpotWarp keeps quietly re-checking Vast.ai for that GPU model. The moment a candidate reappears, it migrates the workload back automatically and releases the bridge-cloud host — no manual intervention, no forgotten expensive instance left running.

SpotWarp is distributed under the MIT License. The code executes 100% locally in user space on your local computer, ensuring full transparency and compliance.