Scheduling jobs across Slurm clusters (and K8s, and cloud) from one place

SkyPilot, an open-source framework, enables users to schedule jobs across multiple Slurm clusters, Kubernetes, and cloud providers from a single interface, addressing the pain points of managing multiple clusters manually. The tool treats all clusters as one resource pool, automatically finding available GPUs and handling environment differences, which is significant given that 60% of Top500 supercomputers rely on Slurm.

Slurm dominates large-scale computing - 60% of Top500 supercomputers https://en.wikipedia.org/wiki/Slurm Workload Manager rely on it for workload management. If you’re working in ML research or HPC, having access to multiple GPU clusters sounds like a good problem to have. More resources = more compute, which means faster experiments and shorter iteration cycles. In practice, however, managing multiple Slurm clusters is anything but enjoyable, it’s a manual, error-prone process that becomes a bottleneck as your team scales. Imagine this: your primary cluster is full, so you SSH to a different login node, remember which partition names are different there, tweak your job script, and resubmit. You rinse and repeat across three, four, or five different clusters. You start thinking “there’s gotta be a better way.” What if you could treat all your Slurm clusters as a single resource pool? What if your jobs could automatically find and run on whichever cluster has available GPUs? That’s exactly what SkyPilot enables. The multi-cluster pain points the-multi-cluster-pain-points So what makes managing multiple Slurm clusters difficult? No unified view of resources Each Slurm cluster is self-contained. To see what GPUs are available across your clusters, you need to: - SSH to cluster-a’s login node - Run sinfo or check the queue - SSH to cluster-b’s login node - Run sinfo again - Compare in your head or in a spreadsheet - Decide where to submit This becomes especially time consuming when you’re looking for specific GPU types e.g. H200s vs H100 vs B300s or need multi-node allocations. There’s no single command that shows you “here’s what’s available everywhere.” Manual failover when clusters are full When your primary cluster hits capacity, you have to: - Notice that your job is stuck in the queue - Realize you should try another cluster - SSH to a different login node - Adapt your submission script different partition names, different module environments, different paths - Submit again - Hope that cluster has capacity For teams running continuous training pipelines or parameter sweeps, this manual intervention breaks the workflow. You can’t just submit 100 jobs and let them find capacity-you have to babysit the process. Inconsistent environments across clusters Different Slurm clusters often have: - Different partition naming schemes gpu vs b300 vs h100-cluster - Different module environments module load cuda/12.1 vs module load cuda/12.4 - Different storage mounts home directory paths, scratch filesystem locations - Different networking configurations Every time you move a job to a different cluster, you’re effectively debugging environment differences. This is both tedious and it becomes a source of subtle bugs when jobs fail due to environment mismatches. Bad observability = low utilization Without visibility across clusters, you can’t make smart scheduling decisions: - Should you wait for H100s on cluster-a, or use available B300s on cluster-b? - Should you preempt lower-priority jobs, or fail over to another cluster? - Which cluster will give you the fastest queue time? You’ll end up underutilizing your infra based on incomplete information. What about native Slurm multi-cluster support? Slurm does offer multi-cluster operation and federation, but both have limitations. Multi-cluster mode routes jobs to the cluster with the earliest start time at submission, but “Slurm makes no subsequent effort to migrate a job to a different cluster” after that. Federation provides peer-to-peer scheduling with replicated “sibling jobs,” but the docs note it’s “not intended as a high-throughput environment” and suggest configuring fewer clusters if you’re scheduling more than 50,000 jobs a day. It also requires that all compute nodes be reachable from all submission hosts, which is a tall order for clusters in separate data centers. SkyPilot’s unified interface for Slurm skypilots-unified-interface-for-slurm SkyPilot https://skypilot.ai is a control plane that turns fragmented compute into one unified AI compute pool. While SkyPilot supports single or multiple Kubernetes really well, it also has first-class support for Slurm clusters, including the ability to manage multiple Slurm clusters as a unified resource pool. With SkyPilot, you get: Unified resource view : See GPUs across all your clusters with a single command-no more SSH-ing to different login nodes to check availability. Automatic cluster selection and failover : Submit a job and SkyPilot finds a cluster with capacity. If that cluster is full, it tries the next one automatically. Consistent environments : Write your task definition once. The same YAML works on any Slurm cluster and even on cloud VMs or Kubernetes . Unified job management : Monitor, log, and cancel jobs across all clusters from one place. Cloud overflow : When all your Slurm clusters are saturated, automatically burst to cloud VMs for additional capacity. The rest of this post walks through setup, usage, and real-world workflows. Getting started getting-started Prerequisites You need SSH access to your Slurm clusters. SkyPilot uses SSH to connect to login nodes and submit jobs via sbatch , just like you would manually. Configuration Configure your clusters in ~/.slurm/config using SSH config format: This follows the standard SSH config format, so if you’re already using SSH configs for these clusters, you can reuse them. Verify setup Check that SkyPilot can see all your clusters: You can also test connectivity to a specific cluster: SkyPilot automatically discovers available clusters from your config and tests connectivity. Key features key-features Unified resource view The SkyPilot dashboard provides a unified view of all your GPU clusters: If those who prefer the CLI, see GPUs across all your clusters with one command: This gives you instant visibility into what’s available where. No more SSH-ing to different login nodes to check queue status. Automatic cluster selection and failover Write your task once: Note: The GPU name e.g., H100 , B300 , L4 must match what’s configured in your Slurm cluster’s GRES settings. SkyPilot converts this to the appropriate --gres=gpu:H100:8 directive when submitting to Slurm. Launch it: SkyPilot will: - Check resource availability across all configured Slurm clusters - Select the cluster with available H100s - Submit the job via sbatch - If the first cluster is full, automatically try the next one - Handle environment setup consistently across clusters You don’t have to specify which cluster to use - SkyPilot finds capacity and schedules there. Consistent environment across clusters SkyPilot abstracts away cluster-specific details: Partition names : Specifying partitions is optional; SkyPilot can automatically map GPU requests to the right partitions on each cluster Module environments : Your setup block runs on every cluster, so dependencies are consistent Environment variables : SkyPilot provides standardized variables that work everywhere Your task definition is portable. The same YAML works on any Slurm cluster and even on cloud VMs or Kubernetes if needed . Environment variables SkyPilot sets these environment variables automatically for multi-node jobs: | | | | | Resolved, newline-separated node IPs | | | Total number of nodes | | | Node rank 0 to N-1 | | | GPUs per node | | | Unique task identifier | These variables make your code portable-the same script works on Slurm, cloud VMs, and Kubernetes. Interactive development across clusters Use salloc -style interactive sessions that can land on any cluster: SkyPilot finds an available H100 across all your clusters and gives you a dev cluster. SSH in: You’re now on a GPU node on whichever cluster had capacity. You don’t need to know which one-the environment is consistent. You can even set up VSCode remote development with SkyPilot dev clusters see instructions here https://docs.skypilot.ai/en/latest/examples/interactive-development.html vscode . Job management across clusters See all your jobs across all clusters: Cancel or stop jobs uniformly: No need to remember which cluster each job is on or SSH to different login nodes to cancel jobs. Command reference Here’s a quick reference for translating Slurm commands to SkyPilot: | | | Interactive allocation | | | Run command | | | Submit batch job | | | View jobs | | | Cancel job or allocation | | | View resources | | | View logs | | | The key difference: with SkyPilot, you don’t need to specify which cluster to use. SkyPilot automatically selects the best available cluster based on resource availability. Real-world workflow real-world-workflow Let’s walk through a typical workflow: training multiple model variants in parallel. Define your training task Launch multiple variants SkyPilot automatically distributes these across your clusters based on availability. Maybe train-7b lands on slurm-cluster-a, train-13b on slurm-cluster-b, and train-70b on slurm-cluster-c-all without you specifying or caring. Monitor progress Check logs from any cluster No SSH-ing to the right login node-SkyPilot handles it. Advanced features advanced-features Specifying cluster preferences If you want to target a specific cluster due to data locality, cost, etc. , specify it in the infra field: Or use the --infra flag at launch time: When you specify a cluster, SkyPilot will only use that cluster. To enable automatic failover across multiple clusters, omit the cluster specification and let SkyPilot choose based on availability. Using cloud as overflow SkyPilot supports cloud VMs alongside Slurm clusters. If all your Slurm clusters are full, you can automatically overflow to cloud: This gives you infinite overflow capacity when on-prem resources are saturated. Managed jobs for long-running training Use SkyPilot’s managed jobs https://docs.skypilot.ai/en/latest/examples/managed-jobs.html feature for multi-day training runs that need fault tolerance: Managed jobs automatically: - Recover from node failures - Resume from checkpoints This works across all your Slurm clusters, with automatic failover if a cluster goes down. Important notes and limitations important-notes-and-limitations While SkyPilot provides powerful multi-cluster management for Slurm, there are some limitations to be aware of: What’s supported: - SSH-based access to existing Slurm clusters - Multi-node distributed jobs - Interactive development with SSH access - Managed jobs with fault tolerance - Automatic failover across clusters - Container images via Pyxis and enroot see Containers https://docs.skypilot.ai/en/latest/reference/slurm/slurm-getting-started.html containers Current limitations: Autostop : Not supported on Slurm clusters you need to manually stop clusters SkyServe : Model serving https://docs.skypilot.ai/en/latest/serving/sky-serve.html deployments are not available on Slurm Cluster provisioning : SkyPilot cannot create new Slurm clusters only manages existing ones Authentication & permissions: Jobs submit under your configured SSH username and respect your existing Slurm account permissions. If your account has restrictions on certain partitions or GPU types, those same restrictions apply when launching through SkyPilot. If your team runs a shared SkyPilot API server, you can enable submit as user https://docs.skypilot.ai/en/latest/reference/slurm/slurm-getting-started.html submitting-as-authenticated-users so jobs land under each person’s own Unix account rather than a single shared login. Shared filesystems: SkyPilot automatically leverages shared filesystems typically NFS that are mounted on your Slurm clusters. Your home directory and any shared scratch spaces are accessible across all compute nodes without additional configuration. Storage considerations storage-considerations Shared filesystems If your clusters have shared storage e.g., a shared NFS or Lustre filesystem mounted at the same path , you can access data from any cluster: Per-cluster storage If each cluster has its own storage, use SkyPilot’s file mounts to sync data: SkyPilot will mount the S3 bucket on whichever cluster your job lands on, so data is always available. Keeping NFS available on Kubernetes overflow On Slurm, SkyPilot uses your existing NFS mounts automatically. Kubernetes is different - it doesn’t mount your NFS home directory the way Slurm does, so a job that reads from /shared on Slurm won’t find it after it bursts to a Kubernetes cluster. For cloud overflow, reach for object storage instead - see Per-cluster storage above. If your Kubernetes cluster can reach the NFS server, mount it explicitly through pod config : Now /shared points at the same NFS server your Slurm nodes use, so your task reads the same data whether it lands on Slurm or Kubernetes. Put the config block in ~/.sky/config.yaml to apply it to every job. If you’d rather not wire up NFS by hand, SkyPilot Volumes https://docs.skypilot.ai/en/latest/reference/volumes.html volumes-on-kubernetes back a shared ReadWriteMany PVC that multiple jobs mount at the same path. See Shared storage on Kubernetes https://docs.skypilot.ai/en/latest/reference/slurm-migration.html shared-storage-on-kubernetes for the full set of options. Tips for success tips-for-success Run from your laptop. Unlike traditional Slurm workflows that require SSH-ing to a login node, you can run sky launch directly from your laptop. SkyPilot handles the SSH connections and job submission behind the scenes. Start with visibility. Run sky gpus list --infra slurm regularly to understand resource availability across your clusters. This helps you build intuition for capacity patterns. Use Docker instead of module load. Rather than using module load cuda/12.1 in your scripts, consider specifying container images: This ensures consistent environments across all your clusters. Use cluster names consistently. Keep your ~/.slurm/config cluster names descriptive e.g., slurm-cluster-a , slurm-cluster-b so you can easily identify where jobs are running. Standardize paths. If possible, align storage paths across clusters e.g., /home/myusername or /scratch/myusername . This reduces environment differences. Test with small jobs first. Before launching a multi-day training run, test your task definition with a small job to ensure it works across all your clusters. Use managed jobs for critical workloads. The automatic recovery and monitoring features are worth it for expensive training runs. Consider cloud overflow. If you frequently hit capacity limits, configure cloud providers as overflow. You’ll pay cloud rates only when on-prem is full. Wrapping up wrapping-up Managing multiple Slurm clusters doesn’t have to be a manual, error-prone process. With SkyPilot, you get: Unified visibility into resources across all clusters Automatic failover when clusters are full Consistent environments so tasks are portable Simple job management without SSH-ing to different login nodes You write your task definition once, and SkyPilot handles the cluster selection, scheduling, and execution. You focus on your research, not on wrestling with infrastructure. If you’re managing multiple Slurm clusters and tired of the manual overhead, give SkyPilot a try https://skypilot.ai/demo?utm source=blog&utm medium=post&utm campaign=slurm .