Spur: Modern GPU Job Scheduling for HPC and AI Workloads AMD has open-sourced Spur, a modern GPU job scheduler written in Rust under Apache 2.0, to address the complexity of scheduling AI and HPC workloads on GPU clusters. Spur provides Slurm-compatible CLI and APIs, topology-aware GPU scheduling, high availability via Raft consensus, native Kubernetes integration, and vendor-agnostic device management through the Container Device Interface (CDI). The scheduler aims to reduce operational overhead compared to traditional HPC schedulers and Kubernetes-based solutions by offering GPU-first design and built-in features for large-scale model training and inference. Spur: Modern GPU Job Scheduling for HPC and AI Workloads spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads The explosion of AI and large-scale model training has fundamentally changed how organizations think about GPU clusters. Traditional HPC schedulers were built in an era when CPUs dominated, and GPU support was retrofitted as an afterthought. Meanwhile, Kubernetes emerged from the cloud-native world with powerful orchestration primitives—but batch GPU scheduling and Slurm-style job workflows remain friction points unless you assemble a full ecosystem of operators, custom schedulers, and queueing layers on top of the default control plane. We’re introducing Spur and Spur-Cloud —a modern, GPU-first job scheduler written in Rust, developed by AMD and open-sourced under Apache 2.0 as part of AMD’s Open Ecosystem initiative. Spur provides Slurm-compatible CLI and APIs with broader parity actively under development while delivering features designed for today’s AI infrastructure: topology-aware GPU scheduling, embedded high availability via Raft consensus, native Kubernetes integration, and vendor-agnostic device management through the Container Device Interface CDI . In this post, we’ll explore how Spur addresses operational pain points in GPU cluster management, demonstrate its core capabilities with hands-on examples, and show how Spur-Cloud extends the platform into a complete GPU-as-a-Service solution. The GPU Scheduling Challenge the-gpu-scheduling-challenge Managing GPU clusters at scale presents unique challenges. Training a large language model might require 64 MI300X GPUs with XGMI interconnect topology, scheduled across 8 nodes with precise locality constraints. A computer vision pipeline might need fractional GPU allocation—8 independent jobs sharing a single 8-GPU node. Inference workloads demand rapid placement and teardown across heterogeneous hardware MI300X, MI250, A100 . Traditional schedulers struggle with this complexity. Consider a common scenario: you submit a multi-node training job requesting --gres=gpu:mi300x:8 across 4 nodes. Your scheduler must: Identify suitable nodes with the correct GPU type and available capacity Respect topology to minimize inter-GPU communication latency Backfill efficiently by allowing smaller jobs to run without delaying high-priority work Handle failures gracefully when a GPU goes offline mid-job Integrate with containers for reproducible environments Spur was built from the ground up to handle these requirements with first-class GPU support, not as an afterthought. Why a New Scheduler? why-a-new-scheduler The HPC Legacy: Operational Complexity the-hpc-legacy-operational-complexity Traditional HPC schedulers proved the batch job model over decades, but architectures designed for the 2000s carry operational overhead that modern infrastructure shouldn’t require: external database dependencies for accounting, shared state management for HA, C codebases accumulating memory-safety CVEs, and plugin-based GPU support that requires careful flag combinations rather than sensible defaults. The Cloud-Native Path: Assembly Required the-cloud-native-path-assembly-required Kubernetes provides powerful orchestration primitives and a rich ecosystem, but running GPU batch workloads requires assembling multiple components: Install K8s cluster control plane, workers, CNI Deploy GPU device plugin vendor-specific Add custom scheduler Volcano, Yunikorn for advanced placement Install queue manager Kueue for batch job queuing Configure topology awareness via custom schedulers Set up resource quotas, RBAC, monitoring This flexibility is valuable for organizations running diverse workloads, but for GPU-focused clusters, the complexity becomes overhead. Default K8s behavior requires explicit podAffinity and topologySpreadConstraints to guarantee GPU locality—possible, but not automatic. Spur’s Philosophy: GPU-First, Batteries Included spurs-philosophy-gpu-first-batteries-included Spur asks: what if we rebuilt HPC batch scheduling with 2026 technology and GPU workloads as the primary use case? Slurm’s proven patterns GRES syntax, backfill scheduling, partition model plus Rust memory safety Embedded Raft consensus no external database + automatic failover Built-in container runtime no external dependencies + CDI GPU injection Topology-aware GPU scheduling as default behavior, not opt-in flags The result: operational simplicity without sacrificing the batch job model that HPC users rely on. Architecture Overview architecture-overview Spur consists of modular components that can be deployed standalone or within Kubernetes: Components: spurctld — Controller daemon that manages cluster state, runs the scheduler, and handles job submissions gRPC port 6817 spurd — Agent daemon running on each compute node, responsible for resource discovery, job execution, and health reporting port 6818 spurrestd — REST API server exposing Slurm-compatible endpoints for legacy tooling spur-k8s operator — Kubernetes controller that watches SpurJob CRDs and creates Pods with GPU resources spur-cli — Multi-call binary providing sbatch , squeue , scancel , etc., for CLI compatibility State and persistence: Cluster scheduling state—jobs, nodes, partitions, and the queue—is replicated via embedded Raft consensus in spurctld . No external database is required to submit, schedule, and run jobs. Spur-Cloud adds its own PostgreSQL database for billing and session metadata. Core Capabilities core-capabilities 1. GPU-First Scheduling with Topology Awareness gpu-first-scheduling-with-topology-awareness Spur’s backfill scheduler understands GPU topology through CDI device annotations and system topology detection. When you request GPUs, the scheduler considers: Device type matching gpu:mi300x:4 , gpu:a100:2 Interconnect topology XGMI, NVLink, PCIe to minimize communication overhead Fractional allocation for efficient multi-tenancy on high-capacity nodes The default Kubernetes scheduler treats GPUs as opaque extended resources and does not natively understand XGMI/NVLink locality—but production K8s clusters often add topology awareness via schedulers like Volcano , Yunikorn , or Kueue , and Operators with Dynamic Resource Allocation DRA . Spur integrates topology-aware batch scheduling into the controller itself, without requiring a separate scheduler stack for HPC-style jobs. Here’s an example—submitting a distributed training job across 2 nodes with 8 MI300X GPUs each: bash /bin/bash SBATCH --job-name=llama-pretrain SBATCH --nodes=2 SBATCH --gres=gpu:mi300x:8 SBATCH --time=12:00:00 SBATCH --partition=gpu export MASTER ADDR=$ scontrol show hostname $SPUR JOB NODELIST | head -n1 export MASTER PORT=29500 srun torchrun \ --nnodes=$SPUR JOB NUM NODES \ --nproc per node=$SPUR JOB GPUS \ --rdzv id=$SPUR JOB ID \ --rdzv backend=c10d \ --rdzv endpoint=$MASTER ADDR:$MASTER PORT \ train.py --model llama3-70b Submit the job: bash $ spur submit pretrain.sh Submitted batch job 1042 $ spur queue JOBID PARTITION NAME USER ST TIME NODES NODELIST 1042 gpu llama-pretrain alice R 0:23 2 mi300- 1-2 The scheduler automatically: Selected nodes mi300-1 and mi300-2 with available MI300X GPUsSet ROCR VISIBLE DEVICES=0,1,2,3,4,5,6,7 on each allocated nodeProvided environment variables SPUR JOB NODELIST , SPUR JOB GPUS , SPUR JOB NUM NODES for MPI-style coordination 2. Vendor-Agnostic Device Management via CDI vendor-agnostic-device-management-via-cdi Spur uses the Container Device Interface CDI for GPU device management, eliminating hardcoded vendor logic. CDI specifications describe how to inject devices, including AMD ROCm GPUs and other vendor accelerators, into containers. From the CDI implementation https://github.com/ROCm/spur/pull/262 in the spur-devices crate, here’s how Spur discovers GPUs: // Auto-discover AMD GPUs via KFD when CDI cache is empty pub fn discover to cdi vendor: &str - Result