{"slug": "spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads", "title": "Spur: Modern GPU Job Scheduling for HPC and AI Workloads", "summary": "AMD has open-sourced Spur, a modern GPU job scheduler written in Rust under Apache 2.0, to address the complexity of scheduling AI and HPC workloads on GPU clusters. Spur provides Slurm-compatible CLI and APIs, topology-aware GPU scheduling, high availability via Raft consensus, native Kubernetes integration, and vendor-agnostic device management through the Container Device Interface (CDI). The scheduler aims to reduce operational overhead compared to traditional HPC schedulers and Kubernetes-based solutions by offering GPU-first design and built-in features for large-scale model training and inference.", "body_md": "# Spur: Modern GPU Job Scheduling for HPC and AI Workloads[#](#spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads)\n\nThe explosion of AI and large-scale model training has fundamentally changed how organizations think about GPU clusters. Traditional HPC schedulers were built in an era when CPUs dominated, and GPU support was retrofitted as an afterthought. Meanwhile, Kubernetes emerged from the cloud-native world with powerful orchestration primitives—but **batch GPU scheduling** and **Slurm-style job workflows** remain friction points unless you assemble a full ecosystem of operators, custom schedulers, and queueing layers on top of the default control plane.\n\nWe’re introducing **Spur** and **Spur-Cloud**—a modern, GPU-first job scheduler written in Rust, developed by AMD and open-sourced under Apache 2.0 as part of AMD’s Open Ecosystem initiative. Spur provides **Slurm-compatible** CLI and APIs (with broader parity actively under development) while delivering features designed for today’s AI infrastructure: topology-aware GPU scheduling, embedded high availability via Raft consensus, native Kubernetes integration, and vendor-agnostic device management through the Container Device Interface (CDI).\n\nIn this post, we’ll explore how Spur addresses operational pain points in GPU cluster management, demonstrate its core capabilities with hands-on examples, and show how Spur-Cloud extends the platform into a complete GPU-as-a-Service solution.\n\n## The GPU Scheduling Challenge[#](#the-gpu-scheduling-challenge)\n\nManaging GPU clusters at scale presents unique challenges. Training a large language model might require 64 MI300X GPUs with XGMI interconnect topology, scheduled across 8 nodes with precise locality constraints. A computer vision pipeline might need fractional GPU allocation—8 independent jobs sharing a single 8-GPU node. Inference workloads demand rapid placement and teardown across heterogeneous hardware (MI300X, MI250, A100). Traditional schedulers struggle with this complexity.\n\nConsider a common scenario: you submit a multi-node training job requesting `--gres=gpu:mi300x:8`\n\nacross 4 nodes. Your scheduler must:\n\n**Identify suitable nodes** with the correct GPU type and available capacity**Respect topology** to minimize inter-GPU communication latency**Backfill efficiently** by allowing smaller jobs to run without delaying high-priority work**Handle failures** gracefully when a GPU goes offline mid-job**Integrate with containers** for reproducible environments\n\nSpur was built from the ground up to handle these requirements with first-class GPU support, not as an afterthought.\n\n## Why a New Scheduler?[#](#why-a-new-scheduler)\n\n### The HPC Legacy: Operational Complexity[#](#the-hpc-legacy-operational-complexity)\n\nTraditional HPC schedulers proved the batch job model over decades, but architectures designed for the 2000s carry operational overhead that modern infrastructure shouldn’t require: external database dependencies for accounting, shared state management for HA, C codebases accumulating memory-safety CVEs, and plugin-based GPU support that requires careful flag combinations rather than sensible defaults.\n\n### The Cloud-Native Path: Assembly Required[#](#the-cloud-native-path-assembly-required)\n\nKubernetes provides powerful orchestration primitives and a rich ecosystem, but running GPU batch workloads requires assembling multiple components:\n\nInstall K8s cluster (control plane, workers, CNI)\n\nDeploy GPU device plugin (vendor-specific)\n\nAdd custom scheduler (Volcano, Yunikorn) for advanced placement\n\nInstall queue manager (Kueue) for batch job queuing\n\nConfigure topology awareness via custom schedulers\n\nSet up resource quotas, RBAC, monitoring\n\nThis flexibility is valuable for organizations running diverse workloads, but for GPU-focused clusters, the complexity becomes overhead. Default K8s behavior requires explicit `podAffinity`\n\nand `topologySpreadConstraints`\n\nto guarantee GPU locality—possible, but not automatic.\n\n### Spur’s Philosophy: GPU-First, Batteries Included[#](#spurs-philosophy-gpu-first-batteries-included)\n\nSpur asks: what if we rebuilt HPC batch scheduling with 2026 technology and GPU workloads as the primary use case?\n\n**Slurm’s proven patterns**(GRES syntax, backfill scheduling, partition model) plus Rust memory safety** Embedded Raft consensus**(no external database) + automatic failover** Built-in container runtime**(no external dependencies) + CDI GPU injection** Topology-aware GPU scheduling**as default behavior, not opt-in flags\n\nThe result: operational simplicity without sacrificing the batch job model that HPC users rely on.\n\n## Architecture Overview[#](#architecture-overview)\n\nSpur consists of modular components that can be deployed standalone or within Kubernetes:\n\n**Components:**\n\n**spurctld**— Controller daemon that manages cluster state, runs the scheduler, and handles job submissions (gRPC port 6817)** spurd**— Agent daemon running on each compute node, responsible for resource discovery, job execution, and health reporting (port 6818)**spurrestd**— REST API server exposing Slurm-compatible endpoints for legacy tooling** spur-k8s operator**— Kubernetes controller that watches SpurJob CRDs and creates Pods with GPU resources** spur-cli**— Multi-call binary providing`sbatch`\n\n,`squeue`\n\n,`scancel`\n\n, etc., for CLI compatibility\n\n**State and persistence:** Cluster scheduling state—jobs, nodes, partitions, and the queue—is replicated via **embedded Raft consensus** in `spurctld`\n\n. No external database is required to submit, schedule, and run jobs. Spur-Cloud adds its own PostgreSQL database for billing and session metadata.\n\n## Core Capabilities[#](#core-capabilities)\n\n### 1. GPU-First Scheduling with Topology Awareness[#](#gpu-first-scheduling-with-topology-awareness)\n\nSpur’s backfill scheduler understands GPU topology through CDI device annotations and system topology detection. When you request GPUs, the scheduler considers:\n\n**Device type matching**(`gpu:mi300x:4`\n\n,`gpu:a100:2`\n\n)**Interconnect topology**(XGMI, NVLink, PCIe) to minimize communication overhead** Fractional allocation**for efficient multi-tenancy on high-capacity nodes\n\nThe default Kubernetes scheduler treats GPUs as opaque extended resources and does not natively understand XGMI/NVLink locality—but production K8s clusters often add topology awareness via schedulers like **Volcano**, **Yunikorn**, or **Kueue**, and Operators with Dynamic Resource Allocation (DRA). Spur integrates topology-aware batch scheduling into the controller itself, without requiring a separate scheduler stack for HPC-style jobs.\n\nHere’s an example—submitting a distributed training job across 2 nodes with 8 MI300X GPUs each:\n\n``` bash\n#!/bin/bash\n#SBATCH --job-name=llama-pretrain\n#SBATCH --nodes=2\n#SBATCH --gres=gpu:mi300x:8\n#SBATCH --time=12:00:00\n#SBATCH --partition=gpu\n\nexport MASTER_ADDR=$(scontrol show hostname $SPUR_JOB_NODELIST | head -n1)\nexport MASTER_PORT=29500\n\nsrun torchrun \\\n  --nnodes=$SPUR_JOB_NUM_NODES \\\n  --nproc_per_node=$SPUR_JOB_GPUS \\\n  --rdzv_id=$SPUR_JOB_ID \\\n  --rdzv_backend=c10d \\\n  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \\\n  train.py --model llama3-70b\n```\n\nSubmit the job:\n\n``` bash\n$ spur submit pretrain.sh\nSubmitted batch job 1042\n\n$ spur queue\nJOBID  PARTITION  NAME            USER   ST  TIME   NODES  NODELIST\n1042   gpu        llama-pretrain  alice  R   0:23   2      mi300-[1-2]\n```\n\nThe scheduler automatically:\n\nSelected nodes\n\n`mi300-1`\n\nand`mi300-2`\n\nwith available MI300X GPUsSet\n\n`ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7`\n\non each allocated nodeProvided environment variables (\n\n`SPUR_JOB_NODELIST`\n\n,`SPUR_JOB_GPUS`\n\n,`SPUR_JOB_NUM_NODES`\n\n) for MPI-style coordination\n\n### 2. Vendor-Agnostic Device Management via CDI[#](#vendor-agnostic-device-management-via-cdi)\n\nSpur uses the **Container Device Interface (CDI)** for GPU device management, eliminating hardcoded vendor logic. CDI specifications describe how to inject devices, including AMD ROCm GPUs and other vendor accelerators, into containers.\n\nFrom the [CDI implementation](https://github.com/ROCm/spur/pull/262) in the `spur-devices`\n\ncrate, here’s how Spur discovers GPUs:\n\n```\n// Auto-discover AMD GPUs via KFD when CDI cache is empty\npub fn discover_to_cdi(vendor: &str) -> Result<Vec<CdiSpec>> {\n    match vendor {\n        \"amd.com\" => {\n            let gpus = kfd::discover_gpus()?;\n            let specs = gpus.into_iter().map(|gpu| CdiSpec {\n                kind: \"amd.com/gpu\".into(),\n                devices: vec![CdiDevice {\n                    name: format!(\"gpu{}\", gpu.device_id),\n                    container_edits: ContainerEdits {\n                        device_nodes: vec![\n                            DeviceNode { path: \"/dev/kfd\".into(), .. },\n                            DeviceNode { \n                                path: format!(\"/dev/dri/renderD{}\", gpu.render_node),\n                                ..\n                            },\n                        ],\n                        env: vec![format!(\"ROCR_VISIBLE_DEVICES={}\", gpu.device_id)],\n                        ..\n                    },\n                }],\n                ..\n            }).collect();\n            Ok(specs)\n        }\n        _ => bail!(\"Unsupported vendor: {}\", vendor),\n    }\n}\n```\n\nThis approach provides several advantages:\n\n**Vendor neutrality**— Support new GPU types by adding CDI specs, no code changes** Rootless containers**— Device injection works without privileged containers** Interoperability**— CDI is a CNCF standard supported by containerd, CRI-O, Podman\n\nOn startup, `spurd`\n\nreads CDI specs from `/etc/cdi/`\n\nand `/var/run/cdi/`\n\n, falling back to auto-discovery via sysfs for AMD GPUs if no specs exist. Administrators can provide custom CDI specs for exotic hardware without modifying Spur.\n\n### 3. Built-in High Availability with Raft Consensus[#](#built-in-high-availability-with-raft-consensus)\n\nTraditional HPC schedulers rely on external databases (MySQL, MariaDB) for accounting state and often use manual failover for the controller. Spur embeds **Raft consensus** directly in `spurctld`\n\nfor scheduling state, providing:\n\n**Leader election**— Active-passive HA with sub-second failover** State replication**— Job and cluster state replicated to followers via the Raft log** Operational simplicity**— Scheduling durability without running a separate consensus service (contrast with Kubernetes, which uses etcd—a separate Raft-backed store—for control-plane state)\n\nThis is not a novel consensus algorithm; etcd and Spur both rely on Raft. The difference is **deployment model**: Spur co-locates scheduling state with the controller binary, which reduces moving parts for bare-metal and batch-focused clusters.\n\nDeploy a 3-node Raft cluster in Kubernetes:\n\n```\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: spurctld\n  namespace: spur\nspec:\n  serviceName: spurctld\n  replicas: 3\n  selector:\n    matchLabels:\n      app: spurctld\n  template:\n    metadata:\n      labels:\n        app: spurctld\n    spec:\n      containers:\n      - name: spurctld\n        image: ghcr.io/rocm/spur:v0.9.0\n        command: [\"/usr/local/bin/spurctld\"]\n        args:\n        - --config=/etc/spur/spur.conf\n        - --log-level=info\n        volumeMounts:\n        - name: config\n          mountPath: /etc/spur\n        - name: state\n          mountPath: /var/spool/spur\n      volumes:\n      - name: config\n        configMap:\n          name: spur-config\n  volumeClaimTemplates:\n  - metadata:\n      name: state\n    spec:\n      accessModes: [\"ReadWriteOnce\"]\n      resources:\n        requests:\n          storage: 10Gi\n```\n\nThe corresponding config (`spur.conf`\n\n):\n\n```\ncluster_name = \"spur-k8s\"\n\n[controller]\npeers = [\n  \"spurctld-0.spurctld.spur.svc.cluster.local:6821\",\n  \"spurctld-1.spurctld.spur.svc.cluster.local:6821\",\n  \"spurctld-2.spurctld.spur.svc.cluster.local:6821\",\n]\n\n[scheduler]\ninterval_secs = 2\nplugin = \"backfill\"\n\n[[partitions]]\nname = \"default\"\nstate = \"UP\"\ndefault = true\n```\n\nThe Raft node ID is auto-detected from the StatefulSet pod ordinal (`spurctld-0`\n\n→ node 1, `spurctld-1`\n\n→ node 2). On leader failure, the remaining nodes elect a new leader within ~1-3 seconds, and job submissions resume automatically.\n\n### 4. Native Kubernetes Integration[#](#native-kubernetes-integration)\n\nSpur runs jobs as native processes on bare metal or VMs for maximum performance, but it also integrates with Kubernetes for hybrid deployments. The `spur-k8s-operator`\n\nwatches `SpurJob`\n\ncustom resources and submits them to spurctld:\n\n```\napiVersion: spur.amd.com/v1alpha1\nkind: SpurJob\nmetadata:\n  name: benchmark-rocm\nspec:\n  script: |\n    #!/bin/bash\n    #SBATCH --job-name=rocblas-bench\n    #SBATCH --gres=gpu:mi300x:1\n    #SBATCH --time=00:10:00\n    \n    rocblas-bench -f gemm -m 8192 -n 8192 -k 8192 --precision f16_r\n```\n\nApply the CRD:\n\n``` bash\n$ kubectl apply -f benchmark.yaml\nspurjob.spur.amd.com/benchmark-rocm created\n\n$ spur queue\nJOBID  PARTITION  NAME            USER     ST  TIME   NODES  NODELIST\n2001   default    rocblas-bench   k8s-svc  R   0:02   1      gpu-node-3\n```\n\nThe operator creates a Kubernetes Pod with GPU resource requests (`amd.com/gpu: 1`\n\n) on the allocated node. This provides:\n\n**Unified scheduling**— K8s workloads and HPC batch jobs share the same GPU pool** Thin operator**— The K8s operator delegates scheduling logic to spurctld** Topology-aware batch placement**— Spur understands multi-node gang requirements and GPU interconnect locality for distributed training jobs\n\nWhere Spur adds value over the **default** kube-scheduler is multi-node batch coordination: a job requesting 4 nodes with 8 GPUs each needs gang scheduling and topology constraints that vanilla Pod scheduling does not express natively. A single Pod requesting `nvidia.com/gpu: 8`\n\n(or `amd.com/gpu: 8`\n\n) is typically packed onto one node by Kubernetes—but coordinating a 32-GPU job across 4 nodes with locality constraints is where dedicated batch schedulers (Volcano, Kueue) or Spur’s integrated approach matter. Spur targets teams that want Slurm-compatible job semantics on top of (or alongside) their K8s cluster.\n\n### 5. Rootless Containerized Workloads[#](#rootless-containerized-workloads)\n\nSpur includes a **built-in** container runtime for running unprivileged containers without requiring site-wide installation of Singularity, Enroot, or Apptainer—common requirements in Slurm environments. Kubernetes clusters can also run rootless workloads via containerd user namespaces, CRI-O rootless mode, and related runtimes; Spur’s advantage is shipping an integrated, HPC-oriented runtime in the agent itself.\n\nThe container runtime uses:\n\n**Squashfs images**(same format as Enroot) for fast, read-only rootfs** User namespaces**for rootless execution** CDI for GPU injection**via bind mounts of`/dev/kfd`\n\n,`/dev/dri/renderD*`\n\n, ROCm libraries\n\nExample workflow—import a Docker image and run a containerized job:\n\n``` python\n# Import an OCI image into Spur's squashfs format\n$ spur image import docker://rocm/pytorch:rocm6.1_ubuntu22.04_py3.10_pytorch_2.3.0\n\n# Submit a job using the container\n$ cat > container_job.sh << 'EOF'\n#!/bin/bash\n#SBATCH --job-name=pytorch-train\n#SBATCH --gres=gpu:mi300x:4\n#SBATCH --container-image=rocm-pytorch-2.3.0\n#SBATCH --time=02:00:00\n\npython train.py --data /mnt/dataset --epochs 100\nEOF\n\n$ spur submit container_job.sh\nSubmitted batch job 3005\n```\n\nInside the container, GPU devices are automatically available via `ROCR_VISIBLE_DEVICES`\n\n. The container filesystem is read-only by default (overlayfs writeable layer optional), and the job’s home directory is bind-mounted for persistent storage.\n\nThis eliminates the “works on my machine” problem—users package dependencies in container images, and admins don’t need to install CUDA/ROCm toolchains on every node.\n\n### 6. WireGuard Mesh Networking for Multi-Site Clusters[#](#wireguard-mesh-networking-for-multi-site-clusters)\n\nCloud bursting and multi-datacenter GPU clusters require secure, low-latency communication across network boundaries. Spur includes **integrated WireGuard mesh networking** to create a unified overlay network for job communication—configured in the scheduler stack rather than as a separate CNI or VPN project.\n\nKubernetes clusters can achieve similar encryption and multi-cluster connectivity via CNI plugins (**Cilium** and **Calico** support WireGuard) and tools like **Submariner** for multi-cluster networking. Spur’s mesh is aimed at batch jobs spanning on-prem and cloud nodes: agents auto-register WireGuard endpoints and multi-node jobs communicate over the encrypted overlay without per-job VPN setup.\n\nConfigure the mesh in `spur.conf`\n\n:\n\n```\n[network]\nwg_enabled = true\nwg_interface = \"spur0\"\nwg_cidr = \"10.44.0.0/16\"\n```\n\nAgents auto-detect their WireGuard IP and self-report it during registration. Multi-node jobs communicate over the mesh without exposing public IPs—when a job runs across on-prem and cloud nodes, MPI or NCCL communication happens over the encrypted WireGuard tunnel, transparent to the application.\n\n### 7. Slurm Compatibility: Migration Path[#](#slurm-compatibility-migration-path)\n\nOrganizations with existing Slurm workflows can migrate incrementally. Spur provides a growing compatibility surface:\n\n**CLI compatibility**— Core commands (`sbatch`\n\n,`squeue`\n\n,`scancel`\n\n,`sinfo`\n\n,`srun`\n\n) and common`#SBATCH`\n\ndirectives work today; advanced`scontrol`\n\n/`sacctmgr`\n\nfeatures and edge-case script compatibility are still expanding**REST API compatibility**— Slurm REST clients (Jupyter, portals) can connect with minimal changes** C FFI shim**—`libspur_compat.so`\n\nprovides ABI compatibility for Slurm-linked binaries\n\nCreate symlinks for familiar command names:\n\n```\ncd /usr/local/bin\nfor cmd in sbatch srun squeue scancel sinfo sacct scontrol; do\n    ln -sf spur $cmd\ndone\n```\n\nMany legacy scripts work without modification:\n\n``` bash\n$ sbatch --ntasks=16 --cpus-per-task=4 mpi_app.sh\nSubmitted batch job 4201\n\n$ squeue -u alice\nJOBID  PARTITION  NAME     USER   ST  TIME   NODES  NODELIST\n4201   default    mpi_app  alice  R   0:05   2      node-[7-8]\n```\n\nUnder the hood, `sbatch`\n\nis the `spur-cli`\n\nbinary invoked via a symlink. It parses Slurm batch script directives (`#SBATCH`\n\n), converts them to gRPC `SubmitJobRequest`\n\n, and talks to spurctld on port 6817. Sites with complex Slurm plugin ecosystems should plan a phased migration—see the Roadmap for ongoing parity work.\n\n## Comparison with Slurm[#](#comparison-with-slurm)\n\nSlurm has served the HPC community for two decades and remains the de facto standard. It offers unmatched maturity, a vast ecosystem of plugins, and deep integration with HPC site management tools. Spur doesn’t aim to replace Slurm wholesale—instead, it targets operational pain points where a modern architecture provides clear advantages:\n\nCapability |\nSlurm |\nSpur |\n|---|---|---|\n|\nGres plugin (type-aware, no topology by default) |\nFirst-class GPU support with CDI, topology-aware via XGMI/NVLink detection |\n|\n|\nEmbedded Raft in |\n|\n|\nRaft auto-failover in ~1-3 seconds |\n|\nRequires Singularity/Enroot/Apptainer installed |\nBuilt-in rootless runtime, squashfs images |\n|\nVia slurm-operator or federation (complex) |\nNative K8s operator with SpurJob CRD |\n|\nIP-based, external VPN for multi-site |\nIntegrated WireGuard mesh |\n|\nC (legacy), 500K+ LoC |\nRust (memory-safe), 50K LoC |\n\nSpur’s Slurm compatibility layer enables brownfield migration—start by running Spur alongside existing infrastructure, migrate workloads incrementally, and cut over when ready.\n\n## Spur vs. a Full Kubernetes GPU Stack[#](#spur-vs-a-full-kubernetes-gpu-stack)\n\nThis post focuses on Spur’s strengths; a production Kubernetes GPU cluster is rarely “vanilla K8s” alone. The table below compares Spur to the **ecosystem** teams typically deploy—not just the default scheduler:\n\nCapability |\nTypical K8s GPU stack |\nSpur |\n|---|---|---|\n|\nVolcano, Yunikorn, Kueue; Operators with DRA |\nIntegrated in spurctld backfill scheduler |\n|\nKueue, Volcano queues |\nNative partitions, priorities, backfill |\n|\nHardware partitioning, device-plugin time-slicing |\nScheduler-level |\n|\netcd (Raft) + controller leader election |\nEmbedded Raft in spurctld |\n|\nNamespaces, ResourceQuotas, NetworkPolicies |\nPartitions, user quotas; Spur-Cloud adds web UI and billing |\n|\nNot native (requires slurm-on-k8s bridges) |\nFirst-class |\n\n**Complementary, not competing:** AMD’s Open Ecosystem includes Operators (GPU, Network, and more) that bring device-aware scheduling, DRA, and topology awareness to Kubernetes—deployed at scale across AI and HPC infrastructure. Spur addresses a different need—**Slurm-compatible batch scheduling** with an opinionated, integrated stack for HPC and AI training clusters. Many organizations will run both: Kubernetes with Operators for services and cloud-native workloads, Spur for batch GPU jobs (bare metal, VMs, or via the SpurJob CRD).\n\nChoose Spur when you want a unified batch scheduler with Slurm migration paths and integrated HA, containers, and mesh networking. Choose a full K8s GPU stack when your workloads are already Pod-native and you have invested in Volcano/Kueue, GPU operators, and cloud-native tooling.\n\n## Spur-Cloud: GPU-as-a-Service[#](#spur-cloud-gpu-as-a-service)\n\nWhile Spur handles job scheduling, **Spur-Cloud** extends it into a complete GPU-as-a-Service platform with web UI, billing, and multi-tenancy. It provides:\n\n### Web-Based Session Management[#](#web-based-session-management)\n\nUsers launch GPU sessions from a React dashboard without touching YAML or CLI:\n\nSelect GPU type (MI300X, MI250, A100) and count\n\nChoose container image (prebuilt ROCm/PyTorch, CUDA/TensorFlow, or custom)\n\nEnable SSH access (optional, with key injection)\n\nClick “Launch”\n\nSpur-Cloud submits a job to spurctld via gRPC, the scheduler places it, and the K8s operator creates a Pod. The web UI provides:\n\n**Integrated terminal**(xterm.js over WebSocket via`kubectl exec`\n\n)**SSH NodePort** for IDE remote development (VSCode, PyCharm)**Session management**(view status, terminate)\n\n### Billing and Usage Tracking[#](#billing-and-usage-tracking)\n\nSpur-Cloud tracks GPU usage per session and per user, storing records in PostgreSQL:\n\n``` bash\n$ curl -H \"Authorization: Bearer $JWT\" \\\n  https://gpu.example.com/api/billing/summary\n{\n  \"total_gpu_hours\": 482.3,\n  \"by_gpu_type\": [\n    { \"gpu_type\": \"mi300x\", \"hours\": 320.5 },\n    { \"gpu_type\": \"mi250\",  \"hours\": 161.8 }\n  ]\n}\n```\n\nUsage events are emitted when sessions start, stop, or hit time limits. Admins can export usage records for integration with accounting systems or apply their own pricing models for chargebacks.\n\n### Authentication and Multi-Tenancy[#](#authentication-and-multi-tenancy)\n\nSpur-Cloud supports three authentication providers:\n\n**Local**— Email/password with Argon2 hashing** GitHub OAuth**— Login via GitHub, automatic user provisioning** Okta OIDC**— Enterprise SSO with group-based admin role mapping\n\nConfiguration example:\n\n```\n[auth]\njwt_secret = \"generate-with-openssl-rand-hex-32\"\n\n[auth.github]\nenabled = true\nclient_id = \"Iv1.abc123\"\nclient_secret = \"secret\"\n\n[auth.okta]\nenabled = true\nissuer = \"https://mycompany.okta.com/oauth2/default\"\nclient_id = \"0oa123\"\nclient_secret = \"secret\"\nadmin_groups = [\"gpu-admins\"]\n```\n\nAll methods produce a platform JWT used for API authorization. Sessions are isolated per-user—users see only their own jobs and usage data (admins see all). Quotas are enforced as concurrent GPU limits (e.g., max 8 GPUs at once) to prevent resource monopolization.\n\n### Fractional GPU Allocation[#](#fractional-gpu-allocation)\n\nSpur-Cloud leverages Spur’s scheduler-level fractional GPU allocation for cost efficiency—assigning whole GPUs to jobs via `gres`\n\nwithout requiring hardware partitioning configurations. Kubernetes can also fractionalize GPUs via hardware partitioning and device plugins; Spur’s model is simpler for multi-tenant batch and session workloads on bare-metal-style clusters.\n\nA single MI300X node with 8 GPUs can serve multiple concurrent sessions, each isolated via `ROCR_VISIBLE_DEVICES`\n\n. For example:\n\nSession |\nRequest |\nAssigned GPUs |\n|---|---|---|\nA |\n|\nGPU 0 |\nB |\n|\nGPU 1 |\nC |\n|\nGPU 2-3 |\nD |\n|\nGPU 4 |\n\nThe remaining GPUs (5-7) stay available for additional sessions. This maximizes utilization—no need to over-provision nodes or leave GPUs idle.\n\n## Getting Started[#](#getting-started)\n\n### Quick Start (Single Node)[#](#quick-start-single-node)\n\nTry Spur locally in under 5 minutes. Step-by-step instructions are in the [Quickstart guide](https://github.com/ROCm/spur/blob/main/docs/getting-started.rst):\n\n```\n# Install binaries\ncurl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh | bash\nexport PATH=\"$HOME/.local/bin:$PATH\"\n\n# Start controller\nmkdir -p /tmp/spur-state\nspurctld -D --state-dir /tmp/spur-state\n\n# Start agent (new terminal)\nspurd -D --controller http://localhost:6817\n\n# Submit a job\ncat > hello.sh << 'EOF'\n#!/bin/bash\n#SBATCH --job-name=hello\necho \"Hello from $(hostname) with $SPUR_CPUS_ON_NODE CPUs\"\nEOF\n\nspur submit hello.sh\n\n# Check queue\nspur queue\n```\n\n### Production Deployment (Multi-Node)[#](#production-deployment-multi-node)\n\nFor a production GPU cluster, see the [deployment guide](https://github.com/ROCm/spur/tree/main/docs/deployment) and the [native-host reference](https://github.com/ROCm/spur/blob/main/docs/deployment/native-host.rst). A multi-node startup playbook and Kubernetes Helm chart are landing in the repository to simplify first production deployments—check the deployment docs for the latest scripts and charts.\n\nKey steps:\n\n**Install binaries** on all nodes (controller + compute nodes)**Configure**`/etc/spur/spur.conf`\n\nwith partition and node definitions:\n\n```\ncluster_name = \"ai-cluster\"\n\n[controller]\nlisten_addr = \"[::]:6817\"\nstate_dir = \"/var/spool/spur\"\n\n[scheduler]\nplugin = \"backfill\"\ninterval_secs = 1\n\n[[partitions]]\nname = \"gpu\"\ndefault = true\nstate = \"UP\"\nmax_time = \"7-00:00:00\"\n\n[[nodes]]\nnames = \"gpu-[01-08]\"\ncpus = 192\nmemory_mb = 1536000\ngres = [\"gpu:mi300x:8\"]\n```\n\n**Start services** via systemd:\n\n```\nsudo systemctl enable --now spurctld\nsudo systemctl enable --now spurd\n```\n\n**Verify** cluster health:\n\n``` bash\n$ spur nodes\nNODELIST    STATE   CPUS  MEMORY  GRES\ngpu-01      idle    192   1536000 gpu:mi300x:8\ngpu-02      idle    192   1536000 gpu:mi300x:8\n...\n```\n\nFor Kubernetes, see the [Kubernetes deployment guide](https://github.com/ROCm/spur/blob/main/docs/deployment/kubernetes.rst).\n\n### Spur-Cloud Deployment[#](#spur-cloud-deployment)\n\nDeploy Spur-Cloud on top of an existing Spur cluster:\n\n```\n# Clone repo\ngit clone https://github.com/ROCm/spur-cloud\ncd spur-cloud\n\n# Build backend and frontend\ncargo build --release\ncd frontend && npm install && npm run build && cd ..\n\n# Configure\ncp spur-cloud.toml.example spur-cloud.toml\n# Edit: set database URL, Spur controller address, auth providers\n\n# Start API server (migrations run automatically on startup)\n./target/release/spur-cloud-api --config spur-cloud.toml\n\n# Serve frontend (or use nginx)\ncd frontend && npm run preview\n```\n\nAccess the UI at `http://localhost:5173`\n\n, create an account, and launch your first GPU session.\n\nFor Kubernetes deployment, manifests are in `deploy/k8s/`\n\n. The platform integrates with your existing K8s cluster—no separate infrastructure needed.\n\n## Example Workflows[#](#example-workflows)\n\nThe scenarios below illustrate how Spur fits common GPU cluster patterns—they are representative workflows, not references to specific production deployments.\n\n### Example: Large-Scale LLM Training[#](#example-large-scale-llm-training)\n\nConsider a team training a large model across 128 MI300X GPUs (16 nodes, 8 GPUs each). Key requirements:\n\n**Topology-aware allocation**— Scheduler places nodes with XGMI-connected GPUs** Reproducible environments**— PyTorch, ROCm, and custom tokenizers in a container** Operational visibility**— Queue management and node health via familiar Slurm-style commands\n\nSpur configuration:\n\n```\n[[nodes]]\nnames = \"mi300x-[01-16]\"\ncpus = 192\nmemory_mb = 1536000\ngres = [\"gpu:mi300x:8\"]\nfeatures = [\"xgmi\", \"nvme\"]\n```\n\nSubmit the job:\n\n``` bash\n#!/bin/bash\n#SBATCH --job-name=llama3-405b\n#SBATCH --nodes=16\n#SBATCH --gres=gpu:mi300x:8\n#SBATCH --constraint=xgmi\n#SBATCH --time=72:00:00\n#SBATCH --container-image=rocm-pytorch-2.3.0\n\nsrun python train.py \\\n  --model llama3-405b \\\n  --data /mnt/nvme/pile \\\n  --checkpoint-dir /mnt/shared/checkpoints\n```\n\nThe `--constraint=xgmi`\n\nensures only nodes with XGMI interconnect are selected. Node failures during a long-running job are surfaced to operators via agent health reporting; automatic checkpoint resume and gang re-queue behaviors are on the Roadmap.\n\n### Example: Multi-Tenant Inference Platform[#](#example-multi-tenant-inference-platform)\n\nA startup offers a fine-tuning API where customers upload datasets and train custom models. Each customer job needs isolated GPU resources without over-provisioning.\n\nSpur-Cloud provides:\n\n**Per-user quotas**(e.g., max 4 concurrent GPUs per user)** Session isolation**via K8s namespaces and Spur’s user-based scheduling** Usage tracking**to monitor GPU consumption by customer\n\nCustomers interact via web UI (no CLI knowledge required), and sessions auto-terminate after a configurable timeout to prevent runaway usage.\n\n### Example: Hybrid Cloud Bursting[#](#example-hybrid-cloud-bursting)\n\nAn autonomous vehicle company runs daily simulation workloads on-prem but bursts to AWS for peak demand. Spur’s WireGuard mesh connects on-prem nodes (`10.44.0.0/24`\n\n) and cloud instances (`10.44.1.0/24`\n\n) into a unified cluster.\n\nWhen local capacity is exhausted, Spur schedules jobs to cloud nodes:\n\n``` bash\n$ spur nodes\nNODELIST        STATE  LOCATION   GRES\non-prem-01      idle   datacenter gpu:mi300x:8\non-prem-02      idle   datacenter gpu:mi300x:8\naws-gpu-001     idle   us-west-2  gpu:a100:8 (cloud)\naws-gpu-002     idle   us-west-2  gpu:a100:8 (cloud)\n```\n\nJobs submitted without `--constraint`\n\ncan land anywhere. Jobs requiring low-latency data access use `--constraint=datacenter`\n\nto stay on-prem.\n\n## Roadmap and Community[#](#roadmap-and-community)\n\nSpur is under active development by AMD and the open-source community. Upcoming features include:\n\n**Broader Slurm parity**— Deeper CLI and REST API coverage, migration tooling, and seamless replacement for more site-specific Slurm workflows** GPU Portioning and SR-IOV support**— Fractional GPU partitioning at the hardware level** Advanced preemption**— Suspend low-priority jobs to free GPUs for urgent work** Multi-cluster federation**— Submit once, route to the cluster with availability** Observability enhancements**— Prometheus metrics, OpenTelemetry tracing, scheduler dashboards\n\nAs part of AMD’s Open Ecosystem initiative, Spur is fully open-source (Apache 2.0) and welcomes contributions:\n\n**GitHub**:[ROCm/spur](https://github.com/ROCm/spur)** Discussions**:[ROCm/spur#discussions](https://github.com/ROCm/spur/discussions)** Issues**:[ROCm/spur#issues](https://github.com/ROCm/spur/issues)\n\nWhether you’re managing a 10-node lab cluster or a 1000-GPU datacenter, we’d love to hear your feedback and see how Spur fits your workflow. AMD backs this project as part of our broader open ecosystem—alongside ROCm, Operators deployed at scale, and cloud-native tooling.\n\n## Summary[#](#summary)\n\nIn this blog, you explored how Spur reimagines GPU job scheduling for the AI era. You saw why traditional HPC schedulers and full Kubernetes GPU stacks fall short for GPU-first batch workloads, and how Spur closes that gap with topology-aware GPU scheduling, vendor-agnostic device management through CDI, embedded high availability via Raft consensus, a built-in rootless container runtime, and WireGuard mesh networking — all behind a Slurm-compatible CLI and API for incremental migration. You also walked through hands-on examples spanning single-node quickstart, multi-node LLM training, Kubernetes integration via the SpurJob CRD, and how Spur-Cloud layers on a web UI, billing, and multi-tenancy to deliver GPU-as-a-Service.\n\nThis is only the beginning. Our team is actively expanding Spur toward broader Slurm parity, hardware-level GPU partitioning (SR-IOV), advanced preemption, multi-cluster federation, and richer observability — which we’ll cover in future series posts. Whether you’re managing a 10-node lab cluster or a 1000-GPU datacenter, we’d love to hear your feedback and see how Spur fits your workflow. We invite you to try Spur on your own cluster, join the discussion, and help shape where the project goes next.\n\nGPU scheduling doesn’t have to be painful. As an AMD-backed, fully open-source project built on AMD’s Open Ecosystem principles, Spur is designed to work across your entire GPU infrastructure stack. Get started, explore the docs, and join the community on [GitHub](https://github.com/ROCm/spur).\n\n## Additional Resources[#](#additional-resources)\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, Instinct, EPYC, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/spur-gpu-job/README.html", "published_at": "2026-07-22 00:00:00+00:00", "updated_at": "2026-07-22 16:57:32.428448+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["AMD", "Spur", "Spur-Cloud", "Kubernetes", "Slurm", "Container Device Interface (CDI)", "Raft"], "alternates": {"html": "https://wpnews.pro/news/spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads", "markdown": "https://wpnews.pro/news/spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads.md", "text": "https://wpnews.pro/news/spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads.txt", "jsonld": "https://wpnews.pro/news/spur-modern-gpu-job-scheduling-for-hpc-and-ai-workloads.jsonld"}}