GPUlayer: One Layer Between Your Code and Every GPU You Can Reach

Developer Francesco Antonello released GPUlayer, an Apache 2.0-licensed tool published on PyPI that adds a scheduling and control layer between code and GPUs spread across laptops, servers, Kubernetes pools and cloud VMs, letting a single `gpulayer submit` command replace manual ssh, rsync and tmux workflows. GPUlayer routes jobs through one queue and capability matcher across four workload types, supports Python functions via a `@gl.remote` decorator with `.map()` fan-out, and is validated on Apple Silicon, NVIDIA CUDA (Tesla T4), automatic Google Cloud GPU provisioning and fleets of up to 30 machines across two cloud regions. Antonello said he built it partly so coding agents such as Claude Code and MCP-enabled editors could inspect and submit work to GPUs without being handed raw SSH keys.

GPUlayer: One Layer Between Your Code and Every GPU You Can Reach Most GPU software assumes the GPU is attached to the machine where you type python . PyTorch, Lightning, Hugging Face Accelerate, JAX, MLX, and CuPy all do a great job deciding how a computation uses the hardware it is given. None of them decide which machine should run the job, ship your project and inputs there, check whether the driver and libraries actually work before starting, or bring logs and checkpoints back when it finishes. I built GPUlayer https://www.gpulayer.dev/ GitHub https://github.com/antonellof/gpulayer to solve two closely related problems: 1. My GPUs are spread across different machines : an Apple Silicon laptop, Linux servers, and rented cloud VMs. Every time I started a new workload, I ended up repeating the same manual ritual— ssh , rsync , hoping the virtualenv and CUDA build still matched the driver, starting tmux , and copying checkpoints back. 2. I needed a way to give my coding agents control over all of my GPUs : I do most of my day-to-day work inside coding agents—Claude Code, Pi agents, and MCP-enabled editors—running on my laptop. Handing an agent raw SSH keys and teaching it which box is free, which one has CUDA vs. Metal mps / mlx , and how to rsync artifacts back is fragile and dangerous. I wanted one structured control layer so an agent on my MacBook can inspect every GPU I can reach, submit a training run or benchmark to the right machine, stream the logs, and pull back the results. Before GPUlayer, running a job on another machine looked like this: ssh gpu-box rsync -av my project/ gpu-box:~/my project/ hope the virtualenv and CUDA build still match the driver tmux new -s train cd ~/my project && python train.py check back later, rsync checkpoints home Multiply that by several machines with different drivers, a Mac with Metal mps and mlx instead of CUDA, a 96-run learning-rate sweep, or a multi-node job where torchrun needs matching ranks, addresses, and ports on every host. With GPUlayer, that becomes: pip install gpulayer gpulayer submit --accelerator cuda --source my project -- python train.py GPUlayer is published on PyPI under Apache 2.0, and validated on Apple Silicon, NVIDIA CUDA Tesla T4 , automatic Google Cloud GPU provisioning, and fleets of up to 30 machines across two cloud regions. This post walks through what it does, how the scheduler and distributed runtime work, how it performs against PyTorch Lightning and torchrun , and where its boundaries are today. What GPUlayer does GPUlayer sits underneath your frameworks and above your machines: your laptop, servers that joined with gpulayer join , Kubernetes worker pools, or cloud VMs provisioned on demand. Everything goes through the same queue, capability matcher, and wire protocol across four workload types: 1. Any command or binary Scripts, shell pipelines, C++ or Rust binaries, Blender renders, or ffmpeg transcodes run as jobs without wrapper code. You point GPUlayer at a source directory, declare inputs and outputs, and state the hardware requirements: Run a script on any worker with a CUDA GPU gpulayer submit --accelerator cuda --source ./app -- ./render.sh Stage an input file and run a shell job gpulayer submit --source examples/shell job \ --input examples/shell job/sales.csv:sales.csv -- bash report.sh 2. Python functions with any library Decorate a function with @gl.remote , specify the accelerator and framework versions it needs, and call it directly or fan it out across every free worker with .map : python import gpulayer as gl @gl.remote accelerator="mps", frameworks={"mlx": " =0.10"} def matmul benchmark n: int - dict: import mlx.core as mx ... Scatters across available workers in parallel results = matmul benchmark.map 1024, 2048, 4096 Arguments and return values are plain JSON, and code travels as a clean source bundle rather than brittle pickled bytecode. 3. PyTorch training and distributed launch GPUlayer includes a focused PyTorch integration gpulayer.pytorch with a Trainer , Runtime , and launch . The exact same script runs in-process on your laptop, as multi-process DDP on one machine, or across multiple nodes: Single process python examples/trainer regression/train.py 2-process DDP on CPU python examples/trainer regression/train.py --accelerator cpu --devices 2 2 nodes x 2 processes across remote workers python examples/trainer regression/train.py --accelerator cpu --nodes 2 --devices 2 --target remote Or directly in Python: python import gpulayer.pytorch as glt trainer = glt.Trainer accelerator="cuda", devices=4, num nodes=2, max epochs=10, target="remote", trainer.fit model, train loader, val loader The Trainer handles device placement, mixed precision fp16 / bf16 , torch.compile , gradient accumulation with DDP no sync , sanity validation steps, callbacks EarlyStopping , ModelCheckpoint save top k=N , LearningRateMonitor , test , predict , CUDA side-stream batch prefetching, two-level gradient sync for multi-node CPU/gloo runs, and experimental FSDP2 strategy="fsdp" . If you already use PyTorch Lightning or Accelerate, glt.launch places the job and starts the processes with the standard torchrun environment variables they expect. 4. Long-running services For inference endpoints or persistent workers, gpulayer submit --service or gl.service ... runs replicas on distinct machines, assigns each replica a free $PORT , waits for it to accept connections, reports host:port endpoints via Job.endpoints , restarts crashed replicas with exponential backoff, and re-places replicas if a worker machine disappears. Placement that explains itself When a job fails on a cluster, the worst outcome is waiting three minutes for container setup and import torch only to hit RuntimeError: No CUDA GPUs are available or a missing library. In GPUlayer, workers probe their hardware and environment on startup. For GPU libraries torch.cuda , jax.cuda , cupy.cuda , torch.mps , mlx , GPUlayer does not merely check whether the Python package is importable—it runs a lightweight GPU probe in a separate interpreter to verify that the library can actually execute on the device. A CPU-only jax or broken CUDA installation will never advertise itself as a working GPU worker. When a job arrives, the scheduler evaluates every worker against the job’s requirements. If no worker qualifies, the job is rejected immediately with per-worker reasons: scheduling: no eligible worker gpu-box-1 needs 2 usable cuda device s , worker has 1 mac-studio framework 'jax' not installed laptop torch on this worker cannot use CUDA gpu-box-2 eligible Topology-aware groups and NVLink/PCIe GPU selection Placement goes beyond counting free GPUs: - Rack, zone, and region locality --group-locality : Workers report their cloud region , zone read automatically from the GCE metadata server on Google Cloud or passed via flags/env , and an optional gpulayer.dev/network-group label for rack or InfiniBand islands. When gang-scheduling a multi-node job, the default prefer policy keeps all nodes in the same network group if possible, then the same zone, then the same region, before falling back to cross-region placement. - Best-connected GPUs on a host : On multi-GPU machines, workers parse nvidia-smi topo -m into link speeds and NUMA affinities. When a task asks for multiple GPUs on an 8-GPU server, GPUlayer picks the free subset whose slowest pairwise interconnect is fastest preferring an NVLink island or GPUs behind a single PCIe switch over arbitrary lowest indices and exports CUDA DEVICE ORDER=PCI BUS ID so mixed-GPU machines never mismatch device indices. From one laptop to cloud GPUs, Kubernetes, and AI agents Adding machines in one command For local work, the control plane runs embedded inside your Python process or CLI call with zero setup. When you want to pool multiple machines, start the server on one host: gpulayer up gpulayer up generates a TLS certificate with a pinned fingerprint, creates named role-based tokens admin , submitter , join , metrics , and prints the exact gpulayer join command to paste on any other machine: gpulayer join https://192.168.1.50:8443 --token <join-token --tls-fingerprint SHA256:... Because workers poll the control plane over outbound HTTPS connections, a worker on a home workstation or lab server behind NAT can join without port forwarding or VPN tunnels. Automatic Google Cloud provisioning When your standing machines are busy—or when you develop on a MacBook and only want to pay for an NVIDIA GPU while a job runs—GPUlayer can provision cloud VMs per job gpulayer gcp . Configure your allowed machine offers in providers.json , start gpulayer up --providers providers.json , and submit with --provider gcp --accelerator cuda . When no existing worker fits, GPUlayer creates a Compute Engine VM such as a Spot Tesla T4 , boots it with a single-use join token, runs the job as soon as the worker reports torch.cuda , downloads the checkpoints and logs, and deletes the VM when the job finishes. In our validation runs, a fresh Spot T4 VM went from API creation to running a CUDA Trainer job in 78 seconds and tore itself down immediately on completion. Web dashboard, Prometheus, and Kubernetes The server includes a built-in web dashboard gpulayer dashboard showing queued and running jobs, per-rank live logs, placement diagnostics, worker hardware inventory, and live GPU/CPU utilization, alongside a /metrics Prometheus endpoint and prebuilt Grafana dashboard. For teams running on Kubernetes, GPUlayer ships container images on GitHub Container Registry ghcr.io/antonellof/gpulayer and a Helm chart deploy/kubernetes/helm/gpulayer that deploys the control plane as a StatefulSet alongside CPU and NVIDIA GPU worker pools. Giving coding agents control over all your GPUs gpulayer mcp and SKILL.md One of my primary reasons for building GPUlayer was letting my coding agents— Claude Code , Pi agents , Cursor, and Codex—use every GPU I own or rent without giving them raw SSH access or make-shift bash loops. When an agent is writing a kernel, tuning hyperparameters, or debugging a training script on my laptop, it needs to run that code on an NVIDIA GPU or across a multi-node fleet, wait for the outcome, read the logs, and inspect the output files. GPUlayer gives agents three ways to do that through the same capability scheduler and scoped token permissions submitter or --read-only , never needing an admin token or SSH keys : 1. MCP server gpulayer mcp : A built-in Model Context Protocol server over stdio exposing structured tools: gpulayer resources , gpulayer submit , gpulayer wait , gpulayer job status , gpulayer job logs , gpulayer list jobs , gpulayer cancel , and gpulayer download outputs . 2. Agent Skill integrations/skills/gpulayer/SKILL.md : For CLI-native harnesses like Pi and Claude Code , the repository ships a ready-to-use SKILL.md that teaches the agent how to inspect available accelerators gpulayer resources list --json , submit jobs with --accelerator cuda or --accelerator mps , stream logs gpulayer job logs <id -f , interpret per-worker rejection reasons instead of blindly retrying, and download outputs. 3. HTTP / Python SDK : Custom agent loops can hit the REST API or gpulayer.Client directly. Wiring it into Claude Code or Pi takes one command: Add the MCP server to Claude Code against your GPUlayer control plane claude mcp add gpulayer \ -e GPULAYER URL=https://gpu-server:8470 \ -e GPULAYER TOKEN=glc ... \ -e GPULAYER FINGERPRINT=sha256:... \ -- gpulayer mcp Or install the CLI skill for Pi / Claude Code cp -r integrations/skills/gpulayer ~/.claude/skills/ When you tell your agent “Train train.py on an NVIDIA GPU for 3 epochs and tell me the validation loss” , it calls gpulayer resources , sees which worker has cuda and torch , calls gpulayer submit with source dir: "." and accelerator: "cuda" , waits with gpulayer wait , and reads the metrics back—while every agent-submitted job is tracked by token identity submitted by on the dashboard. Performance and benchmarks Every benchmark below comes from reproducible scripts in benchmarks/ https://github.com/antonellof/gpulayer/tree/main/benchmarks , with raw logs checked into benchmarks/results/ https://github.com/antonellof/gpulayer/tree/main/benchmarks/results . 1. Single-GPU training speed vs. PyTorch Lightning We trained the exact same models, datasets, and SGD optimizer with one logged loss per step using GPUlayer’s Trainer and PyTorch Lightning’s Trainer with Lightning’s logger, progress bar, model summary, and checkpointing turned off to make the loop comparison fair on a Google Cloud n1-standard-8 VM with an NVIDIA Tesla T4 driver 580, Python 3.12, PyTorch 2.15.0, CUDA 13.2, Lightning 2.6.6 . Each configuration ran 100 and 600 steps three times in fresh processes, taking the difference of median wall times divided by 500 steps: | Workload Tesla T4 / CPU | GPUlayer ms/step | Lightning ms/step | Speedup | GPUlayer startup | Lightning startup | |---|---|---|---|---|---| | MLP, fp32, batch 256 | 4.06 ms | 5.64 ms | 1.39x faster | 4.77 s | 5.31 s | | MLP, fp16 mixed precision | 5.00 ms | 6.25 ms | 1.25x faster | 4.82 s | 5.55 s | | CNN, fp16 mixed, batch 128 | 10.82 ms | 12.33 ms | 1.14x faster | 5.65 s | 5.74 s | | DDP, 2 CPU processes | 3.20 ms | 9.56 ms | 2.99x faster | 10.32 s | 9.35 s | Why the numbers differ: - Small models MLP : When a forward/backward pass takes 4 ms, framework bookkeeping per step is visible. GPUlayer’s loop avoids synchronous device-to-host stalls when recording step losses. - Compute-bound CNN : On larger convolutional steps, GPUlayer enables two optimizations by default on CUDA: cuDNN autotuning cudnn.benchmark = True unless deterministic=True and asynchronous side-stream batch prefetching Runtime.batches , which pins the next batch and copies it to the GPU on a secondary CUDA stream while the current step computes. Ablating those two flags on the same T4 benchmarks/results/2026-10-31-gcp-t4/ isolated their contribution: 12.67 ms/step with both off, 12.10 ms with prefetch only ~5% , 11.53 ms with cuDNN autotuning only ~9% , and 10.93 ms with both enabled. - CPU DDP : GPUlayer pins each worker process to a dedicated thread count by default, whereas Lightning leaves PyTorch’s default multi-threading untouched, causing two ranks to contend over the 8 vCPUs. - Startup overhead : On a single GPU, GPUlayer starts training 0.1 to 0.7 seconds sooner than Lightning. 2. Multi-node strong scaling: 2 to 8 machines 32 processes Next, we ran a multi-node CNN training job 32,768 images, global batch size 256 split across all ranks, 2 epochs across 2, 4, and 8 GCP n1-standard-4 VMs 4 processes per machine, up to 32 processes total over gloo . Both trainers ran on the exact same VMs in interleaved order. With GPUlayer, a single Trainer num nodes=N, devices=4 call scheduled and launched all machines; with Lightning, torchrun was started on every node with matching rendezvous addresses. | Machines | Total Processes | GPUlayer Wall Time | Lightning torchrun | GPUlayer Lead | Final Loss GL / PL | |---|---|---|---|---|---| | 2 | 8 | 118.9 s | 120.8 s | 1.6% faster | 2.129 / 2.125 | | 4 | 16 | 73.3 s | 77.4 s | 5.3% faster | 2.143 / 2.137 | | 8 | 32 | 50.5 s | 57.3 s | 11.9% faster | 2.162 / 2.164 | From 2 to 8 machines, GPUlayer scaled 2.35x vs. Lightning’s 2.11x , finishing 11.9% faster on 8 machines while converging to the same loss. The difference comes from two-level gradient synchronization ADR-0020 https://www.gpulayer.dev/docs/decisions/ADR-0020-two-level-gradient-sync/ . When running multi-node DDP over gloo with multiple processes per host, standard flat ring all-reduce sends messages across every rank on every machine. GPUlayer instead forms an intra-node group on each machine and an inter-node group across local rank 0s: gradients are summed inside each machine in shared memory first, followed by a single all-reduce across the machines, and broadcast back locally. On 8 machines x 4 processes, that cut the per-step time from 117 ms to 102 ms. 3. Direct distributed launcher vs. torchrun By default launcher="direct" , GPUlayer workers spawn rank processes directly with the exact environment variables torch.distributed expects MASTER ADDR , MASTER PORT , WORLD SIZE , RANK , LOCAL RANK instead of going through torchrun ’s elastic agent rendezvous. Across 2 to 16 machines, the direct launcher saves 1.1 to 2.3 seconds per job compared to torchrun . On an 8-machine 16-process job, startup + collective finished in 4.3 s vs. 6.9 s with torchrun ; on 16 machines 32 processes , 5.4 s vs. 8.1 s . 4. Fleet throughput and 30-machine scale tests To test mixed-workload scheduling across heterogeneous machines, we first ran two GCP VMs one n1-standard-8 with a Tesla T4 and one e2-standard-8 , 15 job slots total : | Workload | Jobs | Wall Clock | Cumulative Task Time | Jobs per Worker cpu8 , t4-cpu , t4 | |---|---|---|---|---| | Tiny jobs 0.02 s each | 500 | 9.0 s | 11.0 s | 243, 188, 69 | | Data shards 0.6 s each | 200 | 11.8 s | 127.8 s | 115, 72, 13 | | LR training sweep | 96 | 55.2 s | 241.7 s | 56, 34, 6 | We then scaled the fleet to 30 worker VMs n1-standard-4 , 120 concurrent slots split across two regions—23 in Belgium europe-west1 and 7 in the Netherlands europe-west4 : - 5,000 tiny jobs, 2,000 data shards, and a 480-run training sweep all completed cleanly across all 30 machines. - Fault tolerance and fail-fast : When one rank in an 8-machine distributed job exited with an error, GPUlayer aborted every rank across all 8 machines within 4.9 seconds . When 3 worker VMs were killed mid-run during a 600-job batch, the 9 in-flight tasks on those hosts were automatically rescheduled onto healthy workers with 0 failed jobs . - 92.7 jobs/second throughput : Submitting 2,000 jobs with Client.submit batch POST /v1/jobs/batch ingested all 2,000 jobs in 1.1 seconds and drained them in 21.6 seconds across 8 workers. Getting from early prototypes 7.8 jobs/s to 92.7 jobs/s took four concrete systems fixes that are documented in the changelog: 1. TCP NODELAY and thread-shared HTTPS connection pools : On Linux, Nagle’s algorithm + delayed ACKs were holding responses for 40 ms and opening fresh TLS handshakes per worker thread. Reuse + TCP NODELAY cut per-request latency from 43 ms to 0.6 ms. 2. Keyed long-poll wakeups : Idle workers wait on condition variables keyed to capability changes rather than polling every 200 ms, cutting a 480-job training sweep on 30 workers from 102 s to 45 s. 3. Partial SQLite indexes on active tasks : As historical tasks grew past 13,000 rows, scanning for active reservations took 2.2 ms per scheduling pass; a partial index over active states dropped that query to 0.002 ms . 4. WAL mode with synchronous=NORMAL and batch submission : Reduced disk commit latency on standard cloud persistent disks from 2.25 ms to 0.02 ms and server CPU cost to 4.1 ms per job. How GPUlayer compares to other tools Most tools in the GPU ecosystem live either inside the training script or at the cloud VM provisioning layer. Often they complement GPUlayer rather than replace it: | Tool | Primary Layer | Chooses Hardware? | Uses Existing Machines? | Workloads | |---|---|---|---|---| | GPUlayer | Workload placement & execution layer | Yes GPU, VRAM, frameworks, rack/zone topology, + GCP provisioning | Yes gpulayer join , laptop, servers, K8s | Commands in any language, @gl.remote Python functions, distributed PyTorch, services | | PyTorch Lightning / Accelerate | In-process training loop organization | No runs where launched | Yes manual SSH / torchrun | PyTorch training & inference | | Ray | Distributed Python runtime & actor framework | Inside a standing Ray cluster | Yes requires Ray cluster | Python tasks/actors serialized against Ray APIs | | SkyPilot / dstack | Multi-cloud VM & container orchestrators | Yes cloud instance catalogs & fleets | dstack yes; SkyPilot via SSH node pools | Task specs / containers / services | | Modal | Hosted serverless GPU cloud | Yes Modal’s cloud only | No cannot bring your own GPUs | Python functions, containers, web endpoints | | Slurm / Kubernetes | Multi-tenant cluster schedulers | Yes within the cluster | Yes requires cluster ops | Batch scripts Slurm / containers K8s | - PyTorch Lightning & Accelerate : Choose them when you need rich training features like DeepSpeed, dozens of loggers, or complex model parallelism. You can pair them with gpulayer.pytorch.launch so GPUlayer handles machine selection, code staging, and multi-node process launch while Lightning or Accelerate runs the loop. - Ray : Choose Ray when you need stateful distributed actors, a shared in-memory object store, or Ray Tune/Serve. Choose GPUlayer when you want to run arbitrary shell commands, binaries, MLX on Macs, or standard Python/PyTorch scripts without rewriting them around a cluster runtime. - Modal : @gl.remote and .map feel very similar to Modal, except GPUlayer is self-hosted and runs on hardware you already own or rent—including Apple Silicon Macs and local workstations. Current boundaries What is not built or validated yet: - Multi-GPU per host and multi-node NCCL : Selecting the best-connected GPUs via nvidia-smi topo -m and multi-GPU reservations are implemented and tested against recorded topologies, but hardware validation so far has used single-GPU T4 VMs and multi-node CPU gloo up to 30 VMs while waiting on higher cloud GPU quota. - Cloud providers beyond Google Cloud : Automatic per-job VM creation and teardown is validated on GCP CPU and Spot T4 . AWS, Azure, and specialized GPU clouds are not wired up yet, though any VM you boot yourself can join with gpulayer join . - Service load balancing : Long-running services manage replicas, $PORT assignment, readiness, and restarts, but do not yet include a reverse proxy or autoscaler in front of replica endpoints. - Untrusted multi-tenant sandboxing : Workloads execute as normal OS processes on workers. GPUlayer is designed for an engineer or a team sharing hardware they trust, not for running untrusted third-party code. Try it in five minutes GPUlayer requires Python 3.10+. PyTorch is optional if you only run commands or non-PyTorch Python functions. Install from PyPI or pip install "gpulayer torch " for the PyTorch Trainer pip install gpulayer Inspect what GPUs, memory, and frameworks this machine offers gpulayer resources detect Run a command job locally gpulayer submit --source examples/shell job \ --input examples/shell job/sales.csv:sales.csv -- bash report.sh Start a control plane and open the web dashboard gpulayer up gpulayer dashboard - Website & Documentation : gpulayer.dev https://www.gpulayer.dev/ · Quickstart https://www.gpulayer.dev/docs/get-started/quickstart/ - Detailed Comparisons : gpulayer.dev/compare https://www.gpulayer.dev/compare/ - Benchmarks & Raw Logs : Performance and comparison https://www.gpulayer.dev/docs/get-started/performance-and-comparison/ - Source & Issues : github.com/antonellof/gpulayer https://github.com/antonellof/gpulayer