Most GPU software assumes the GPU is attached to the machine where you type python.
PyTorch, Lightning, Hugging Face Accelerate, JAX, MLX, and CuPy all do a great job deciding how a computation uses the hardware it is given. None of them decide which machine should run the job, ship your project and inputs there, check whether the driver and libraries actually work before starting, or bring logs and checkpoints back when it finishes.
I built GPUlayer (GitHub) to solve two closely related problems:
- My GPUs are spread across different machines : an Apple Silicon laptop, Linux servers, and rented cloud VMs. Every time I started a new workload, I ended up repeating the same manual ritual—
ssh,rsync, hoping the virtualenv and CUDA build still matched the driver, startingtmux, and copying checkpoints back. - I needed a way to give my coding agents control over all of my GPUs : I do most of my day-to-day work inside coding agents—Claude Code, Pi agents, and MCP-enabled editors—running on my laptop. Handing an agent raw SSH keys and teaching it which box is free, which one has CUDA vs. Metal (
mps/mlx), and how torsyncartifacts back is fragile and dangerous. I wanted one structured control layer so an agent on my MacBook can inspect every GPU I can reach, submit a training run or benchmark to the right machine, stream the logs, and pull back the results.
Before GPUlayer, running a job on another machine looked like this:
ssh gpu-box
rsync -av my_project/ gpu-box:~/my_project/
tmux new -s train
cd ~/my_project && python train.py
Multiply that by several machines with different drivers, a Mac with Metal (mps and mlx) instead of CUDA, a 96-run learning-rate sweep, or a multi-node job where torchrun needs matching ranks, addresses, and ports on every host.
With GPUlayer, that becomes:
pip install gpulayer
gpulayer submit --accelerator cuda --source my_project -- python train.py
GPUlayer is published on PyPI under Apache 2.0, and validated on Apple Silicon, NVIDIA CUDA (Tesla T4), automatic Google Cloud GPU provisioning, and fleets of up to 30 machines across two cloud regions. This post walks through what it does, how the scheduler and distributed runtime work, how it performs against PyTorch Lightning and torchrun, and where its boundaries are today.
What GPUlayer does #
GPUlayer sits underneath your frameworks and above your machines: your laptop, servers that joined with gpulayer join, Kubernetes worker pools, or cloud VMs provisioned on demand.
Everything goes through the same queue, capability matcher, and wire protocol across four workload types:
1. Any command or binary
Scripts, shell pipelines, C++ or Rust binaries, Blender renders, or ffmpeg transcodes run as jobs without wrapper code. You point GPUlayer at a source directory, declare inputs and outputs, and state the hardware requirements:
gpulayer submit --accelerator cuda --source ./app -- ./render.sh
gpulayer submit --source examples/shell_job \
--input examples/shell_job/sales.csv:sales.csv -- bash report.sh
2. Python functions with any library
Decorate a function with @gl.remote, specify the accelerator and framework versions it needs, and call it directly or fan it out across every free worker with .map():
import gpulayer as gl
@gl.remote(accelerator="mps", frameworks={"mlx": ">=0.10"})
def matmul_benchmark(n: int) -> dict:
import mlx.core as mx
...
results = matmul_benchmark.map([1024, 2048, 4096])
Arguments and return values are plain JSON, and code travels as a clean source bundle rather than brittle pickled bytecode.
3. PyTorch training and distributed launch
GPUlayer includes a focused PyTorch integration (gpulayer.pytorch) with a Trainer, Runtime, and launch(). The exact same script runs in-process on your laptop, as multi-process DDP on one machine, or across multiple nodes:
python examples/trainer_regression/train.py
python examples/trainer_regression/train.py --accelerator cpu --devices 2
python examples/trainer_regression/train.py --accelerator cpu --nodes 2 --devices 2 --target remote
Or directly in Python:
import gpulayer.pytorch as glt
trainer = glt.Trainer(
accelerator="cuda",
devices=4,
num_nodes=2,
max_epochs=10,
target="remote",
)
trainer.fit(model, train_, val_)
The Trainer handles device placement, mixed precision (fp16/ bf16), torch.compile, gradient accumulation with DDP no_sync, sanity validation steps, callbacks ( EarlyStopping, ModelCheckpoint(save_top_k=N), LearningRateMonitor), test(), predict(), CUDA side-stream batch prefetching, two-level gradient sync for multi-node CPU/gloo runs, and experimental FSDP2 (strategy="fsdp"). If you already use PyTorch Lightning or Accelerate, glt.launch() places the job and starts the processes with the standard torchrun environment variables they expect.
4. Long-running services
For inference endpoints or persistent workers, gpulayer submit --service (or gl.service(...)) runs replicas on distinct machines, assigns each replica a free $PORT, waits for it to accept connections, reports host:port endpoints via Job.endpoints(), restarts crashed replicas with exponential backoff, and re-places replicas if a worker machine disappears.
Placement that explains itself #
When a job fails on a cluster, the worst outcome is waiting three minutes for container setup and import torch only to hit RuntimeError: No CUDA GPUs are available or a missing library.
In GPUlayer, workers probe their hardware and environment on startup. For GPU libraries (torch.cuda, jax.cuda, cupy.cuda, torch.mps, mlx), GPUlayer does not merely check whether the Python package is importable—it runs a lightweight GPU probe in a separate interpreter to verify that the library can actually execute on the device. A CPU-only jax or broken CUDA installation will never advertise itself as a working GPU worker.
When a job arrives, the scheduler evaluates every worker against the job’s requirements. If no worker qualifies, the job is rejected immediately with per-worker reasons:
scheduling: no eligible worker
gpu-box-1 needs 2 usable cuda device(s), worker has 1
mac-studio framework 'jax' not installed
laptop torch on this worker cannot use CUDA
gpu-box-2 eligible
Topology-aware groups and NVLink/PCIe GPU selection
Placement goes beyond counting free GPUs:
- Rack, zone, and region locality (
--group-locality) : Workers report their cloudregion,zone(read automatically from the GCE metadata server on Google Cloud or passed via flags/env), and an optionalgpulayer.dev/network-grouplabel for rack or InfiniBand islands. When gang-scheduling a multi-node job, the defaultpreferpolicy keeps all nodes in the same network group if possible, then the same zone, then the same region, before falling back to cross-region placement. - Best-connected GPUs on a host : On multi-GPU machines, workers parse
nvidia-smi topo -minto link speeds and NUMA affinities. When a task asks for multiple GPUs on an 8-GPU server, GPUlayer picks the free subset whose slowest pairwise interconnect is fastest (preferring an NVLink island or GPUs behind a single PCIe switch over arbitrary lowest indices) and exportsCUDA_DEVICE_ORDER=PCI_BUS_IDso mixed-GPU machines never mismatch device indices.
From one laptop to cloud GPUs, Kubernetes, and AI agents #
Adding machines in one command
For local work, the control plane runs embedded inside your Python process or CLI call with zero setup. When you want to pool multiple machines, start the server on one host:
gpulayer up
gpulayer up generates a TLS certificate with a pinned fingerprint, creates named role-based tokens (admin, submitter, join, metrics), and prints the exact gpulayer join command to paste on any other machine:
gpulayer join https://192.168.1.50:8443 --token <join-token> --tls-fingerprint SHA256:...
Because workers poll the control plane over outbound HTTPS connections, a worker on a home workstation or lab server behind NAT can join without port forwarding or VPN tunnels.
Automatic Google Cloud provisioning
When your standing machines are busy—or when you develop on a MacBook and only want to pay for an NVIDIA GPU while a job runs—GPUlayer can provision cloud VMs per job (gpulayer[gcp]).
Configure your allowed machine offers in providers.json, start gpulayer up --providers providers.json, and submit with --provider gcp --accelerator cuda. When no existing worker fits, GPUlayer creates a Compute Engine VM (such as a Spot Tesla T4), boots it with a single-use join token, runs the job as soon as the worker reports torch.cuda, downloads the checkpoints and logs, and deletes the VM when the job finishes. In our validation runs, a fresh Spot T4 VM went from API creation to running a CUDA Trainer job in 78 seconds and tore itself down immediately on completion.
Web dashboard, Prometheus, and Kubernetes
The server includes a built-in web dashboard (gpulayer dashboard) showing queued and running jobs, per-rank live logs, placement diagnostics, worker hardware inventory, and live GPU/CPU utilization, alongside a /metrics Prometheus endpoint and prebuilt Grafana dashboard.
For teams running on Kubernetes, GPUlayer ships container images on GitHub Container Registry (ghcr.io/antonellof/gpulayer) and a Helm chart ( deploy/kubernetes/helm/gpulayer) that deploys the control plane as a StatefulSet alongside CPU and NVIDIA GPU worker pools.
Giving coding agents control over all your GPUs (gpulayer mcp and SKILL.md)
One of my primary reasons for building GPUlayer was letting my coding agents—Claude Code, Pi agents, Cursor, and Codex—use every GPU I own or rent without giving them raw SSH access or make-shift bash loops.
When an agent is writing a kernel, tuning hyperparameters, or debugging a training script on my laptop, it needs to run that code on an NVIDIA GPU or across a multi-node fleet, wait for the outcome, read the logs, and inspect the output files. GPUlayer gives agents three ways to do that through the same capability scheduler and scoped token permissions (submitter or --read-only, never needing an admin token or SSH keys):
- MCP server (
gpulayer mcp) : A built-in Model Context Protocol server over stdio exposing structured tools:gpulayer_resources,gpulayer_submit,gpulayer_wait,gpulayer_job_status,gpulayer_job_logs,gpulayer_list_jobs,gpulayer_cancel, andgpulayer_download_outputs. - Agent Skill (
integrations/skills/gpulayer/SKILL.md) : For CLI-native harnesses likePi andClaude Code , the repository ships a ready-to-useSKILL.mdthat teaches the agent how to inspect available accelerators (gpulayer resources list --json), submit jobs with--accelerator cudaor--accelerator mps, stream logs (gpulayer job logs <id> -f), interpret per-worker rejection reasons instead of blindly retrying, and download outputs. - HTTP / Python SDK : Custom agent loops can hit the REST API or
gpulayer.Clientdirectly.
Wiring it into Claude Code or Pi takes one command:
claude mcp add gpulayer \
-e GPULAYER_URL=https://gpu-server:8470 \
-e GPULAYER_TOKEN=glc_... \
-e GPULAYER_FINGERPRINT=sha256:... \
-- gpulayer mcp
cp -r integrations/skills/gpulayer ~/.claude/skills/
When you tell your agent “Train train.py on an NVIDIA GPU for 3 epochs and tell me the validation loss”, it calls gpulayer_resources, sees which worker has cuda and torch, calls gpulayer_submit with source_dir: "." and accelerator: "cuda", waits with gpulayer_wait, and reads the metrics back—while every agent-submitted job is tracked by token identity (submitted_by) on the dashboard.
Performance and benchmarks #
Every benchmark below comes from reproducible scripts in benchmarks/, with raw logs checked into benchmarks/results/.
1. Single-GPU training speed vs. PyTorch Lightning
We trained the exact same models, datasets, and SGD optimizer with one logged loss per step using GPUlayer’s Trainer and PyTorch Lightning’s Trainer (with Lightning’s logger, progress bar, model summary, and checkpointing turned off to make the loop comparison fair) on a Google Cloud n1-standard-8 VM with an NVIDIA Tesla T4 (driver 580, Python 3.12, PyTorch 2.15.0, CUDA 13.2, Lightning 2.6.6). Each configuration ran 100 and 600 steps three times in fresh processes, taking the difference of median wall times divided by 500 steps:
| Workload (Tesla T4 / CPU) | GPUlayer (ms/step) | Lightning (ms/step) | Speedup | GPUlayer startup | Lightning startup |
|---|---|---|---|---|---|
| MLP, fp32, batch 256 | 4.06 ms | 5.64 ms | 1.39x faster | 4.77 s | 5.31 s |
| MLP, fp16 mixed precision | 5.00 ms | 6.25 ms | 1.25x faster | 4.82 s | 5.55 s |
| CNN, fp16 mixed, batch 128 | 10.82 ms | 12.33 ms | 1.14x faster | 5.65 s | 5.74 s |
| DDP, 2 CPU processes | 3.20 ms | 9.56 ms | 2.99x faster | 10.32 s | 9.35 s |
Why the numbers differ:
- Small models (MLP) : When a forward/backward pass takes 4 ms, framework bookkeeping per step is visible. GPUlayer’s loop avoids synchronous device-to-host stalls when recording step losses.
- Compute-bound CNN : On larger convolutional steps, GPUlayer enables two optimizations by default on CUDA: cuDNN autotuning (
cudnn.benchmark = Trueunlessdeterministic=True) and asynchronous side-stream batch prefetching (Runtime.batches), which pins the next batch and copies it to the GPU on a secondary CUDA stream while the current step computes. Ablating those two flags on the same T4 (benchmarks/results/2026-10-31-gcp-t4/) isolated their contribution:12.67 ms/step with both off,12.10 ms with prefetch only (~5%),11.53 ms with cuDNN autotuning only (~9%), and10.93 ms with both enabled. - CPU DDP : GPUlayer pins each worker process to a dedicated thread count by default, whereas Lightning leaves PyTorch’s default multi-threading untouched, causing two ranks to contend over the 8 vCPUs.
- Startup overhead : On a single GPU, GPUlayer starts training 0.1 to 0.7 seconds sooner than Lightning.
2. Multi-node strong scaling: 2 to 8 machines (32 processes)
Next, we ran a multi-node CNN training job (32,768 images, global batch size 256 split across all ranks, 2 epochs) across 2, 4, and 8 GCP n1-standard-4 VMs (4 processes per machine, up to 32 processes total over gloo). Both trainers ran on the exact same VMs in interleaved order. With GPUlayer, a single Trainer(num_nodes=N, devices=4) call scheduled and launched all machines; with Lightning, torchrun was started on every node with matching rendezvous addresses.
| Machines | Total Processes | GPUlayer Wall Time | Lightning ( torchrun ) |
GPUlayer Lead | Final Loss (GL / PL) |
|---|---|---|---|---|---|
| 2 | 8 | 118.9 s | 120.8 s | 1.6% faster | 2.129 / 2.125 |
| 4 | 16 | 73.3 s | 77.4 s | 5.3% faster | 2.143 / 2.137 |
| 8 | 32 | 50.5 s | 57.3 s | 11.9% faster | 2.162 / 2.164 |
From 2 to 8 machines, GPUlayer scaled 2.35x vs. Lightning’s 2.11x, finishing 11.9% faster on 8 machines while converging to the same loss.
The difference comes from two-level gradient synchronization (ADR-0020). When running multi-node DDP over gloo with multiple processes per host, standard flat ring all-reduce sends messages across every rank on every machine. GPUlayer instead forms an intra-node group on each machine and an inter-node group across local rank 0s: gradients are summed inside each machine in shared memory first, followed by a single all-reduce across the machines, and broadcast back locally. On 8 machines x 4 processes, that cut the per-step time from 117 ms to 102 ms.
3. Direct distributed launcher vs. torchrun
By default (launcher="direct"), GPUlayer workers spawn rank processes directly with the exact environment variables torch.distributed expects (MASTER_ADDR, MASTER_PORT, WORLD_SIZE, RANK, LOCAL_RANK) instead of going through torchrun’s elastic agent rendezvous.
Across 2 to 16 machines, the direct launcher saves 1.1 to 2.3 seconds per job compared to torchrun. On an 8-machine (16-process) job, startup + collective finished in 4.3 s vs. 6.9 s with torchrun; on 16 machines (32 processes), 5.4 s vs. 8.1 s.
4. Fleet throughput and 30-machine scale tests
To test mixed-workload scheduling across heterogeneous machines, we first ran two GCP VMs (one n1-standard-8 with a Tesla T4 and one e2-standard-8, 15 job slots total):
| Workload | Jobs | Wall Clock | Cumulative Task Time | Jobs per Worker ( cpu8 ,t4-cpu ,t4 ) |
|---|---|---|---|---|
| Tiny jobs (0.02 s each) | 500 | 9.0 s | 11.0 s | 243, 188, 69 |
| Data shards (0.6 s each) | 200 | 11.8 s | 127.8 s | 115, 72, 13 |
| LR training sweep | 96 | 55.2 s | 241.7 s | 56, 34, 6 |
We then scaled the fleet to 30 worker VMs (n1-standard-4, 120 concurrent slots) split across two regions—23 in Belgium (europe-west1) and 7 in the Netherlands ( europe-west4):
- 5,000 tiny jobs, 2,000 data shards, and a 480-run training sweep all completed cleanly across all 30 machines.
- Fault tolerance and fail-fast : When one rank in an 8-machine distributed job exited with an error, GPUlayer aborted every rank across all 8 machines within4.9 seconds . When 3 worker VMs were killed mid-run during a 600-job batch, the 9 in-flight tasks on those hosts were automatically rescheduled onto healthy workers with0 failed jobs .
- 92.7 jobs/second throughput : Submitting 2,000 jobs with
Client.submit_batch(POST /v1/jobs/batch) ingested all 2,000 jobs in1.1 seconds and drained them in21.6 seconds across 8 workers.
Getting from early prototypes (7.8 jobs/s) to 92.7 jobs/s took four concrete systems fixes that are documented in the changelog:
TCP_NODELAYand thread-shared HTTPS connection pools : On Linux, Nagle’s algorithm + delayed ACKs were holding responses for 40 ms and opening fresh TLS handshakes per worker thread. Reuse +TCP_NODELAYcut per-request latency from 43 ms to 0.6 ms.- Keyed long-poll wakeups : Idle workers wait on condition variables keyed to capability changes rather than polling every 200 ms, cutting a 480-job training sweep on 30 workers from 102 s to 45 s.
- Partial SQLite indexes on active tasks : As historical tasks grew past 13,000 rows, scanning for active reservations took 2.2 ms per scheduling pass; a partial index over active states dropped that query to0.002 ms .
- WAL mode with
synchronous=NORMALand batch submission : Reduced disk commit latency on standard cloud persistent disks from 2.25 ms to 0.02 ms and server CPU cost to 4.1 ms per job.
How GPUlayer compares to other tools #
Most tools in the GPU ecosystem live either inside the training script or at the cloud VM provisioning layer. Often they complement GPUlayer rather than replace it:
| Tool | Primary Layer | Chooses Hardware? | Uses Existing Machines? | Workloads |
|---|---|---|---|---|
| GPUlayer | Workload placement & execution layer | Yes (GPU, VRAM, frameworks, rack/zone topology, + GCP provisioning) | Yes ( gpulayer join , laptop, servers, K8s) |
Commands in any language, @gl.remote Python functions, distributed PyTorch, services |
| PyTorch Lightning / Accelerate | In-process training loop organization | No (runs where launched) | Yes (manual SSH / torchrun ) |
PyTorch training & inference |
| Ray | Distributed Python runtime & actor framework | Inside a standing Ray cluster | Yes (requires Ray cluster) | Python tasks/actors serialized against Ray APIs |
| SkyPilot / dstack | Multi-cloud VM & container orchestrators | Yes (cloud instance catalogs & fleets) | dstack yes; SkyPilot via SSH node pools | Task specs / containers / services |
| Modal | Hosted serverless GPU cloud | Yes (Modal’s cloud only) | No (cannot bring your own GPUs) | Python functions, containers, web endpoints |
| Slurm / Kubernetes | Multi-tenant cluster schedulers | Yes (within the cluster) | Yes (requires cluster ops) | Batch scripts (Slurm) / containers (K8s) |
- PyTorch Lightning & Accelerate : Choose them when you need rich training features like DeepSpeed, dozens of loggers, or complex model parallelism. You can pair them with
gpulayer.pytorch.launch()so GPUlayer handles machine selection, code staging, and multi-node process launch while Lightning or Accelerate runs the loop. - Ray : Choose Ray when you need stateful distributed actors, a shared in-memory object store, or Ray Tune/Serve. Choose GPUlayer when you want to run arbitrary shell commands, binaries, MLX on Macs, or standard Python/PyTorch scripts without rewriting them around a cluster runtime.
- Modal :
@gl.remoteand.map()feel very similar to Modal, except GPUlayer is self-hosted and runs on hardware you already own or rent—including Apple Silicon Macs and local workstations.
Current boundaries #
What is not built or validated yet:
- Multi-GPU per host and multi-node NCCL : Selecting the best-connected GPUs via
nvidia-smi topo -mand multi-GPU reservations are implemented and tested against recorded topologies, but hardware validation so far has used single-GPU T4 VMs and multi-node CPUgloo(up to 30 VMs) while waiting on higher cloud GPU quota. - Cloud providers beyond Google Cloud : Automatic per-job VM creation and teardown is validated on GCP (CPU and Spot T4). AWS, Azure, and specialized GPU clouds are not wired up yet, though any VM you boot yourself can join with
gpulayer join. - Service load balancing : Long-running services manage replicas,
$PORTassignment, readiness, and restarts, but do not yet include a reverse proxy or autoscaler in front of replica endpoints. - Untrusted multi-tenant sandboxing : Workloads execute as normal OS processes on workers. GPUlayer is designed for an engineer or a team sharing hardware they trust, not for running untrusted third-party code.
Try it in five minutes #
GPUlayer requires Python 3.10+. PyTorch is optional if you only run commands or non-PyTorch Python functions.
pip install gpulayer
gpulayer resources detect
gpulayer submit --source examples/shell_job \
--input examples/shell_job/sales.csv:sales.csv -- bash report.sh
gpulayer up
gpulayer dashboard
- Website & Documentation :gpulayer.dev ·Quickstart
- Detailed Comparisons :gpulayer.dev/compare
- Benchmarks & Raw Logs :Performance and comparison
- Source & Issues :github.com/antonellof/gpulayer