Metal Sandbox: Isolated Execution and Command Scheduling for GPU Coding Agents on Apple Silicon

A developer built Metal Sandbox, a system that isolates file workspaces and schedules command execution for GPU coding agents across a pool of Apple Silicon nodes, migrating eligible commands to idle machines when local capacity is saturated. The accompanying paper (arXiv 2611.17294v1) reports 1.68× to 2.05× command throughput gains on four Mac mini M4 nodes under concentrated demand and a 22.3% reduction in batch elapsed time when dynamic scheduling replaces static assignment. Replaying recorded agent sessions at four sessions per node yielded an estimated 2.14× sustained throughput.

Agents that generate GPU kernels need to compile, test, and iterate on Metal code without monopolizing the host GPU or crashing when concurrent sessions compete for command queues. Metal Sandbox addresses this by isolating file workspaces, scheduling command execution across a pool of Apple Silicon nodes, and migrating eligible commands to idle machines when local capacity is saturated. The paper arXiv 2611.17294v1 reports 1.68× to 2.05× command throughput gains on four Mac mini M4 nodes under concentrated demand, and 22.3% reductions in batch elapsed time when dynamic scheduling replaces static assignment. A typical GPU coding agent alternates between three phases: xcrun metal or similar toolchain commands. Each phase touches the GPU differently. Inference saturates compute units. Compilation spawns CPU-bound toolchain processes but may trigger shader validation on the GPU. Execution submits command buffers that block until the GPU finishes. When multiple agents share one Mac, naive scheduling leads to: .metallib files or test harnesses. Metal Sandbox provides three layers: Each agent session gets a private directory tree: /sandbox/agent-<uuid / src/ generated .metal files build/ compiled .metallib artifacts test/ execution harnesses logs/ stdout/stderr from commands File access is restricted via macOS sandbox profiles. An agent cannot read or write outside its assigned prefix. This prevents one session from corrupting another's build artifacts or leaking intermediate results. Commands fall into two categories: The scheduler maintains one serial execution slot per node. Parallel commands run concurrently up to a configurable limit typically matching CPU core count . When a serial command arrives and the local slot is occupied, the scheduler can: Migration copies the agent's workspace snapshot to the remote node, executes the command, and returns results exit code, stdout, stderr, modified files to the originating agent. Harbor is the workflow orchestration layer. It: Harbor exposes a REST API. Agents submit commands as JSON payloads: { "agent id": "uuid", "command": "xcrun metal -o kernel.metallib kernel.metal", "type": "parallel", "workspace": "/sandbox/agent-uuid" } The scheduler returns a handle. The agent polls or subscribes to a webhook for completion. The scheduler uses a two-level queue: | Queue Type | Scope | Policy | |---|---|---| | Local | Per-node | FIFO for serial, concurrent for parallel | | Global | Cross-node | Work-stealing for eligible commands | Eligibility for migration : a command can migrate if: Migration cost model : the scheduler estimates: migration cost = snapshot size / network bandwidth + remote queue wait local cost = local queue wait If migration cost < local cost , the command migrates. Failure handling : if a remote execution crashes GPU hang, kernel panic , the scheduler: The agent sees a retry, not a crash. The host process remains stable. Metal Sandbox exposes metrics via Prometheus endpoints: metal command duration seconds{type, node} : histogram of command execution time. metal gpu utilization{node} : sampled GPU active time from ioreg . metal compilation failures{node} : counter of failed metal migration count{from, to} : cross-node command transfers. Agents can query these metrics to adjust their generation strategy. For example, if compilation failures spike, the agent might reduce code complexity or switch to a simpler kernel template. The paper tests two scenarios: Each agent submits a burst of commands. Without scheduling, agents queue locally and wait. With Metal Sandbox: The gain comes from parallel commands spreading across nodes and serial commands migrating to idle slots instead of waiting behind local queues. Replaying recorded agent sessions with four sessions per node versus one yields an estimated 2.14× sustained throughput . Dynamic scheduling further reduces: These gains assume phase durations remain constant. In practice, remote execution may add network latency, but the paper's controlled LAN setup 1 Gbps Ethernet keeps migration overhead below 200 ms for typical workspaces. | Failure | Impact | Mitigation | |---|---|---| | GPU hang during kernel execution | Serial slot blocked until watchdog timeout | Per-command timeout default 30s , node marked degraded | | Compilation crash invalid Metal syntax | Command fails, agent retries | Sandbox isolation prevents host crash, logs captured for debugging | | Network partition during migration | Command lost, agent sees timeout | Resubmit to local queue, mark remote node unavailable | | Workspace snapshot corruption | Execution fails with missing files | Checksum validation before migration, retry with fresh snapshot | The scheduler does not attempt distributed consensus. If a node becomes unreachable, commands in flight are resubmitted. Agents tolerate retries because kernel generation is idempotent same input code produces same compiled artifact . python import requests import time def execute kernel agent id, workspace, kernel path : payload = { "agent id": agent id, "command": f"./run kernel {kernel path}", "type": "serial", "workspace": workspace, "timeout": 30 } resp = requests.post "http://scheduler:8080/submit", json=payload handle = resp.json "handle" while True: status = requests.get f"http://scheduler:8080/status/{handle}" .json if status "state" == "completed": return status "exit code" , status "stdout" elif status "state" == "failed": raise RuntimeError status "error" time.sleep 0.5 The agent does not know or care whether the command ran locally or migrated. The scheduler handles placement transparently. A typical setup: Each node runs macOS with Metal developer tools installed. The scheduler is a Go binary that uses os/exec for command execution and net/http for RPC. Harbor is a Python service with a SQLite database for rollout tracking. For cloud deployments, replace Mac minis with EC2 Mac instances. Network latency increases migration cost, so tune the eligibility threshold or disable migration for latency-sensitive workloads. /etc/passwd or writing to system directories. The scheduler does not authenticate agents. In a multi-tenant environment, add API keys or mTLS between agents and Harbor. Use Metal Sandbox when: Avoid it when: The infrastructure is open source and integrates cleanly with existing agent frameworks. If you are building GPU code generation agents on Apple Silicon, Metal Sandbox solves the resource contention problem without requiring a rewrite of your agent loop.