{"slug": "xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the", "title": "xrun: smoke-test on a free Kaggle GPU, train on vast.ai, let the agent read the metrics", "summary": "A developer built xrun, a Rust CLI with a Python Textual TUI that automates GPU training runs across six vendors — vast.ai, Kaggle, Lightning AI, Google Colab, SSH servers and local machines — from a single YAML manifest. The tool rents instances, uploads data, launches training, follows stages and metrics, pulls checkpoints and destroys instances on completion, failure, idle timeout, metric plateau or cost/time limits, and ships a Claude Code/Codex skill so an agent can drive the same commands. A training-script hook writes events.jsonl and metrics.jsonl for a background poller, with a separate watchdog run every five minutes to catch a dead poller while an instance is still billing.", "body_md": "I train 3D computer vision models, and classical ones at hackathons. For a long time every run came with the same chores: `vastai create instance`, `scp` the data up, start the script, check on it, `scp` the checkpoint back, remember to destroy the instance. Kaggle had its own version of that with `kaggle kernels push`. I wanted to hand the whole loop to an agent: launch on whichever platform fits, smoke-test on a free Kaggle GPU before paying for the real training, and collect metrics the same way everywhere, so the agent can read them and decide what to do next.\n\nThat became xrun, a Rust CLI with a Python Textual TUI on top. One YAML manifest describes a run: where the GPU comes from, what to upload, what to run and what to bring back. xrun rents the instance, uploads the data, starts training, follows stages and metrics, pulls the checkpoints and destroys the instance when it's over. It supports six vendors: vast.ai, Kaggle, Lightning AI, Google Colab, your own server over SSH, and the local machine.\n\n```\nname: resnet50_baseline\nvendor: vast                    # vast | kaggle | lightning | colab | ssh | local\n\nvast:\n  image: pytorch/pytorch:2.4.1-cuda12.1-cudnn9-devel\n  gpu: { type: \"RTX 4090\", count: 1 }\n  price: { max_per_hour: 0.55 }\n\ndata:\n  - src: data/train.h5\n    dst: /workspace/data/train.h5\n\nrun:\n  cmd: python train.py\n  args:\n    --lr: 5e-4\n    --epochs: 30\n\nartifacts:\n  patterns: [\"checkpoints/best*.pt\"]\n\npolicy:\n  on_idle_minutes: 30           # no output for 30 minutes: destroy\n  early_stop:                   # metric plateau: pull the best, then destroy\n    metric: val_f1\n    patience: 5\n```\n\nAnd the day-to-day loop:\n\n```\nxrun launch exp/resnet50.yaml --detach --max-cost 5\nxrun events <id> --follow\nxrun metrics <id> --key val_f1 --ascii\nxrun pull <id> --ckpt best --into models/\n```\n\nThe smoke test is a second manifest with `vendor: kaggle` and a small config. The training script and the metrics code stay the same, so if it survives a few epochs on Kaggle, I launch the vast.ai one.\n\nA copy of the manifest is stored with the run, and its hash is part of the run's identity. I can edit the original freely and `xrun rerun <id>` still reproduces exactly what ran. For hyperparameters, `xrun sweep exp/base.yaml --grid run.args.--lr=1e-3,5e-4 --launch` writes out one manifest per variant.\n\nEvery read command (`ls`, `show`, `events`, `metrics`) has `--json`, and whatever the TUI can do, the CLI can do too. `xrun show` prints each metric's last value and step, so a score logged once at the end of a cross-validation run is visible without plotting anything. `xrun install skill --claude` (or `--codex`) installs a skill that teaches Claude Code or Codex to use xrun instead of hand-rolled `vastai` and `ssh` calls. So the agent can rerun the last experiment with a different learning rate, or pull the checkpoint of the run with the best val_f1 from the last day, through the same commands I would type.\n\nMetrics come from a small hook in the training script:\n\n``` python\nfrom xrun_hook import stage, metric, metrics, done, notify\n\nwith stage(\"train\"):\n    for ep in range(epochs):\n        loss = train_one_epoch(model, loader)\n        metric(\"train_loss\", loss, step=ep)\n        metrics({\"val_loss\": v.loss, \"val_f1\": v.f1}, step=ep)\n\nnotify(\"training\", f\"best val_f1 = {best:.3f}\")\ndone()\n```\n\nThe hook writes `events.jsonl` and `metrics.jsonl`, and a background poller picks them up. On Kaggle xrun injects the hook into the kernel by itself. An uncaught exception gets recorded as an `error` event.\n\n`xrun launch --detach` leaves that poller running. It counts the spend and destroys the instance when the run finishes, fails, goes silent, hits a metric plateau or runs into `--max-cost` / `--max-hours`. If the destroy call fails, you get a push saying the instance is still billing. The poller can't report its own death, so `xrun watchdog` runs from the OS scheduler every five minutes: no heartbeat from the poller while the instance is alive means a notification and a respawned poller. It also catches instances that no run record knows about.\n\nNotifications go to ntfy, Telegram, a webhook (Slack, Discord) or desktop toasts. Replying `/stop <id>` to the Telegram bot kills a run.\n\nThe run history lives in a local SQLite database, with no tracking service or account involved. MLflow and W&B plug in as mirrors if you want their charts.\n\nThe docs under `docs/` are in Russian (the English README covers the basics, and the commands are the same). `xrun_hook` isn't on PyPI yet, so outside Kaggle it travels with your code. Kaggle shows metrics mid-run only when an MLflow server is configured; without one you see the start and the end. Lightning AI was smoke-tested live on a CPU Studio only, and for Lightning and Colab xrun doesn't know the price, so `--max-cost` doesn't apply there.\n\nmacOS and Linux:\n\n```\ncurl -sSf https://raw.githubusercontent.com/Gerrux/xrun/master/install.sh | sh\n```\n\nWindows (PowerShell):\n\n```\nirm https://raw.githubusercontent.com/Gerrux/xrun/master/install.ps1 | iex\n```\n\n`xrun doctor` and then `xrun launch exp/templates/quickstart.yaml` check the whole chain locally with no credentials, GPU or data. The code is on GitHub under MIT: [Gerrux/xrun](https://github.com/Gerrux/xrun/blob/master/README.en.md).\n\nIf you run training on one of these vendors, a report of what broke is the most useful thing you could send me. Each of their APIs has its own habits, and half of the fixes so far came from exactly that.\n\n*I built xrun together with my AI sidekick, Claude Code (Opus 5.5), and this post was written with its help too.*", "url": "https://wpnews.pro/news/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the", "canonical_source": "https://dev.to/gerrux/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vastai-let-the-agent-read-the-metrics-2803", "published_at": "2026-11-01 18:48:47+00:00", "updated_at": "2026-11-01 18:52:07.905244+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "mlops", "ai-infrastructure", "developer-tools"], "entities": ["xrun", "vast.ai", "Kaggle", "Lightning AI", "Google Colab", "Claude Code", "Codex", "Rust"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the", "markdown": "https://wpnews.pro/news/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the.md", "text": "https://wpnews.pro/news/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the.txt", "jsonld": "https://wpnews.pro/news/xrun-smoke-test-on-a-free-kaggle-gpu-train-on-vast-ai-let-the-agent-read-the.jsonld"}}