Run your LLM evals every night on a GPU that only exists for a minute

A developer demonstrated setting up nightly LLM evaluations using Runcrate's CLI, which spins up a fresh GPU on a cron schedule and tears it down when the eval exits. Running EleutherAI's lm-evaluation-harness on Qwen2.5-0.5B-Instruct over 200 ARC-Easy questions, the job scored 0.565 accuracy and cost about $0.01 on a single A6000 at $0.65/hr, with roughly one minute of the 1m3s runtime spent on the eval itself. The setup uses a `--max-runtime` cap and a timezone-aware cron so runs stay at 02:00 local across daylight saving changes.

Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago. The fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal. I'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: Qwen/Qwen2.5-0.5B-Instruct on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change. The command that runs on the GPU machine: pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && \ uv pip install -q "lm-eval hf " torch --torch-backend=cu128 && \ lm eval --model hf --model args pretrained=Qwen/Qwen2.5-0.5B-Instruct \ --tasks arc easy --limit 200 --batch size 16 The first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. --torch-backend=cu128 picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain pip install first and got a PyTorch wheel that refused the driver, so pin this to your driver. brew install aeonmindai/tap/runcrate runcrate login login opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there. runcrate schedule create --cron '0 2 ' --tz Europe/London --name nightly-eval \ --gpu A6000 --max-runtime 30m -- \ 'pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && uv pip install -q "lm-eval hf " torch --torch-backend=cu128 && lm eval --model hf --model args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc easy --limit 200 --batch size 16' What it printed: Schedule created: nightly-eval ID: e18aeb6f-6d15-4f51-804b-65b258ce71e1 Kind: job When: 0 2 Europe/London Overlap: skip Enabled: yes Next run: 2026-10-22 02:00 BST GPU: 1x A6000 Max runtime: 30m A few things worth knowing about those flags: -- makes it a job. --max-runtime is required. --tz keeps wall-clock time. 0 2 in Europe/London stays at 02:00 local across daylight saving changes. runcrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1 Started job nightly-eval ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 . Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 Run now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs schedules are checked every 2 minutes . runcrate jobs logs prints the job's output. The end of it: hf {'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'} , gen kwargs: {} , limit: 200.0, num fewshot: None, batch size: 16 | Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr| |--------|------:|------|-----:|--------|---|----:|---|-----:| |arc easy| 1|none | 0|acc |↑ |0.565|± |0.0351| | | |none | 0|acc norm|↑ |0.560|± |0.0352| And runcrate jobs status for what it cost: Name: nightly-eval Status: ● succeeded Exit code: 0 GPU: 1x A6000 Max runtime: 30m Runtime: 1m3s Cost: $0.01 $0.65/hr Created: 2026-10-21T01:08:32.934+00:00 Started: 2026-10-21T01:12:50.771+00:00 Ended: 2026-10-21T01:13:53.945+00:00 About four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU. runcrate jobs list shows every night's run with its status, exit code and cost, so a week of runs is one command: ID NAME STATUS GPU EXIT RUNTIME AGE COST ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 nightly-eval ● succeeded A6000 0 1m 7m ago $0.01 The toy version prints a number. To make it catch regressions: pretrained= at the checkpoint under /workspace . Volumes are covered in the --output path /workspace/evals to lm eval , then a two-line Python check that exits non-zero when accuracy drops below your threshold. The job ends failed with that exit code, which is what you watch for. HF TOKEN and friends in the environment's secrets set in the dashboard , which every job receives as environment variables. runcrate schedule list runcrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1 runcrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y Pausing stops future runs and leaves anything already running alone. If you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so "every night at 2am London time, run my eval on an A6000 for at most 30 minutes" works as a prompt. Setup for each client is on the MCP setup page https://www.runcrate.ai/docs/mcp/setup?utm source=devto&utm medium=community&utm campaign=tutorials . If you set this up for your own model and something gets in the way, tell me. Reply here or write to hello@runcrate.ai mailto:hello@runcrate.ai . Written with AI assistance. Every command and output above was run on 2026-10-21.