# Run your LLM evals every night on a GPU that only exists for a minute > Source: > Published: 2026-10-22 18:00:00+00:00 Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago. The fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal. I'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: `Qwen/Qwen2.5-0.5B-Instruct` on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change. The command that runs on the GPU machine: ``` pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && \ uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && \ lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct \ --tasks arc_easy --limit 200 --batch_size 16 ``` The first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. `--torch-backend=cu128` picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain `pip install` first and got a PyTorch wheel that refused the driver, so pin this to your driver. ``` brew install aeonmindai/tap/runcrate runcrate login ``` `login` opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there. ``` runcrate schedule create --cron '0 2 * * *' --tz Europe/London --name nightly-eval \ --gpu A6000 --max-runtime 30m -- \ 'pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && uv pip install -q "lm-eval[hf]" torch --torch-backend=cu128 && lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc_easy --limit 200 --batch_size 16' ``` What it printed: ``` Schedule created: nightly-eval ID: e18aeb6f-6d15-4f51-804b-65b258ce71e1 Kind: job When: 0 2 * * * (Europe/London) Overlap: skip Enabled: yes Next run: 2026-10-22 02:00 BST GPU: 1x A6000 Max runtime: 30m ``` A few things worth knowing about those flags: `--` makes it a job.`--max-runtime` is required.`--tz` keeps wall-clock time.`0 2 * * *` in Europe/London stays at 02:00 local across daylight saving changes. ``` runcrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1 Started job nightly-eval (ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33). Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 ``` Run now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs (schedules are checked every 2 minutes). `runcrate jobs logs` prints the job's output. The end of it: ``` hf ({'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'}), gen_kwargs: ({}), limit: 200.0, num_fewshot: None, batch_size: 16 | Tasks |Version|Filter|n-shot| Metric | |Value| |Stderr| |--------|------:|------|-----:|--------|---|----:|---|-----:| |arc_easy| 1|none | 0|acc |↑ |0.565|± |0.0351| | | |none | 0|acc_norm|↑ |0.560|± |0.0352| ``` And `runcrate jobs status` for what it cost: ``` Name: nightly-eval Status: ● succeeded Exit code: 0 GPU: 1x A6000 Max runtime: 30m Runtime: 1m3s Cost: $0.01 ($0.65/hr) Created: 2026-10-21T01:08:32.934+00:00 Started: 2026-10-21T01:12:50.771+00:00 Ended: 2026-10-21T01:13:53.945+00:00 ``` About four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU. `runcrate jobs list` shows every night's run with its status, exit code and cost, so a week of runs is one command: ``` ID NAME STATUS GPU EXIT RUNTIME AGE COST ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33 nightly-eval ● succeeded A6000 0 1m 7m ago $0.01 ``` The toy version prints a number. To make it catch regressions: `pretrained=` at the checkpoint under `/workspace`. Volumes are covered in the `--output_path /workspace/evals` to `lm_eval`, then a two-line Python check that exits non-zero when accuracy drops below your threshold. The job ends `failed` with that exit code, which is what you watch for.`HF_TOKEN` and friends in the environment's secrets (set in the dashboard), which every job receives as environment variables. ``` runcrate schedule list runcrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1 runcrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y ``` Pausing stops future runs and leaves anything already running alone. If you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so "every night at 2am London time, run my eval on an A6000 for at most 30 minutes" works as a prompt. Setup for each client is on the [MCP setup page](https://www.runcrate.ai/docs/mcp/setup?utm_source=devto&utm_medium=community&utm_campaign=tutorials). If you set this up for your own model and something gets in the way, tell me. Reply here or write to [hello@runcrate.ai](mailto:hello@runcrate.ai). *Written with AI assistance. Every command and output above was run on 2026-10-21.*