{"slug": "run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute", "title": "Run your LLM evals every night on a GPU that only exists for a minute", "summary": "A developer demonstrated setting up nightly LLM evaluations using Runcrate's CLI, which spins up a fresh GPU on a cron schedule and tears it down when the eval exits. Running EleutherAI's lm-evaluation-harness on Qwen2.5-0.5B-Instruct over 200 ARC-Easy questions, the job scored 0.565 accuracy and cost about $0.01 on a single A6000 at $0.65/hr, with roughly one minute of the 1m3s runtime spent on the eval itself. The setup uses a `--max-runtime` cap and a timezone-aware cron so runs stay at 02:00 local across daylight saving changes.", "body_md": "Most eval suites run when someone remembers to run them. That's usually right before a release, which is the worst time to find out a model regressed two weeks ago.\n\nThe fix is boring: run the evals every night, on a GPU that exists only while they run, and read the number in the morning. This post sets that up with Runcrate's CLI: one command creates a nightly schedule, each run is a job on a fresh GPU that stops when the eval exits, and a run costs about a cent for the example here. I ran every command below on 2026-10-21 and the outputs are copied from my terminal.\n\nI'm using EleutherAI's lm-evaluation-harness on a small model so the run is quick: `Qwen/Qwen2.5-0.5B-Instruct` on 200 questions of ARC-Easy. Swap in your own model and tasks; the scheduling part doesn't change.\n\nThe command that runs on the GPU machine:\n\n```\npip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && \\\nuv pip install -q \"lm-eval[hf]\" torch --torch-backend=cu128 && \\\nlm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct \\\n  --tasks arc_easy --limit 200 --batch_size 16\n```\n\nThe first line makes a clean virtualenv with uv, so the eval doesn't fight with whatever Python packages the machine image ships. `--torch-backend=cu128` picks the PyTorch build that matches the machine's CUDA 12.8 driver. I tried a plain `pip install` first and got a PyTorch wheel that refused the driver, so pin this to your driver.\n\n```\nbrew install aeonmindai/tap/runcrate\nruncrate login\n```\n\n`login` opens the browser. You pick a workspace and an environment, and everything the CLI starts runs and bills there.\n\n```\nruncrate schedule create --cron '0 2 * * *' --tz Europe/London --name nightly-eval \\\n  --gpu A6000 --max-runtime 30m -- \\\n  'pip install -q uv && uv venv -q /opt/env && . /opt/env/bin/activate && uv pip install -q \"lm-eval[hf]\" torch --torch-backend=cu128 && lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-0.5B-Instruct --tasks arc_easy --limit 200 --batch_size 16'\n```\n\nWhat it printed:\n\n```\n  Schedule created: nightly-eval\n  ID:           e18aeb6f-6d15-4f51-804b-65b258ce71e1\n  Kind:         job\n  When:         0 2 * * * (Europe/London)\n  Overlap:      skip\n  Enabled:      yes\n  Next run:     2026-10-22 02:00 BST\n  GPU:          1x A6000\n  Max runtime:  30m\n```\n\nA few things worth knowing about those flags:\n\n`--` makes it a job.`--max-runtime` is required.`--tz` keeps wall-clock time.`0 2 * * *` in Europe/London stays at 02:00 local across daylight saving changes.\n\n```\nruncrate schedule run-now e18aeb6f-6d15-4f51-804b-65b258ce71e1\nStarted job nightly-eval (ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33). Follow it with: runcrate jobs logs -f ca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33\n```\n\nRun now starts the same job straight away and doesn't move the next scheduled run. I also let a schedule fire on its own while testing: it was due at 02:00 BST and launched at 02:02, which matches the docs (schedules are checked every 2 minutes).\n\n`runcrate jobs logs` prints the job's output. The end of it:\n\n```\nhf ({'pretrained': 'Qwen/Qwen2.5-0.5B-Instruct'}), gen_kwargs: ({}), limit: 200.0, num_fewshot: None, batch_size: 16\n| Tasks  |Version|Filter|n-shot| Metric |   |Value|   |Stderr|\n|--------|------:|------|-----:|--------|---|----:|---|-----:|\n|arc_easy|      1|none  |     0|acc     |↑  |0.565|±  |0.0351|\n|        |       |none  |     0|acc_norm|↑  |0.560|±  |0.0352|\n```\n\nAnd `runcrate jobs status` for what it cost:\n\n```\n  Name:         nightly-eval\n  Status:       ● succeeded\n  Exit code:    0\n  GPU:          1x A6000\n  Max runtime:  30m\n  Runtime:      1m3s\n  Cost:         $0.01 ($0.65/hr)\n  Created:      2026-10-21T01:08:32.934+00:00\n  Started:      2026-10-21T01:12:50.771+00:00\n  Ended:        2026-10-21T01:13:53.945+00:00\n```\n\nAbout four minutes went to getting the machine ready and one minute to the eval itself. The max runtime counts from when the command starts, so setup doesn't eat into your 30 minutes. A6000 was $0.65 an hour when I ran this; check the live price before you pick a GPU.\n\n`runcrate jobs list` shows every night's run with its status, exit code and cost, so a week of runs is one command:\n\n```\nID                                    NAME          STATUS       GPU    EXIT  RUNTIME  AGE     COST\nca55cfe7-7dcf-4f1f-a1a3-45e6bb2eab33  nightly-eval  ● succeeded  A6000  0     1m       7m ago  $0.01\n```\n\nThe toy version prints a number. To make it catch regressions:\n\n`pretrained=` at the checkpoint under `/workspace`. Volumes are covered in the `--output_path /workspace/evals` to `lm_eval`, then a two-line Python check that exits non-zero when accuracy drops below your threshold. The job ends `failed` with that exit code, which is what you watch for.`HF_TOKEN` and friends in the environment's secrets (set in the dashboard), which every job receives as environment variables.\n\n```\nruncrate schedule list\nruncrate schedule pause e18aeb6f-6d15-4f51-804b-65b258ce71e1\nruncrate schedule delete e18aeb6f-6d15-4f51-804b-65b258ce71e1 -y\n```\n\nPausing stops future runs and leaves anything already running alone.\n\nIf you use Claude Code, Codex or Cursor, the Runcrate MCP server has the same schedule tools, so \"every night at 2am London time, run my eval on an A6000 for at most 30 minutes\" works as a prompt. Setup for each client is on the [MCP setup page](https://www.runcrate.ai/docs/mcp/setup?utm_source=devto&utm_medium=community&utm_campaign=tutorials).\n\nIf you set this up for your own model and something gets in the way, tell me. Reply here or write to [hello@runcrate.ai](mailto:hello@runcrate.ai).\n\n*Written with AI assistance. Every command and output above was run on 2026-10-21.*", "url": "https://wpnews.pro/news/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute", "canonical_source": "https://dev.to/runcrate/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute-1fj9", "published_at": "2026-10-22 18:00:00+00:00", "updated_at": "2026-10-22 18:11:42.226768+00:00", "lang": "en", "topics": ["mlops", "ai-tools", "large-language-models", "ai-infrastructure"], "entities": ["Runcrate", "EleutherAI", "Qwen", "lm-evaluation-harness", "ARC-Easy", "NVIDIA A6000"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute", "markdown": "https://wpnews.pro/news/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute.md", "text": "https://wpnews.pro/news/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute.txt", "jsonld": "https://wpnews.pro/news/run-your-llm-evals-every-night-on-a-gpu-that-only-exists-for-a-minute.jsonld"}}