Free Workers Don't Ship Free Disk: Cap Artifact Retention First

A developer documented a batch-processing incident in which 4,000 jobs writing roughly 2 MB of artifacts each filled a worker's disk after three days, causing write failures and requeued jobs that re-spent tokens on already-completed work. The fix is a pre-flight admission check that computes required bytes from queue depth, average artifact size and a safety factor against free space and a reserve, exiting with code 75 (EX_TEMPFAIL) to signal retryable-later, plus a retention script that caps the artifact store and defaults to dry-run with pinned artifacts protected.

At 03:12 the alert fired on queue age. Not on errors, not on latency. The batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts. Nobody had set a retention policy. Three days of runs had filled the worker's disk. Writes started failing. The consumer requeued failed jobs. Tokens were spent again on work already paid for once. The evidence said storage. The dashboard said AI capacity. Which action do you take first? Observed: OSError: Errno 28 No space left on device , then a growing retry count. Observed: p95 queue age crossed 15 minutes while model latency stayed flat. Inferred: the model got slower. That inference was wrong in this incident, and it usually is. Disk pressure looks like throughput pressure from the outside. Retries hide the real cause inside the token bill. A free worker gives you compute. It does not give you unbounded storage. Write the budget down before you enqueue anything: required bytes = queue depth avg artifact bytes safety factor + reserve bytes reserve bytes = total disk bytes reserve ratio Declared workload for the reference run below: queue depth = 4000 avg artifact bytes = 2 000 000 JSON result, stdout log, preview safety factor = 1.3 variance between jobs total disk bytes = 20 GB , reserve ratio = 0.2 That is 10.4 GB of artifacts against a 4 GB reserve. Tight, not impossible. Without the reserve, it fails quietly at the end of the batch. Reject the batch before it starts, not after job 3,400. python admit.py - refuse a batch the disk cannot finish import shutil def admit store, queue depth, avg bytes, reserve ratio=0.20, safety=1.3 : total, used, free = shutil.disk usage store need = int queue depth avg bytes safety reserve = int total reserve ratio verdict = free - need = reserve return verdict, { "free bytes": free, "need bytes": need, "reserve bytes": reserve, "headroom bytes": free - need - reserve, } if name == " main ": ok, report = admit "/srv/artifacts", 4000, 2 000 000 print report raise SystemExit 0 if ok else 75 75 = EX TEMPFAIL Exit code 75 tells your scheduler this is retryable-later, not a bug. Do not enqueue on False . Check the same number from the shell before you trust it: df -B1 --output=source,avail,pcent /srv/artifacts find /srv/artifacts -type f -newermt '-24 hours' -printf '%s\n' \ | awk '{s+=$1; n++} END {printf "files=%d avg bytes=%d\n", n, s/n}' Default is dry-run. Deletion requires --apply . Pinned artifacts survive every pass. bash /usr/bin/env python3 retain.py - cap artifact store before the worker disk fills import argparse, os, shutil from pathlib import Path def pinned store : p = Path store / ".pinned" if not p.exists : return set return {line.strip for line in p.read text .splitlines if line.strip } def candidates store : keep = pinned store out = for entry in Path store .iterdir : if not entry.is dir or entry.name in keep: continue size = sum f.stat .st size for f in entry.rglob " " if f.is file out.append entry.stat .st mtime, entry, size return sorted out oldest first def main : ap = argparse.ArgumentParser ap.add argument "--store", required=True ap.add argument "--max-bytes", type=int, required=True ap.add argument "--keep-newest", type=int, default=50 ap.add argument "--apply", action="store true" a = ap.parse args items = candidates a.store protected = items -a.keep newest: used = sum size for , , size in items freed = 0 for , path, size in items: if used - freed <= a.max bytes: break if path.stat .st mtime, path, size in protected: continue print f"{'rm' if a.apply else 'would rm'} {path} {size}" if a.apply: shutil.rmtree path, ignore errors=True freed += size print f"used={used} freed={freed} target={a.max bytes} apply={a.apply}" if name == " main ": main Run it on a timer, not in your head: python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --keep-newest 50 inspect, then arm it python3 retain.py --store /srv/artifacts --max-bytes 12000000000 --apply Do this on a scratch worker before production finds it for you. dd if=/dev/zero of=/srv/artifacts/fill bs=1M count=3000 status=progress admit.py . Confirm it returns exit 75 once headroom goes negative. fill , run retain.py --apply , confirm the store drops under the cap. Step 4 is the one to instrument. Emit these fields per job: job id , attempt , artifact bytes , write ok disk avail bytes at start and end tokens in , tokens out , tokens wasted on retry If tokens wasted on retry is non-zero, storage is charging you twice. Arithmetic on the declared inputs above, not measurements from your cluster: | Item | Per 1,000 jobs | Note | |---|---|---| | Artifacts at 2 MB | 2 GB | grows linearly with retention window | | Retry tokens after write failure | 1x generation cost | artifacts lost, model output not cached | | Pruner CPU | seconds | rmtree on local disk | | Object storage offload | per GB-month | check egress before assuming cheap | Expected output shape from admit.py on a healthy worker: {'free bytes': 14800000000, 'need bytes': 10400000000, 'reserve bytes': 4000000000, 'headroom bytes': 400000000} A 400 KB headroom is a warning, not a pass. Raise safety factor or split the batch. Stop and re-plan if any of these hold: In those cases, write artifacts to durable object storage and keep only recent pointers on the worker. Pay for storage, not for retries. I use MonkeyCode's free model access and free server option for batch experimentation. Both are operator-supplied availability claims, not measured guarantees here. Treat the disk as a budget you declare yourself. The workflow above does not depend on them. Run it on any worker with a local disk. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This is a single-node pattern. It does not give you replication, cross-zone durability, or shared storage across workers. Skip it if you need multi-writer concurrency on the same store. Skip it if your compliance team requires immutable retention. Skip it if your artifacts are larger than your reserve can absorb. Do not use the pruner as your only backup. It deletes. That is the point. If the arithmetic above says the disk cannot finish the batch, then free capacity is the wrong bet for that batch. Before arming the timer, record the current state: du -sb /srv/artifacts /srv/artifacts.size.before find /srv/artifacts -maxdepth 1 -type d | wc -l /srv/artifacts.count.before Rollback is two commands: systemctl disable --now artifact-retain.timer rm -f /srv/artifacts/fill Then restore what you deleted from your offload bucket. If you have no offload bucket, you did not have a rollback. Start with a dry-run on one day of artifacts. Compare the file list against what you expected to keep. Only then arm --apply . If you want a cheap place to rehearse this drill, the free server option is a reasonable scratch target. Keep the pin list short and the blast radius smaller than your resume.