{"slug": "free-workers-don-t-ship-free-disk-cap-artifact-retention-first", "title": "Free Workers Don't Ship Free Disk: Cap Artifact Retention First", "summary": "A developer documented a batch-processing incident in which 4,000 jobs writing roughly 2 MB of artifacts each filled a worker's disk after three days, causing write failures and requeued jobs that re-spent tokens on already-completed work. The fix is a pre-flight admission check that computes required bytes from queue depth, average artifact size and a safety factor against free space and a reserve, exiting with code 75 (EX_TEMPFAIL) to signal retryable-later, plus a retention script that caps the artifact store and defaults to dry-run with pinned artifacts protected.", "body_md": "At 03:12 the alert fired on queue age. Not on errors, not on latency.\n\nThe batch had 4,000 jobs. Each accepted job wrote about 2 MB of artifacts. Nobody had set a retention policy.\n\nThree days of runs had filled the worker's disk. Writes started failing. The consumer requeued failed jobs. Tokens were spent again on work already paid for once.\n\nThe evidence said storage. The dashboard said AI capacity. Which action do you take first?\n\nObserved: `OSError: [Errno 28] No space left on device`, then a growing retry count.\n\nObserved: p95 queue age crossed 15 minutes while model latency stayed flat.\n\nInferred: the model got slower. That inference was wrong in this incident, and it usually is.\n\nDisk pressure looks like throughput pressure from the outside. Retries hide the real cause inside the token bill.\n\nA free worker gives you compute. It does not give you unbounded storage.\n\nWrite the budget down before you enqueue anything:\n\n```\nrequired_bytes = queue_depth * avg_artifact_bytes * safety_factor + reserve_bytes\nreserve_bytes  = total_disk_bytes * reserve_ratio\n```\n\nDeclared workload for the reference run below:\n\n`queue_depth = 4000`` avg_artifact_bytes = 2_000_000` (JSON result, stdout log, preview)`safety_factor = 1.3` (variance between jobs)`total_disk_bytes = 20 GB`, `reserve_ratio = 0.2`\nThat is 10.4 GB of artifacts against a 4 GB reserve. Tight, not impossible. Without the reserve, it fails quietly at the end of the batch.\n\nReject the batch before it starts, not after job 3,400.\n\n``` python\n# admit.py - refuse a batch the disk cannot finish\nimport shutil\n\ndef admit(store, queue_depth, avg_bytes, reserve_ratio=0.20, safety=1.3):\n    total, used, free = shutil.disk_usage(store)\n    need = int(queue_depth * avg_bytes * safety)\n    reserve = int(total * reserve_ratio)\n    verdict = (free - need) >= reserve\n    return verdict, {\n        \"free_bytes\": free,\n        \"need_bytes\": need,\n        \"reserve_bytes\": reserve,\n        \"headroom_bytes\": free - need - reserve,\n    }\n\nif __name__ == \"__main__\":\n    ok, report = admit(\"/srv/artifacts\", 4000, 2_000_000)\n    print(report)\n    raise SystemExit(0 if ok else 75)  # 75 = EX_TEMPFAIL\n```\n\nExit code 75 tells your scheduler this is retryable-later, not a bug. Do not enqueue on `False`.\n\nCheck the same number from the shell before you trust it:\n\n```\ndf -B1 --output=source,avail,pcent /srv/artifacts\nfind /srv/artifacts -type f -newermt '-24 hours' -printf '%s\\n' \\\n  | awk '{s+=$1; n++} END {printf \"files=%d avg_bytes=%d\\n\", n, s/n}'\n```\n\nDefault is dry-run. Deletion requires `--apply`. Pinned artifacts survive every pass.\n\n``` bash\n#!/usr/bin/env python3\n# retain.py - cap artifact store before the worker disk fills\nimport argparse, os, shutil\nfrom pathlib import Path\n\ndef pinned(store):\n    p = Path(store) / \".pinned\"\n    if not p.exists():\n        return set()\n    return {line.strip() for line in p.read_text().splitlines() if line.strip()}\n\ndef candidates(store):\n    keep = pinned(store)\n    out = []\n    for entry in Path(store).iterdir():\n        if not entry.is_dir() or entry.name in keep:\n            continue\n        size = sum(f.stat().st_size for f in entry.rglob(\"*\") if f.is_file())\n        out.append((entry.stat().st_mtime, entry, size))\n    return sorted(out)  # oldest first\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--store\", required=True)\n    ap.add_argument(\"--max-bytes\", type=int, required=True)\n    ap.add_argument(\"--keep-newest\", type=int, default=50)\n    ap.add_argument(\"--apply\", action=\"store_true\")\n    a = ap.parse_args()\n\n    items = candidates(a.store)\n    protected = items[-a.keep_newest:]\n    used = sum(size for _, _, size in items)\n    freed = 0\n\n    for _, path, size in items:\n        if used - freed <= a.max_bytes:\n            break\n        if (path.stat().st_mtime, path, size) in protected:\n            continue\n        print(f\"{'rm' if a.apply else 'would rm'} {path} {size}\")\n        if a.apply:\n            shutil.rmtree(path, ignore_errors=True)\n        freed += size\n\n    print(f\"used={used} freed={freed} target={a.max_bytes} apply={a.apply}\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it on a timer, not in your head:\n\n```\npython3 retain.py --store /srv/artifacts --max-bytes 12000000000 --keep-newest 50\n# inspect, then arm it\npython3 retain.py --store /srv/artifacts --max-bytes 12000000000 --apply\n```\n\nDo this on a scratch worker before production finds it for you.\n\n`dd if=/dev/zero of=/srv/artifacts/fill bs=1M count=3000 status=progress``admit.py`. Confirm it returns exit 75 once headroom goes negative.` fill`, run `retain.py --apply`, confirm the store drops under the cap.\nStep 4 is the one to instrument. Emit these fields per job:\n\n`job_id`, `attempt`, `artifact_bytes`, `write_ok`\n`disk_avail_bytes` at start and end`tokens_in`, `tokens_out`, `tokens_wasted_on_retry`\nIf `tokens_wasted_on_retry` is non-zero, storage is charging you twice.\n\nArithmetic on the declared inputs above, not measurements from your cluster:\n\n| Item | Per 1,000 jobs | Note | \n|---|---|---|\n| Artifacts at 2 MB | 2 GB | grows linearly with retention window | \n| Retry tokens after write failure | 1x generation cost | artifacts lost, model output not cached | \n| Pruner CPU | seconds | `rmtree` on local disk | \n| Object storage offload | per GB-month | check egress before assuming cheap | \n\nExpected output shape from `admit.py` on a healthy worker:\n\n```\n{'free_bytes': 14800000000, 'need_bytes': 10400000000, 'reserve_bytes': 4000000000, 'headroom_bytes': 400000000}\n```\n\nA 400 KB headroom is a warning, not a pass. Raise `safety_factor` or split the batch.\n\nStop and re-plan if any of these hold:\n\nIn those cases, write artifacts to durable object storage and keep only recent pointers on the worker. Pay for storage, not for retries.\n\nI use MonkeyCode's free model access and free server option for batch experimentation. Both are operator-supplied availability claims, not measured guarantees here. Treat the disk as a budget you declare yourself.\n\nThe workflow above does not depend on them. Run it on any worker with a local disk.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThis is a single-node pattern. It does not give you replication, cross-zone durability, or shared storage across workers.\n\nSkip it if you need multi-writer concurrency on the same store. Skip it if your compliance team requires immutable retention. Skip it if your artifacts are larger than your reserve can absorb.\n\nDo not use the pruner as your only backup. It deletes. That is the point.\n\nIf the arithmetic above says the disk cannot finish the batch, then free capacity is the wrong bet for that batch.\n\nBefore arming the timer, record the current state:\n\n```\ndu -sb /srv/artifacts > /srv/artifacts.size.before\nfind /srv/artifacts -maxdepth 1 -type d | wc -l > /srv/artifacts.count.before\n```\n\nRollback is two commands:\n\n```\nsystemctl disable --now artifact-retain.timer\nrm -f /srv/artifacts/fill\n```\n\nThen restore what you deleted from your offload bucket. If you have no offload bucket, you did not have a rollback.\n\nStart with a dry-run on one day of artifacts. Compare the file list against what you expected to keep. Only then arm `--apply`.\n\nIf you want a cheap place to rehearse this drill, the free server option is a reasonable scratch target. Keep the pin list short and the blast radius smaller than your resume.", "url": "https://wpnews.pro/news/free-workers-don-t-ship-free-disk-cap-artifact-retention-first", "canonical_source": "https://dev.to/odd_background_328/free-workers-dont-ship-free-disk-cap-artifact-retention-first-2l1c", "published_at": "2026-10-27 12:31:22+00:00", "updated_at": "2026-10-27 12:43:01.424861+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-agents", "developer-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/free-workers-don-t-ship-free-disk-cap-artifact-retention-first", "markdown": "https://wpnews.pro/news/free-workers-don-t-ship-free-disk-cap-artifact-retention-first.md", "text": "https://wpnews.pro/news/free-workers-don-t-ship-free-disk-cap-artifact-retention-first.txt", "jsonld": "https://wpnews.pro/news/free-workers-don-t-ship-free-disk-cap-artifact-retention-first.jsonld"}}