{"slug": "checkpoint-run-resume-long-shell-jobs-after-your-connection-drops", "title": "Checkpoint-run: resume long shell jobs after your connection drops", "summary": "Checkpoint-run, a new developer utility for resuming long shell jobs after connection drops, was released on 2026-08-18 and hardened through three rounds, adding step timeouts, retry with exponential backoff, and checksum-verified completion. The tool targets rural NZ broadband users, where connectivity issues are documented, and allows jobs to resume from the last completed step instead of restarting. It includes 60 tests and a GitHub Actions CI workflow.", "body_md": "Checkpoint-and-resume runner for long multi-step jobs on flaky connections.\n\n**New market (2026-08-18 round):** not an AI wrapper, not a prompt pack, not a narration pack.\nReplacement product after the critical audit marked `nz-swms-builder`\n\n**NO-GO** (false legal\npremise — see `../AUDIT-REPORT.md`\n\n). This product makes **no legal, safety, or regulatory\nclaims**: it is a purely functional developer utility.\n\n**Hardened (2026-08-19, round 2):** step timeouts (`--timeout`\n\n/ `--step-timeout`\n\n),\n`--continue-on-error`\n\n, checksum-verified step completion (`--step-artifact`\n\n), `--dry-run`\n\npreview, and a shell-execution safety banner.\n\n**Hardened again (2026-08-19, round 3):** per-attempt retry with exponential backoff\n(`--retry N`\n\n/ `--backoff`\n\n), a failure cap for `--continue-on-error`\n\n(`--max-failures`\n\n),\nper-step output capture (`--step-log`\n\n/ `--logs-dir`\n\n), and a GitHub Actions CI workflow\n(`.github/workflows/ci.yml`\n\n). See the flags below.\n\nRural NZ broadband is a documented crisis (Federated Farmers: \"rural connectivity crisis\ndemands urgent action\" — [scoop.co.nz](https://www.scoop.co.nz/stories/print.html?path=AK2509/S00830/rural-connectivity-crisis-demands-urgent-action.htm);\n\"Northland farmers losing time and money to poor internet\" — [ruralnewsgroup](https://www.ruralnewsgroup.co.nz/rural-news/rural-general-news/northland-farmers-losing-money-poor-internet)).\nOn a dropped hotspot connection, long jobs (LLM batch, video render, big download) die mid-run\nand restart from zero. This machine's own AGENTS.md notes: \"drops kill long API runs.\"\n\n**Buyer:** NZ rural/dev users running long jobs; any developer with flaky local networking.**Value:** split a long job into named steps (`id:command`\n\n); completed steps are persisted to a local JSON state file; re-run**resumes where the job died** instead of starting over.\n\n| Path | Purpose |\n|---|---|\n`checkpoint_runner/runner.py` |\nCore: state file (atomic writes), merge/resume logic, injectable shell runner, timeout watchdog, artifact checksums, retry/backoff loop, per-step log capture |\n`checkpoint_runner/cli.py` |\nCLI entry point (`checkpoint-run` ) |\n`tests/test_runner.py` |\n60 tests: completion, resume-skips-completed, failure handling, reset, force, determinism, CLI end-to-end, timeouts, continue-on-error, artifact checksums, dry-run, retry/backoff, max-failures, step logs |\n`sample_output/` |\nExample state files + run logs (round 1 demo, round 2 hardening demo, round 3 retry/logs demo) |\n\n```\ncd checkpoint-run\n\n# Run a 3-step job; if the hotspot drops mid-run, just re-run the same command.\nuv run python -m checkpoint_runner.cli \\\n  --job render-batch \\\n  --state-dir . \\\n  --step download:\"curl -L -o model.bin https://example.com/model.bin\" \\\n  --step convert:\"ffmpeg -i in.mov out.mp4\" \\\n  --step upload:\"rsync -P out.mp4 backup:render/\"\n\n# Inspect progress / clear state\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . --status\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . --reset\n\n# Preview what would run WITHOUT executing anything (shell-safety check)\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step download:\"curl -L -o model.bin URL\" --step render:\"ffmpeg -i in.mov out.mp4\" --dry-run\n\n# Per-step time budget (a step exceeding it is recorded 'timeout' and stops the run)\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step download:\"curl -L -o model.bin URL\" --step render:\"ffmpeg -i in.mov out.mp4\" \\\n  --timeout 3600 --step-timeout download:600\n\n# Keep running later steps even if one fails\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step a:\"cmd\" --step b:\"cmd2\" --continue-on-error\n\n# Checksum-verified completion: re-runs a 'done' step if its output is missing/changed\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step render:\"ffmpeg -i in.mov out.mp4\" --step-artifact render:out.mp4\n\n# Retry a flaky step up to 2 extra times with exponential backoff\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step download:\"curl -L -o model.bin URL\" --retry 2 --backoff 1.0\n\n# Stop after 2 recorded failures even with --continue-on-error\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step a:\"cmd\" --step b:\"cmd2\" --continue-on-error --max-failures 2\n\n# Capture every step's stdout+stderr to logs/<id>.log (or --step-log id:path per step)\nuv run python -m checkpoint_runner.cli --job render-batch --state-dir . \\\n  --step render:\"ffmpeg -i in.mov out.mp4\" --logs-dir logs\n```\n\nOr install it (from the [v0.1.0 release](https://github.com/agenticaotearoa/checkpoint-run/releases/tag/v0.1.0)):\n\n```\npython -m venv .venv && .venv/bin/pip install \\\n  https://github.com/agenticaotearoa/checkpoint-run/releases/download/v0.1.0/checkpoint_run-0.1.0-py3-none-any.whl\ncheckpoint-run --job demo --step \"a:echo hi\" --step \"b:echo bye\"\n```\n\nNo runtime dependencies: the wheel is pure Python (3.10+) on the standard library.\n\n**Resume:** re-running skips steps already`done`\n\n(same id + same command) and starts at the first incomplete step.**Failure:** a failing step is recorded as`failed`\n\n(exit code stored) and later steps do not run (unless`--continue-on-error`\n\n); re-running retries from that step.**Timeout:** with`--timeout SECONDS`\n\n(or`--step-timeout id:SECONDS`\n\n), a step that exceeds its budget is killed (default runner) and recorded as`timeout`\n\n; later steps stop unless`--continue-on-error`\n\n. Re-running retries the timed-out step. The watchdog only observes a*single*budget; it does not bound grandchild processes (documented limitation).**Artifact checksums:** with`--step-artifact id:PATH`\n\n, the step's output file hash (sha256) is recorded on completion. On resume, a`done`\n\nstep is trusted only if the artifact still exists with the same content — missing or changed output (deleted file, restored snapshot, corrupted write) forces the step to re-run. A step that claims success but produces no declared artifact is recorded`failed`\n\n(exit -2) and retried.**Dry run:**`--dry-run`\n\nprints which steps would run and which would be skipped, executing nothing — a safe way to inspect the plan before letting commands touch the machine.**Retry with backoff:**`--retry N`\n\nre-attempts a failing/timeout step up to N extra times*within the same run*— a transient hotspot drop often succeeds on a second try. Between attempts the runner waits`--backoff * 2^(attempt-1)`\n\nseconds (default base 1.0 s). Only the final attempt's result is persisted; the state file records the attempt count. A step that succeeds but still lacks its declared artifact is also retried.**Failure cap:** with`--continue-on-error`\n\n,`--max-failures N`\n\nstops the run once N failed/timeout steps have been recorded (later steps stay`pending`\n\nand run on the next resume). Without`--continue-on-error`\n\nthe run already stops at the first failure.**Step logs:**`--step-log id:path`\n\n(or`--logs-dir dir`\n\nfor all steps) captures a step's stdout+stderr into a file instead of the terminal. The file is truncated per attempt, so a retried step's log shows its final attempt. (Output capture applies to the built-in shell runner; a custom injected runner controls its own I/O.)**Changed command:** if a step's command text changes, it is treated as pending again.**Force:**`--force`\n\nre-runs everything from scratch (fresh state).**State:**`.checkpoint-run-<job>.json`\n\nnext to the job (or in`--state-dir`\n\n), written atomically (temp file + rename) immediately after each step exits — a kill/drop never loses completed work. No cloud, no network, no LLM, no secrets.\n\nSteps are executed through the shell **exactly as you write them** (`subprocess.run(shell=True)`\n\n)\n— the CLI prints a reminder of this on every run, and `--dry-run`\n\npreviews the plan without\nexecuting. You are responsible for the commands you pass. This is a resume orchestrator, not a\nsandbox.\n\n```\npython -m pytest tests/ -v\n# 60 passed  (2026-08-19, round 4 — full re-verify on this machine)\n# note: the suite invokes `python` through the shell, so run it with the\n# interpreter on PATH (e.g. `.venv/bin`).\n```\n\n`checkpoint-run`\n\nis free and open source (MIT). This is a genuine demand test, not a storefront:\n\n- If a\n**maintained, supported edition**(priority bug fixes, team/CI onboarding, or a packaged retry service for unattended jobs) at ~US$49 one-time per seat would help you, say so in[Discussions](https://github.com/agenticaotearoa/checkpoint-run/discussions)— start or upvote a thread titled**\"Paid pilot interest\"**. - If you'd pay a different amount, that's even more useful: state the number.\n- Responses decide whether a paid pilot ships. No payment is taken here; nothing is collected, and there is no hidden paywall.\n\n**Bugs / feature requests**→[Issues](https://github.com/agenticaotearoa/checkpoint-run/issues)(use the templates: bug report or feature request).** Questions, show-and-tell, demand votes**→[Discussions](https://github.com/agenticaotearoa/checkpoint-run/discussions).- Both are watched; the fastest way to shape the roadmap is a Discussion.\n\nThis product's claims are deliberately low-risk: it records which steps completed and resumes\nthere. It does not assert legal requirements, safety authority, or regulatory compliance —\nthe failure class that killed `nz-swms-builder`\n\n(see `../AUDIT-REPORT.md`\n\n).", "url": "https://wpnews.pro/news/checkpoint-run-resume-long-shell-jobs-after-your-connection-drops", "canonical_source": "https://github.com/agenticaotearoa/checkpoint-run", "published_at": "2026-08-19 06:12:27+00:00", "updated_at": "2026-08-19 06:42:13.045341+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Checkpoint-run", "Federated Farmers", "GitHub Actions"], "alternates": {"html": "https://wpnews.pro/news/checkpoint-run-resume-long-shell-jobs-after-your-connection-drops", "markdown": "https://wpnews.pro/news/checkpoint-run-resume-long-shell-jobs-after-your-connection-drops.md", "text": "https://wpnews.pro/news/checkpoint-run-resume-long-shell-jobs-after-your-connection-drops.txt", "jsonld": "https://wpnews.pro/news/checkpoint-run-resume-long-shell-jobs-after-your-connection-drops.jsonld"}}