Checkpoint-run: resume long shell jobs after your connection drops

Checkpoint-run, a new developer utility for resuming long shell jobs after connection drops, was released on 2026-08-18 and hardened through three rounds, adding step timeouts, retry with exponential backoff, and checksum-verified completion. The tool targets rural NZ broadband users, where connectivity issues are documented, and allows jobs to resume from the last completed step instead of restarting. It includes 60 tests and a GitHub Actions CI workflow.

Checkpoint-and-resume runner for long multi-step jobs on flaky connections. New market 2026-08-18 round : not an AI wrapper, not a prompt pack, not a narration pack. Replacement product after the critical audit marked nz-swms-builder NO-GO false legal premise — see ../AUDIT-REPORT.md . This product makes no legal, safety, or regulatory claims : it is a purely functional developer utility. Hardened 2026-08-19, round 2 : step timeouts --timeout / --step-timeout , --continue-on-error , checksum-verified step completion --step-artifact , --dry-run preview, and a shell-execution safety banner. Hardened again 2026-08-19, round 3 : per-attempt retry with exponential backoff --retry N / --backoff , a failure cap for --continue-on-error --max-failures , per-step output capture --step-log / --logs-dir , and a GitHub Actions CI workflow .github/workflows/ci.yml . See the flags below. Rural NZ broadband is a documented crisis Federated Farmers: "rural connectivity crisis demands urgent action" — scoop.co.nz https://www.scoop.co.nz/stories/print.html?path=AK2509/S00830/rural-connectivity-crisis-demands-urgent-action.htm ; "Northland farmers losing time and money to poor internet" — ruralnewsgroup https://www.ruralnewsgroup.co.nz/rural-news/rural-general-news/northland-farmers-losing-money-poor-internet . On a dropped hotspot connection, long jobs LLM batch, video render, big download die mid-run and restart from zero. This machine's own AGENTS.md notes: "drops kill long API runs." Buyer: NZ rural/dev users running long jobs; any developer with flaky local networking. Value: split a long job into named steps id:command ; completed steps are persisted to a local JSON state file; re-run resumes where the job died instead of starting over. | Path | Purpose | |---|---| checkpoint runner/runner.py | Core: state file atomic writes , merge/resume logic, injectable shell runner, timeout watchdog, artifact checksums, retry/backoff loop, per-step log capture | checkpoint runner/cli.py | CLI entry point checkpoint-run | tests/test runner.py | 60 tests: completion, resume-skips-completed, failure handling, reset, force, determinism, CLI end-to-end, timeouts, continue-on-error, artifact checksums, dry-run, retry/backoff, max-failures, step logs | sample output/ | Example state files + run logs round 1 demo, round 2 hardening demo, round 3 retry/logs demo | cd checkpoint-run Run a 3-step job; if the hotspot drops mid-run, just re-run the same command. uv run python -m checkpoint runner.cli \ --job render-batch \ --state-dir . \ --step download:"curl -L -o model.bin https://example.com/model.bin" \ --step convert:"ffmpeg -i in.mov out.mp4" \ --step upload:"rsync -P out.mp4 backup:render/" Inspect progress / clear state uv run python -m checkpoint runner.cli --job render-batch --state-dir . --status uv run python -m checkpoint runner.cli --job render-batch --state-dir . --reset Preview what would run WITHOUT executing anything shell-safety check uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" --dry-run Per-step time budget a step exceeding it is recorded 'timeout' and stops the run uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" \ --timeout 3600 --step-timeout download:600 Keep running later steps even if one fails uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step a:"cmd" --step b:"cmd2" --continue-on-error Checksum-verified completion: re-runs a 'done' step if its output is missing/changed uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step render:"ffmpeg -i in.mov out.mp4" --step-artifact render:out.mp4 Retry a flaky step up to 2 extra times with exponential backoff uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step download:"curl -L -o model.bin URL" --retry 2 --backoff 1.0 Stop after 2 recorded failures even with --continue-on-error uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step a:"cmd" --step b:"cmd2" --continue-on-error --max-failures 2 Capture every step's stdout+stderr to logs/<id .log or --step-log id:path per step uv run python -m checkpoint runner.cli --job render-batch --state-dir . \ --step render:"ffmpeg -i in.mov out.mp4" --logs-dir logs Or install it from the v0.1.0 release https://github.com/agenticaotearoa/checkpoint-run/releases/tag/v0.1.0 : python -m venv .venv && .venv/bin/pip install \ https://github.com/agenticaotearoa/checkpoint-run/releases/download/v0.1.0/checkpoint run-0.1.0-py3-none-any.whl checkpoint-run --job demo --step "a:echo hi" --step "b:echo bye" No runtime dependencies: the wheel is pure Python 3.10+ on the standard library. Resume: re-running skips steps already done same id + same command and starts at the first incomplete step. Failure: a failing step is recorded as failed exit code stored and later steps do not run unless --continue-on-error ; re-running retries from that step. Timeout: with --timeout SECONDS or --step-timeout id:SECONDS , a step that exceeds its budget is killed default runner and recorded as timeout ; later steps stop unless --continue-on-error . Re-running retries the timed-out step. The watchdog only observes a single budget; it does not bound grandchild processes documented limitation . Artifact checksums: with --step-artifact id:PATH , the step's output file hash sha256 is recorded on completion. On resume, a done step is trusted only if the artifact still exists with the same content — missing or changed output deleted file, restored snapshot, corrupted write forces the step to re-run. A step that claims success but produces no declared artifact is recorded failed exit -2 and retried. Dry run: --dry-run prints which steps would run and which would be skipped, executing nothing — a safe way to inspect the plan before letting commands touch the machine. Retry with backoff: --retry N re-attempts a failing/timeout step up to N extra times within the same run — a transient hotspot drop often succeeds on a second try. Between attempts the runner waits --backoff 2^ attempt-1 seconds default base 1.0 s . Only the final attempt's result is persisted; the state file records the attempt count. A step that succeeds but still lacks its declared artifact is also retried. Failure cap: with --continue-on-error , --max-failures N stops the run once N failed/timeout steps have been recorded later steps stay pending and run on the next resume . Without --continue-on-error the run already stops at the first failure. Step logs: --step-log id:path or --logs-dir dir for all steps captures a step's stdout+stderr into a file instead of the terminal. The file is truncated per attempt, so a retried step's log shows its final attempt. Output capture applies to the built-in shell runner; a custom injected runner controls its own I/O. Changed command: if a step's command text changes, it is treated as pending again. Force: --force re-runs everything from scratch fresh state . State: .checkpoint-run-<job .json next to the job or in --state-dir , written atomically temp file + rename immediately after each step exits — a kill/drop never loses completed work. No cloud, no network, no LLM, no secrets. Steps are executed through the shell exactly as you write them subprocess.run shell=True — the CLI prints a reminder of this on every run, and --dry-run previews the plan without executing. You are responsible for the commands you pass. This is a resume orchestrator, not a sandbox. python -m pytest tests/ -v 60 passed 2026-08-19, round 4 — full re-verify on this machine note: the suite invokes python through the shell, so run it with the interpreter on PATH e.g. .venv/bin . checkpoint-run is free and open source MIT . This is a genuine demand test, not a storefront: - If a maintained, supported edition priority bug fixes, team/CI onboarding, or a packaged retry service for unattended jobs at ~US$49 one-time per seat would help you, say so in Discussions https://github.com/agenticaotearoa/checkpoint-run/discussions — start or upvote a thread titled "Paid pilot interest" . - If you'd pay a different amount, that's even more useful: state the number. - Responses decide whether a paid pilot ships. No payment is taken here; nothing is collected, and there is no hidden paywall. Bugs / feature requests → Issues https://github.com/agenticaotearoa/checkpoint-run/issues use the templates: bug report or feature request . Questions, show-and-tell, demand votes → Discussions https://github.com/agenticaotearoa/checkpoint-run/discussions .- Both are watched; the fastest way to shape the roadmap is a Discussion. This product's claims are deliberately low-risk: it records which steps completed and resumes there. It does not assert legal requirements, safety authority, or regulatory compliance — the failure class that killed nz-swms-builder see ../AUDIT-REPORT.md .