Checkpoint-and-resume runner for long multi-step jobs on flaky connections.
New market (2026-08-18 round): not an AI wrapper, not a prompt pack, not a narration pack.
Replacement product after the critical audit marked nz-swms-builder
NO-GO (false legal
premise — see ../AUDIT-REPORT.md
). This product makes no legal, safety, or regulatory claims: it is a purely functional developer utility.
Hardened (2026-08-19, round 2): step timeouts (--timeout
/ --step-timeout
),
--continue-on-error
, checksum-verified step completion (--step-artifact
), --dry-run
preview, and a shell-execution safety banner.
Hardened again (2026-08-19, round 3): per-attempt retry with exponential backoff
(--retry N
/ --backoff
), a failure cap for --continue-on-error
(--max-failures
),
per-step output capture (--step-log
/ --logs-dir
), and a GitHub Actions CI workflow
(.github/workflows/ci.yml
). See the flags below.
Rural NZ broadband is a documented crisis (Federated Farmers: "rural connectivity crisis demands urgent action" — scoop.co.nz; "Northland farmers losing time and money to poor internet" — ruralnewsgroup). On a dropped hotspot connection, long jobs (LLM batch, video render, big download) die mid-run and restart from zero. This machine's own AGENTS.md notes: "drops kill long API runs."
Buyer: NZ rural/dev users running long jobs; any developer with flaky local networking.Value: split a long job into named steps (id:command
); completed steps are persisted to a local JSON state file; re-runresumes where the job died instead of starting over.
| Path | Purpose |
|---|---|
checkpoint_runner/runner.py |
|
| Core: state file (atomic writes), merge/resume logic, injectable shell runner, timeout watchdog, artifact checksums, retry/backoff loop, per-step log capture | |
checkpoint_runner/cli.py |
|
CLI entry point (checkpoint-run ) |
|
tests/test_runner.py |
|
| 60 tests: completion, resume-skips-completed, failure handling, reset, force, determinism, CLI end-to-end, timeouts, continue-on-error, artifact checksums, dry-run, retry/backoff, max-failures, step logs | |
sample_output/ |
|
| Example state files + run logs (round 1 demo, round 2 hardening demo, round 3 retry/logs demo) |
cd checkpoint-run
uv run python -m checkpoint_runner.cli \
--job render-batch \
--state-dir . \
--step download:"curl -L -o model.bin https://example.com/model.bin" \
--step convert:"ffmpeg -i in.mov out.mp4" \
--step upload:"rsync -P out.mp4 backup:render/"
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --status
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --reset
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" --dry-run
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" \
--timeout 3600 --step-timeout download:600
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step a:"cmd" --step b:"cmd2" --continue-on-error
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step render:"ffmpeg -i in.mov out.mp4" --step-artifact render:out.mp4
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step download:"curl -L -o model.bin URL" --retry 2 --backoff 1.0
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step a:"cmd" --step b:"cmd2" --continue-on-error --max-failures 2
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
--step render:"ffmpeg -i in.mov out.mp4" --logs-dir logs
Or install it (from the v0.1.0 release):
python -m venv .venv && .venv/bin/pip install \
https://github.com/agenticaotearoa/checkpoint-run/releases/download/v0.1.0/checkpoint_run-0.1.0-py3-none-any.whl
checkpoint-run --job demo --step "a:echo hi" --step "b:echo bye"
No runtime dependencies: the wheel is pure Python (3.10+) on the standard library.
Resume: re-running skips steps alreadydone
(same id + same command) and starts at the first incomplete step.Failure: a failing step is recorded asfailed
(exit code stored) and later steps do not run (unless--continue-on-error
); re-running retries from that step.Timeout: with--timeout SECONDS
(or--step-timeout id:SECONDS
), a step that exceeds its budget is killed (default runner) and recorded astimeout
; later steps stop unless--continue-on-error
. Re-running retries the timed-out step. The watchdog only observes asinglebudget; it does not bound grandchild processes (documented limitation).Artifact checksums: with--step-artifact id:PATH
, the step's output file hash (sha256) is recorded on completion. On resume, adone
step is trusted only if the artifact still exists with the same content — missing or changed output (deleted file, restored snapshot, corrupted write) forces the step to re-run. A step that claims success but produces no declared artifact is recordedfailed
(exit -2) and retried.Dry run:--dry-run
prints which steps would run and which would be skipped, executing nothing — a safe way to inspect the plan before letting commands touch the machine.Retry with backoff:--retry N
re-attempts a failing/timeout step up to N extra timeswithin the same run— a transient hotspot drop often succeeds on a second try. Between attempts the runner waits--backoff * 2^(attempt-1)
seconds (default base 1.0 s). Only the final attempt's result is persisted; the state file records the attempt count. A step that succeeds but still lacks its declared artifact is also retried.Failure cap: with--continue-on-error
,--max-failures N
stops the run once N failed/timeout steps have been recorded (later steps staypending
and run on the next resume). Without--continue-on-error
the run already stops at the first failure.Step logs:--step-log id:path
(or--logs-dir dir
for all steps) captures a step's stdout+stderr into a file instead of the terminal. The file is truncated per attempt, so a retried step's log shows its final attempt. (Output capture applies to the built-in shell runner; a custom injected runner controls its own I/O.)Changed command: if a step's command text changes, it is treated as pending again.Force:--force
re-runs everything from scratch (fresh state).State:.checkpoint-run-<job>.json
next to the job (or in--state-dir
), written atomically (temp file + rename) immediately after each step exits — a kill/drop never loses completed work. No cloud, no network, no LLM, no secrets.
Steps are executed through the shell exactly as you write them (subprocess.run(shell=True)
)
— the CLI prints a reminder of this on every run, and --dry-run
previews the plan without executing. You are responsible for the commands you pass. This is a resume orchestrator, not a sandbox.
python -m pytest tests/ -v
checkpoint-run
is free and open source (MIT). This is a genuine demand test, not a storefront:
- If a maintained, supported edition(priority bug fixes, team/CI onboarding, or a packaged retry service for unattended jobs) at ~US$49 one-time per seat would help you, say so inDiscussions— start or upvote a thread titled**"Paid pilot interest"**. - If you'd pay a different amount, that's even more useful: state the number.
- Responses decide whether a paid pilot ships. No payment is taken here; nothing is collected, and there is no hidden paywall.
Bugs / feature requests→Issues(use the templates: bug report or feature request).** Questions, show-and-tell, demand votes**→Discussions.- Both are watched; the fastest way to shape the roadmap is a Discussion.
This product's claims are deliberately low-risk: it records which steps completed and resumes
there. It does not assert legal requirements, safety authority, or regulatory compliance —
the failure class that killed nz-swms-builder
(see ../AUDIT-REPORT.md
).