# Checkpoint-run: resume long shell jobs after your connection drops

> Source: <https://github.com/agenticaotearoa/checkpoint-run>
> Published: 2026-08-19 06:12:27+00:00

Checkpoint-and-resume runner for long multi-step jobs on flaky connections.

**New market (2026-08-18 round):** not an AI wrapper, not a prompt pack, not a narration pack.
Replacement product after the critical audit marked `nz-swms-builder`

**NO-GO** (false legal
premise — see `../AUDIT-REPORT.md`

). This product makes **no legal, safety, or regulatory
claims**: it is a purely functional developer utility.

**Hardened (2026-08-19, round 2):** step timeouts (`--timeout`

/ `--step-timeout`

),
`--continue-on-error`

, checksum-verified step completion (`--step-artifact`

), `--dry-run`

preview, and a shell-execution safety banner.

**Hardened again (2026-08-19, round 3):** per-attempt retry with exponential backoff
(`--retry N`

/ `--backoff`

), a failure cap for `--continue-on-error`

(`--max-failures`

),
per-step output capture (`--step-log`

/ `--logs-dir`

), and a GitHub Actions CI workflow
(`.github/workflows/ci.yml`

). See the flags below.

Rural NZ broadband is a documented crisis (Federated Farmers: "rural connectivity crisis
demands urgent action" — [scoop.co.nz](https://www.scoop.co.nz/stories/print.html?path=AK2509/S00830/rural-connectivity-crisis-demands-urgent-action.htm);
"Northland farmers losing time and money to poor internet" — [ruralnewsgroup](https://www.ruralnewsgroup.co.nz/rural-news/rural-general-news/northland-farmers-losing-money-poor-internet)).
On a dropped hotspot connection, long jobs (LLM batch, video render, big download) die mid-run
and restart from zero. This machine's own AGENTS.md notes: "drops kill long API runs."

**Buyer:** NZ rural/dev users running long jobs; any developer with flaky local networking.**Value:** split a long job into named steps (`id:command`

); completed steps are persisted to a local JSON state file; re-run**resumes where the job died** instead of starting over.

| Path | Purpose |
|---|---|
`checkpoint_runner/runner.py` |
Core: state file (atomic writes), merge/resume logic, injectable shell runner, timeout watchdog, artifact checksums, retry/backoff loop, per-step log capture |
`checkpoint_runner/cli.py` |
CLI entry point (`checkpoint-run` ) |
`tests/test_runner.py` |
60 tests: completion, resume-skips-completed, failure handling, reset, force, determinism, CLI end-to-end, timeouts, continue-on-error, artifact checksums, dry-run, retry/backoff, max-failures, step logs |
`sample_output/` |
Example state files + run logs (round 1 demo, round 2 hardening demo, round 3 retry/logs demo) |

```
cd checkpoint-run

# Run a 3-step job; if the hotspot drops mid-run, just re-run the same command.
uv run python -m checkpoint_runner.cli \
  --job render-batch \
  --state-dir . \
  --step download:"curl -L -o model.bin https://example.com/model.bin" \
  --step convert:"ffmpeg -i in.mov out.mp4" \
  --step upload:"rsync -P out.mp4 backup:render/"

# Inspect progress / clear state
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --status
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . --reset

# Preview what would run WITHOUT executing anything (shell-safety check)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" --dry-run

# Per-step time budget (a step exceeding it is recorded 'timeout' and stops the run)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --step render:"ffmpeg -i in.mov out.mp4" \
  --timeout 3600 --step-timeout download:600

# Keep running later steps even if one fails
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step a:"cmd" --step b:"cmd2" --continue-on-error

# Checksum-verified completion: re-runs a 'done' step if its output is missing/changed
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step render:"ffmpeg -i in.mov out.mp4" --step-artifact render:out.mp4

# Retry a flaky step up to 2 extra times with exponential backoff
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step download:"curl -L -o model.bin URL" --retry 2 --backoff 1.0

# Stop after 2 recorded failures even with --continue-on-error
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step a:"cmd" --step b:"cmd2" --continue-on-error --max-failures 2

# Capture every step's stdout+stderr to logs/<id>.log (or --step-log id:path per step)
uv run python -m checkpoint_runner.cli --job render-batch --state-dir . \
  --step render:"ffmpeg -i in.mov out.mp4" --logs-dir logs
```

Or install it (from the [v0.1.0 release](https://github.com/agenticaotearoa/checkpoint-run/releases/tag/v0.1.0)):

```
python -m venv .venv && .venv/bin/pip install \
  https://github.com/agenticaotearoa/checkpoint-run/releases/download/v0.1.0/checkpoint_run-0.1.0-py3-none-any.whl
checkpoint-run --job demo --step "a:echo hi" --step "b:echo bye"
```

No runtime dependencies: the wheel is pure Python (3.10+) on the standard library.

**Resume:** re-running skips steps already`done`

(same id + same command) and starts at the first incomplete step.**Failure:** a failing step is recorded as`failed`

(exit code stored) and later steps do not run (unless`--continue-on-error`

); re-running retries from that step.**Timeout:** with`--timeout SECONDS`

(or`--step-timeout id:SECONDS`

), a step that exceeds its budget is killed (default runner) and recorded as`timeout`

; later steps stop unless`--continue-on-error`

. Re-running retries the timed-out step. The watchdog only observes a*single*budget; it does not bound grandchild processes (documented limitation).**Artifact checksums:** with`--step-artifact id:PATH`

, the step's output file hash (sha256) is recorded on completion. On resume, a`done`

step is trusted only if the artifact still exists with the same content — missing or changed output (deleted file, restored snapshot, corrupted write) forces the step to re-run. A step that claims success but produces no declared artifact is recorded`failed`

(exit -2) and retried.**Dry run:**`--dry-run`

prints which steps would run and which would be skipped, executing nothing — a safe way to inspect the plan before letting commands touch the machine.**Retry with backoff:**`--retry N`

re-attempts a failing/timeout step up to N extra times*within the same run*— a transient hotspot drop often succeeds on a second try. Between attempts the runner waits`--backoff * 2^(attempt-1)`

seconds (default base 1.0 s). Only the final attempt's result is persisted; the state file records the attempt count. A step that succeeds but still lacks its declared artifact is also retried.**Failure cap:** with`--continue-on-error`

,`--max-failures N`

stops the run once N failed/timeout steps have been recorded (later steps stay`pending`

and run on the next resume). Without`--continue-on-error`

the run already stops at the first failure.**Step logs:**`--step-log id:path`

(or`--logs-dir dir`

for all steps) captures a step's stdout+stderr into a file instead of the terminal. The file is truncated per attempt, so a retried step's log shows its final attempt. (Output capture applies to the built-in shell runner; a custom injected runner controls its own I/O.)**Changed command:** if a step's command text changes, it is treated as pending again.**Force:**`--force`

re-runs everything from scratch (fresh state).**State:**`.checkpoint-run-<job>.json`

next to the job (or in`--state-dir`

), written atomically (temp file + rename) immediately after each step exits — a kill/drop never loses completed work. No cloud, no network, no LLM, no secrets.

Steps are executed through the shell **exactly as you write them** (`subprocess.run(shell=True)`

)
— the CLI prints a reminder of this on every run, and `--dry-run`

previews the plan without
executing. You are responsible for the commands you pass. This is a resume orchestrator, not a
sandbox.

```
python -m pytest tests/ -v
# 60 passed  (2026-08-19, round 4 — full re-verify on this machine)
# note: the suite invokes `python` through the shell, so run it with the
# interpreter on PATH (e.g. `.venv/bin`).
```

`checkpoint-run`

is free and open source (MIT). This is a genuine demand test, not a storefront:

- If a
**maintained, supported edition**(priority bug fixes, team/CI onboarding, or a packaged retry service for unattended jobs) at ~US$49 one-time per seat would help you, say so in[Discussions](https://github.com/agenticaotearoa/checkpoint-run/discussions)— start or upvote a thread titled**"Paid pilot interest"**. - If you'd pay a different amount, that's even more useful: state the number.
- Responses decide whether a paid pilot ships. No payment is taken here; nothing is collected, and there is no hidden paywall.

**Bugs / feature requests**→[Issues](https://github.com/agenticaotearoa/checkpoint-run/issues)(use the templates: bug report or feature request).** Questions, show-and-tell, demand votes**→[Discussions](https://github.com/agenticaotearoa/checkpoint-run/discussions).- Both are watched; the fastest way to shape the roadmap is a Discussion.

This product's claims are deliberately low-risk: it records which steps completed and resumes
there. It does not assert legal requirements, safety authority, or regulatory compliance —
the failure class that killed `nz-swms-builder`

(see `../AUDIT-REPORT.md`

).