Rescuing a Long-Running Job with 'Sub-Step Idempotency' — How vault-auto-ingest Escaped Timeout Hell

This is a continuation of my "Claude Code environment" series. Last time I wrote about making Google Calendar the ground truth for my Vault. This time it's about the timeout hell that hit the same vault-auto-ingest

job — and how I climbed out of it with "sub-step idempotency."

When hot.md failed to update three mornings in a row, the cause wasn't "launchd never fired" — it was "launchd fired but the job died partway through." I'd already put the outer retry in place (firing across multiple slots). The problem was on the inside.

step2 of vault-auto-ingest.sh

is the step that digests the last 28 hours of conversation logs and updates the Vault's wiki/. During busy periods, both Claude Code logs and Codex logs would pile up heavily in this step, and once you added the full-article rewrites, it stopped fitting inside the 40-minute window.

The plist has four slots: 4:55 / 8:20 / 10:45 / 12:15. This is the outer retry — a design where, if today's success marker (DONE_MARKER

) is present, later slots are skipped. But the premise for DONE_MARKER being set is that step2 passes — and since that step2 timed out every single time, the marker was never set, and all four slots kept failing in a row.

Note

The outer retry (multiple slots) is a mechanism for "getting it to succeed somewhere today"; it has no ability to "preserve the part that succeeded partway." That's exactly why an inner resume mechanism is needed.

The fix was to split step2 into two — "Claude log digestion" and "Codex log digestion" — and give each of them its own independent marker file.

STEP2_MARKER="$HOME/.claude/logs/.vault-ingest-step2-done-${TODAY}"
STEP2A_MARKER="$HOME/.claude/logs/.vault-ingest-step2a-claude-${TODAY}"
STEP2B_MARKER="$HOME/.claude/logs/.vault-ingest-step2b-codex-${TODAY}"

The markers form a three-layer structure.

And these three layers are handled uniformly through the ingest_src()

function.

ingest_src() {  # $1=マーカー $2=ソースdir $3=ソース名 $4=timeout秒 $5=追加指示
  local marker="$1" src="$2" name="$3" to="$4" extra="$5"
  [ -f "$marker" ] && { echo "[$(date '+%F %T')] step2($name) は本日実施済み — skip" >> "$LOG"; return 0; }
  cd "$VAULT" && run_to "$to" "$CLAUDE" -p \
"... ${src} に直近28時間で追加・更新されたファイル（${name}由来の会話ログ）だけを対象に ... ${extra}" \
    --dangerously-skip-permissions >> "$LOG" 2>&1 \
    && { touch "$marker"; echo "[$(date '+%F %T')] step2($name) 完了" >> "$LOG"; return 0; } \
    || { echo "[$(date '+%F %T')] WARN: step2($name) 失敗/timeout（次スロットで再試行）" >> "$LOG"; return 1; }
}

The crux is the second line: ** [ -f "$marker" ] && return 0**. If the marker exists, return immediately; if not, run

claude -p

and set the marker. That alone gives you the idempotency of "don't repeat an already-completed sub-step on a same-day retry."The calls look like this.

ingest_src "$STEP2A_MARKER" "$KB/raw/conversations/"       "claude" 1500 ""
ingest_src "$STEP2B_MARKER" "$KB/raw/codex-conversations/" "codex"  1500 "Codex由来でも既存記事に統合し、Claude側と重複する話題は新記事を作らず追記でまとめろ。"
[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"

Each timeout is 1500 seconds (25 minutes). By changing the original "40-minute window for both combined" into "25 minutes each × 2 runs, independent," even if one times out, only that one gets retried in the next slot.

There's a spectrum of choices from "one marker for everything" to "one marker per step." My criterion this time was this:

The boundary for drawing a marker = "a unit of work that can be independently re-run."

Claude logs and Codex logs are different sources; when one finishes, the other can be processed independently of it. That's the condition for being an independent unit. Conversely, "the final reconciliation of hot.md" is work that should be done once, looking at the results of both — so the right form is to place it once, independently, after the two ingests.

Splitting too finely raises management cost (marker files proliferate, and they're hard to follow when debugging). This time I used "source type" as the granularity, but if digesting a single source grew to take over 30 minutes, subdividing into "source × time window" would be the next move.

[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"

Setting STEP2_MARKER

with an AND condition is important — you must not use OR. With OR, you fall into the vicious cycle where "even on a day the Codex logs timed out and weren't digested, step2 is treated as complete, and the next day's Codex log volume swells further."

If all four of the day's slots finish with STEP2_MARKER

still unset, DONE_MARKER won't be set either (just before setting DONE_MARKER there's a freshness check on today-brief.md

, and if the ingest is incomplete the brief stays stale → the design deliberately doesn't write DONE_MARKER due to insufficient freshness). The next day's 4:55 slot doesn't run from scratch — only the incomplete ones among STEP2A/B run. This is the behavior of "inner resume."

Note

Only when multi-slot-launchd-retry (the outer retry) and substep-idempotent-marker (the inner resume) are combined is "it will definitely complete somewhere today" guaranteed. With only one of the two, all four slots can still fail.

STEP2_MARKER

is already set, it bails out before the two sub-steps' skip checksSTEP2_MARKER

check outside (above) the ingest_src

calls, returning early at the entrance.TODAY=$(date +%Y%m%d)

, but for a run that crosses midnight it would write to the next day's markerTODAY

is determined just once at the top of the script, so in practice it doesn't cross over. Still, you'll get bitten if you're not aware of it.raw/codex-conversations/

) is empty, ingest_src runs with nothing to do, exits normally, and STEP2B_MARKER gets setingest_src()

function encloses "if the marker exists skip, otherwise run and set the marker" into a single function.Next time: the receiving side of the ingest results — the problem where hot.md's contents grew too large and started to pressure Claude's context, and a design that uses Obsidian's [[WikiLink]]

structure to resolve references lazily.

*Written by Lily — I ship iOS apps and automate my content stack with Claude Code.

Follow along: Portfolio · X · GitHub*