{"slug": "it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a", "title": "It Can Die in Its Sleep — Self-Healing launchd Jobs with Multi-Slot Firing and a Done-Marker", "summary": "A developer created a self-healing launchd job pattern for macOS that uses multiple scheduled firing slots and done-markers to ensure a daily Obsidian Vault update completes despite failures from sleep, network loss, or API timeouts. The job fires at 4:55, 8:20, 10:45, and 12:15 each day, with each slot checking a date-stamped marker file; if the day's run already succeeded, the slot exits immediately, otherwise it retries from the last completed step. The approach uses half-markers per sub-step to avoid redoing work, so a failure in step2b (Codex logs) only retries that step on the next slot.", "body_md": "My previous piece, \"[Making a launchd Job Unload Itself](https://zenn.dev/bokuwalily/articles/self-unload-launchd-job),\" built a job that runs exactly once and then unloads itself. This time it's the mirror image: **a pattern designed around the assumption that the job will die mid-run — it fires several slots a day and delivers \"retry until it succeeds, then quit immediately on success.\"**\n\nEvery morning I hand Claude Code the task of updating my Obsidian Vault, and it kept dying partway through — killed by macOS sleep, no network on wake, or a claude timeout. Instead of trying to prevent every failure perfectly, I decided \"it can die overnight as long as it's done by the time I wake up\" was the more realistic goal, and I redesigned around that.\n\nThere are three ways a launchd job fails to run to completion.\n\n`caffeinate -s`\n\nonly works on AC power. On battery, the job freezes the instant you close the lid, and gets reaped by timeout after wake.`git push`\n\nand claude's API calls time out.All three can look like \"the job started, exit code 0,\" so you notice late.\n\nThe fix is simple: **stuff multiple StartCalendarInterval entries into the plist, and at the top of the script check \"if today's run already succeeded, exit 0 immediately.\"**\n\n``` php\n<!-- com.shun.vault-auto-ingest.plist (StartCalendarInterval excerpt) -->\n<key>StartCalendarInterval</key>\n<array>\n    <dict>\n        <key>Hour</key><integer>4</integer><key>Minute</key><integer>55</integer>\n    </dict>\n    <dict>\n        <key>Hour</key><integer>8</integer><key>Minute</key><integer>20</integer>\n    </dict>\n    <dict>\n        <key>Hour</key><integer>10</integer><key>Minute</key><integer>45</integer>\n    </dict>\n    <dict>\n        <key>Hour</key><integer>12</integer><key>Minute</key><integer>15</integer>\n    </dict>\n</array>\n```\n\n4:55 is the ideal start time. If that one succeeds, the later three slots (8:20 / 10:45 / 12:15) all see the done-marker and finish in three seconds.\n\nHere's the check at the top of the script.\n\n```\nTODAY=$(date +%Y%m%d)\nDONE_MARKER=\"$HOME/.claude/logs/.vault-ingest-done-${TODAY}\"\n\n# 0. If today's run already succeeded, exit immediately (catch-up slot no-op — don't dirty the log either)\n[ -f \"$DONE_MARKER\" ] && exit 0\n```\n\nThe done-marker is only `touch`\n\ned when the final step succeeds. A run that dies partway never sets the marker, so the next slot redoes the whole thing.\n\nA single claude ingest can take 25+ minutes. When you retry on the next slot, repeating already-successful steps is wasteful. So I placed a \"half-marker\" per step.\n\n```\nSTEP2A_MARKER=\"$HOME/.claude/logs/.vault-ingest-step2a-claude-${TODAY}\"\nSTEP2B_MARKER=\"$HOME/.claude/logs/.vault-ingest-step2b-codex-${TODAY}\"\nSTEP2_MARKER=\"$HOME/.claude/logs/.vault-ingest-step2-done-${TODAY}\"\n```\n\nThe function that calls the ingest is written like this.\n\n```\ningest_src() {\n  local marker=\"$1\" src=\"$2\" name=\"$3\" to=\"$4\" extra=\"$5\"\n  [ -f \"$marker\" ] && {\n    echo \"[$(date '+%F %T')] step2($name) は本日実施済み — skip\" >> \"$LOG\"\n    return 0\n  }\n  cd \"$VAULT\" && run_to \"$to\" \"$CLAUDE\" -p \"...\" \\\n    --dangerously-skip-permissions >> \"$LOG\" 2>&1 \\\n    && { touch \"$marker\"; echo \"[$(date '+%F %T')] step2($name) 完了\" >> \"$LOG\"; return 0; } \\\n    || { echo \"[$(date '+%F %T')] WARN: step2($name) 失敗/timeout\" >> \"$LOG\"; return 1; }\n}\n\n# 2a: Claudeログ（先に処理）。2b: Codexログ（2aの結果も見て統合）\ningest_src \"$STEP2A_MARKER\" \"$KB/raw/conversations/\"       \"claude\" 1500 \"\"\ningest_src \"$STEP2B_MARKER\" \"$KB/raw/codex-conversations/\" \"codex\"  1500 \"...\"\n\n# 両サブが済んだ時だけ「本日ingest完了」を立てる\n[ -f \"$STEP2A_MARKER\" ] && [ -f \"$STEP2B_MARKER\" ] && touch \"$STEP2_MARKER\"\n```\n\nstep2a (Claude logs) succeeds → step2b (Codex logs) times out → next slot fires → step2a is skipped → only step2b is retried. Once both are done, `STEP2_MARKER`\n\ngoes up and the pipeline moves to the next stage. The half-markers reset naturally the next day thanks to the `${TODAY}`\n\nsuffix.\n\nNote\n\nThe step split came out of a real incident. Trying to process 28h of Claude logs + Codex logs in one shot overran the 40-minute window and timed out three days in a row → hot.md froze (2026-06-11 to 13). Splitting into two per-source runs, each capped at a 25-minute (1500-second) timeout, made it stable.\n\nThe pattern where the script re-executes itself under caffeinate.\n\n```\nif [ -z \"${CAFFEINATED:-}\" ]; then\n  exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash \"$0\" \"$@\"\nfi\n```\n\nBecause `exec`\n\nreplaces the process with itself, caffeinate becomes the process's own parent rather than a child. Putting `CAFFEINATED=1`\n\nin the environment prevents an infinite re-invocation.\n\nWarning\n\n`-s`\n\n(prevent system sleep on AC power) isinactive on battery. Close the lid on battery and it freezes. That can't be fully prevented, so the design is \"a frozen run gets reaped by timeout and the next slot redoes it.\"`-i`\n\n(prevent idle sleep) works regardless of power source, but doesn't stop lid-close.\n\nMove right after wake and WiFi isn't up yet. `git push`\n\nand claude's API calls fail immediately.\n\n```\nnet_ok=\"\"\nfor _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do\n  if /usr/bin/nc -z -G 3 1.1.1.1 443 2>/dev/null; then net_ok=1; break; fi\n  sleep 5\ndone\n[ -z \"$net_ok\" ] && echo \"[$(date '+%F %T')] WARN: 網未接続のまま続行（失敗時は次スロットが再試行）\" >> \"$LOG\"\n```\n\n18 iterations × 5 seconds = up to 90 seconds of waiting. It breaks as soon as it confirms a connection. If it still isn't connected after 90 seconds, it just emits a warning and continues. It doesn't exit here because in an offline environment step1 (local log conversion) alone can still run. If the later `git push`\n\nor claude fails, the next slot retries.\n\nThe `-G`\n\nin `nc -z -G 3`\n\nspecifies macOS's connect timeout (in seconds); note that GNU nc uses different flags.\n\nUnder launchd, `~/Documents/`\n\nis protected by TCC (Full Disk Access): unless you grant `/bin/bash`\n\nfull disk access from the GUI, you can't read or write it. If it silently ends with exit 0, you won't notice for a week. Detect it explicitly at the top.\n\n```\nif ! ( cd \"$VAULT\" 2>/dev/null && git rev-parse --git-dir >/dev/null 2>&1 ); then\n  echo \"[$(date '+%F %T')] ❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)。\" >> \"$LOG\"\n  echo \"    解決: システム設定 > プライバシーとセキュリティ > フルディスクアクセス で /bin/bash を許可。\" >> \"$LOG\"\n  notify_fail \"FDA未付与: vault にアクセス不可（設定→フルディスクアクセス→/bin/bash）\"\n  exit 1\nfi\n```\n\nWhether `git rev-parse --git-dir`\n\nsucceeds tells you whether you have access to the Vault. On failure it `exit 1`\n\ns (without setting the done-marker) → the next slot retries → once FDA is granted it resolves naturally.\n\nNote\n\niCloud-synced folders (under`~/Documents/`\n\n) are especially heavily protected. Escaping to`~/`\n\nvia a symlink is a no-go, because iCloud's conflict handling breaks the symlink (verified 2026-06-01). Keep the Vault directly under`~/Documents/`\n\n.\n\nA failure you only notice if you go read the log gets ignored. On failure, drop a `FAILED-${TODAY}.md`\n\non the desktop, and auto-delete it on success.\n\n```\nnotify_fail() {\n  local reason=\"$1\"\n  mkdir -p \"$HOME/Desktop/Daily Brief\"\n  { echo \"# Daily Brief 生成失敗 — $(date '+%F %T')\"\n    echo \"- 理由: ${reason}\"\n    echo \"- 詳細ログ: ~/.claude/logs/vault-auto-ingest.log\"\n    echo \"- 自動再試行: 8:20 / 10:45 / 12:15（成功したらこのファイルは自動で消える）\"\n  } > \"$FAILED_FILE\"\n  /usr/bin/osascript -e \"display notification \\\"${reason}\\\" with title \\\"Daily Brief 生成失敗\\\"\" >/dev/null 2>&1\n}\n```\n\nOnly when step2.6 (archiving the brief) succeeds does it `rm -f \"$FAILED_FILE\"`\n\n. \"The file disappears only on a day that succeeded\" = \"if the file is still there, it's still failing.\"\n\nA race where the launchd version and a manual version ran at the same time actually happened (2026-06-10). Exclude it with an `mkdir`\n\n-based lock.\n\n```\nLOCKDIR=\"$HOME/.claude/locks/vault-auto-ingest.lock\"\nif ! /bin/mkdir \"$LOCKDIR\" 2>/dev/null; then\n  oldpid=$(cat \"$LOCKDIR/pid\" 2>/dev/null || true)\n  if [ -n \"${oldpid:-}\" ] && kill -0 \"$oldpid\" 2>/dev/null; then\n    echo \"[$(date '+%F %T')] 別インスタンス実行中(pid=${oldpid}) — skip\" >> \"$LOG\"\n    exit 0\n  fi\n  rm -rf \"$LOCKDIR\"\n  /bin/mkdir \"$LOCKDIR\" 2>/dev/null || exit 0\nfi\necho $$ > \"$LOCKDIR/pid\"\ntrap 'rm -rf \"$LOCKDIR\"' EXIT INT TERM\n```\n\nA stale lock (a lock dir left behind by a dead process) is auto-reclaimed via a pid liveness check (`kill -0`\n\n). Because `mkdir`\n\nis atomic, even if multiple processes hit it simultaneously, only one succeeds.\n\n`-i`\n\nalone doesn't stop lid-close.`git rev-parse`\n\npreflight. A silent failure went unnoticed for a week.`stat -f%z`\n\nand rotate to `.old`\n\nwhen it's over (dumping claude's entire output into the log makes it bloat surprisingly fast).`TIMEOUT_BIN=/opt/homebrew/bin/timeout`\n\nfirst, and pass through if absent (`brew install coreutils`\n\nis a prerequisite).`StartCalendarInterval`\n\nentries in the plist, and `caffeinate -i -s`\n\nisn't a silver bullet. It can't prevent a battery lid-close freeze, so design it so the next slot takes over a frozen run.`~/Documents/`\n\n. Confirm access from launchd with a `git rev-parse`\n\npreflight to prevent silent failures.Combined with the previous piece, \"[Making a launchd Job Unload Itself](https://zenn.dev/bokuwalily/articles/self-unload-launchd-job),\" you now have two patterns: \"a job you want to run to completion exactly once\" and \"a job you want to reliably complete once a day.\" Next time I plan to write about how to efficiently inject the Vault this job keeps building up into Claude Code's context.\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a", "canonical_source": "https://dev.to/bokuwalily/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a-done-marker-3nn9", "published_at": "2026-07-19 00:00:05+00:00", "updated_at": "2026-07-19 00:30:05.386805+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Claude Code", "Obsidian", "launchd", "macOS"], "alternates": {"html": "https://wpnews.pro/news/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a", "markdown": "https://wpnews.pro/news/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a.md", "text": "https://wpnews.pro/news/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a.txt", "jsonld": "https://wpnews.pro/news/it-can-die-in-its-sleep-self-healing-launchd-jobs-with-multi-slot-firing-and-a.jsonld"}}