Is Your 24/7 Automation Actually Alive? Checking It With a Single Command

A developer created a single-command health check script, automation-health.sh, to monitor unattended automation jobs that can silently fail. The script inspects launchd jobs, hooks, and logs, returning a GREEN/WARN/FAIL verdict to catch issues like crashed processes or missing execute permissions before they persist unnoticed.

In my previous autopilot article https://zenn.dev/bokuwalily/articles/claude-autopilot , I walked through a setup that runs Claude unattended via launchd. This time I want to talk about what happens behind the curtain — the problem where an unattended job quietly dies and nobody ever notices — and how to deal with it. The truly scary part of unattended automation isn't bugs; it's silent failure . A launchd job can crash with exit 78, a hooks script can lose its execute permission, or agentmemory can run in a "half-alive" state with the port open but no worker behind it — and on the surface, nothing looks wrong. There's a real case documented in the comments of automation-health.sh where exactly this state persisted for six days with nobody noticing. Once you build autonomous loops, the number of things to monitor grows. Even in just my own setup: I have no desire to check by hand whether every one of these is healthy. But hammering launchctl list individually doesn't give me the "big picture" either. What I need is a single command that inspects all my automation at once and reports back with a three-value verdict: GREEN / WARN / FAIL . The core of ~/.claude/scripts/automation-health.sh is simple. fail=0; warn=0 ok { printf " ${GRN}✓${RST} %s\n" "$1"; } wn { printf " ${YEL}⚠${RST} %s\n" "$1"; warn=$ warn+1 ; } ng { printf " ${RED}✗${RST} %s\n" "$1"; fail=$ fail+1 ; } Each check is evaluated with one of ok / wn / ng , and at the very end the verdict is decided like this: if "$fail" -gt 0 ; then printf " ${RED}✗ FAIL${RST} red=%d warn=%d\n\n" "$fail" "$warn"; exit 1 elif "$warn" -gt 0 ; then printf " ${YEL}⚠ WARN${RST} warn=%d 致命的問題なし \n\n" "$warn"; exit 0 else printf " ${GRN}✓ ALL GREEN${RST} 全自動化が正常稼働\n\n"; exit 0 fi By separating the exit codes, you can chain follow-up processing from daily-brief.sh or CI with || . The script is divided into nine sections. Here are the main checks, taken straight from the real code. for job in com.shun.skill-harvest com.shun.skill-curate; do line=$ launchctl list 2 /dev/null | grep -E "\b${job}\b" if -z "$line" ; then ng "$job: 未ロード launchctl load し直しが必要 " else exitc=$ echo "$line" | awk '{print $2}' if "$exitc" = "0" || "$exitc" = "-" ; then ok "$job: ロード済 / last exit=$exitc" else ng "$job: last exit=$exitc 前回失敗 " fi fi done The second column of launchctl list is the last exit code. - means "hasn't run yet normal ", 0 means success, and anything else means failure. hooks= pre git guard.sh pre secrets check.sh pre env guard.sh \ post audit log.sh post format.sh post tsc check.sh \ stop notify.sh user prompt submit.sh for h in "${hooks @ }"; do f="$CLAUDE/hooks/$h" if -f "$f" ; then ng "$h: 不在" elif -x "$f" ; then wn "$h: 実行権限なし chmod +x 推奨 " else ok "$h" fi done Even if you wire a hook into settings.json , if the underlying script isn't chmod +x 'd it gets silently skipped. This detects that case as a WARN. hlog="$CLAUDE/skills/auto/.harvest.log" a=$ age h "$hlog" now - mtime / 3600 を返すヘルパ if "$a" -ge 0 && "$a" -le 48 ; then ok "最終稼働 ${a}h前: ${last }" else wn "最終稼働 ${a}h前 48h: cron停止の疑い : ${last }" fi nskills=$ find "$CLAUDE/skills/auto" -maxdepth 2 -name SKILL.md | wc -l | tr -d ' ' ok "生成済 auto-skill: ${nskills} 個" If the log's mtime is older than 48h, it's a WARN. "A log exists" and "it ran recently" are two different things, so I check the mtime. cnt=$ find "$CONV" -name ' .md' -name 'INDEX.md' | wc -l | tr -d ' ' ok "会話ログ ${cnt} 件 / 最新 ${la}h前" ia=$ age h "$idx" if "$ia" -le 24 ; then ok "INDEX.md 鮮度 ${ia}h前" else wn "INDEX.md が ${ia}h前 extract が更新していない可能性 "; fi if -f "$hot" && grep -q "recent:start auto-updated" "$hot"; then ha=$ age h "$hot" ; ok "hot.md 自動更新マーカー有 / ${ha}h前" else wn "hot.md の自動更新マーカーが見つからない"; fi index.md カバレッジ: 実ファイル数 vs 記載ページ数 real=$ find "$VAULT" -name ' .md' -not -path ' /. ' | wc -l | tr -d ' ' stated=$ grep -oE '総ページ数： 0-9 +' "$vidx" | grep -oE ' 0-9 +' | head -1 if "$stated" = "$real" ; then ok "index.md カバレッジ一致 ${real}p " else wn "index.md 記載 ${stated:-?}p ≠ 実 ${real}p"; fi for f in now.md recent.md archive.md; do -f "$REMEMBER/$f" && ok "$f 存在" || wn "$f 不在 必須層 " done dup=$ grep -vE '^\s $|^ |^ ' "$nowf" | sort | uniq -c | sort -rn | head -1 | awk '{print $1}' if "${dup:-0}" -ge 3 ; then wn "now.md に同一要約が ${dup} 回 consolidate 遅延の兆候 " else ok "now.md 重複なし consolidate 健全 " fi When consolidation falls behind, now.md gets the same summary appended over and over. Three or more duplicates trigger an early-warning WARN. cdir mb=$ ${cdir total mb:-0} - ${disabled mb:-0} if "${cdir mb:-0}" -lt 2000 ; then ok "~/.claude 実効 ${cdir mb}MB" elif "${cdir mb:-0}" -lt 5000 ; then wn "~/.claude 実効 ${cdir mb}MB 2GB: 大きめ " else ng "~/.claude 実効 ${cdir mb}MB 5GB: 異常肥大 "; fi orphan=$ find "$CLAUDE" -maxdepth 1 -name 'security warnings state .json' -mtime +7 | wc -l | tr -d ' ' if "${orphan:-0}" -ge 5 ; then wn "security warnings state .json が ${orphan} 個 7日前: backups/ へ退避推奨 " fi The reversible cache .disabled-cache is excluded from the effective size before judging. declare -a CRON JOBS= "weekly cleanup-misc:$CLAUDE/logs/cleanup-misc.log:192" "weekly env-audit:$CLAUDE/logs/env-audit-latest.md:192" "monthly plugin-purge:$CLAUDE/logs/plugin-purge.log:744" "weekly plugin-auto-disable:$CLAUDE/logs/plugin-auto-disable.log:192" "weekly dotfiles-snapshot:$CLAUDE/logs/dotfiles-snapshot.log:192" "weekly agents-index:$CLAUDE/logs/agents-index.log:192" "daily plugin-usage:$CLAUDE/scripts/plugin-audit-latest.md:48" for entry in "${CRON JOBS @ }"; do IFS=":" read -r name path budget <<< "$entry" a=$ age h "$path" if "$a" -le "$budget" ; then ok "$name: ${a}h前 budget ${budget}h " else wn "$name: ${a}h前 budget ${budget}h 超過 — launchd 配送失敗の可能性 "; fi done For a weekly job the budget is 192h 8 days , for a monthly one it's 744h 31 days , and anything over budget is a WARN. Only the agentmemory server check uses a two-stage verdict. php am line=$ launchctl list 2 /dev/null | awk '$3=="com.shun.agentmemory"{print $1" "$2}' am pid=${am line%% }; am exit=${am line } if -z "$am line" ; then wn "com.shun.agentmemory が launchd に未ロード" elif "$am pid" = "-" && "$am exit" = "0" ; then ng "com.shun.agentmemory 停止中 last exit=$am exit — クラッシュループの可能性" else am code=$ curl -s -o /dev/null -w '%{http code}' -m 3 \ http://localhost:3111/agentmemory/health 2 /dev/null || echo 000 if "$am code" = "200" ; then ok "稼働中 pid=$am pid / health 200 " elif "$am code" = "000" ; then ng "プロセスは居るが port 3111 無応答" else ng "port 3111 は開くが /agentmemory/health=$am code — worker 不在の半生状態"; fi fi The reason is written in a comment. A launchd exit 78 crash loop went unnoticed by anyone for six days, and on top of that the "port is open but there's no worker, so every API returns 404" half-alive state is invisible to liveness monitoring. A launchd status check alone cannot detect the "the process is there but nothing actually works" case. That's exactly why we confirm HTTP 200 with curl . Note Unless you look atbothlaunchd's PID check and the HTTP health endpoint, you'll miss the half-alive state where "it looks like it started but every API returns 404." If you run an MCP server or API server under launchd, always add the extra step of hitting a /health endpoint. An easy-to-miss item is the ninth check: double-execution detection. cron sh=$ crontab -l 2 /dev/null | grep -vE '^ :space: ' | \ grep -oE '/ ^ +\.sh' | xargs -n1 basename | sort -u launchd sh=$ grep -hoE '/ ^< +\.sh' \ ~/Library/LaunchAgents/com.shun. .plist | xargs -n1 basename | sort -u dup=$ comm -12 < printf '%s\n' "$cron sh" < printf '%s\n' "$launchd sh" | \ grep -vE '^ :space: $' if -n "$dup" ; then ng "cron と launchd の両方に登録され二重実行されるスクリプト: $ printf '%s' "$dup" | tr '\n' ' ' " fi The comment reads: "2026-06-01: while migrating from cron to launchd, I forgot to remove the cron entry, so scripts ended up registered in both = double execution every cycle." It's the classic pattern of forgetting to clear the old crontab after a migration. daily-brief.sh is launched by launchd every morning at 8:00, and it embeds a single line about environment health at the top. echo " 🏥 環境ヘルス" run to 60 ~/.claude/scripts/automation-health.sh 2 &1 | tail -3 | head -1 tail -3 | head -1 slices out just the final summary line the ✓ ALL GREEN / ⚠ WARN / ✗ FAIL line . Furthermore, on the daily-brief.sh side, a 🚨 Action needed section appears only when a live-connectivity probe detects RED . RED FLAGS="" flag red { RED FLAGS="${RED FLAGS}\n- ❌ $1"; } if -n "$RED FLAGS" ; then echo " 🚨 要対応（ライブ疎通でRED）" printf '%b\n' "$RED FLAGS" fi When there's no problem, this section doesn't exist. It's designed as "a section that only appears while something is failing." The brief is written out to ~/Desktop/Daily Brief/today-brief-YYYYMMDD.md , so when you open your desktop, the RED lands right in front of your eyes. Using automation-health.sh 's exit 1, you can add the same pattern to a launchd wrapper. bash ~/.claude/scripts/automation-health.sh || touch ~/Desktop/AUTOMATION FAILED.md bash ~/.claude/scripts/automation-health.sh && rm -f ~/Desktop/AUTOMATION FAILED.md The file exists only while something is failing, and disappears once it's fixed — so even without looking at a dashboard, your desktop tells you the state. launchctl list 's last exit codes 0 and - the same, you get false WARNs chmod +x 'd produces no error from settings.json /health scripts/ to logs/ , the path on the health-check side needed updating too .backup , you get a permanent WARN -not -path ' /. ' comm -12 exit 1 as a machine-readable signal, and hook it into a one-line embed in daily-brief or into generating a FAILED fileNext time, I plan to write about a weekly-report auto-generation mechanism that integrates this health check with the improvement logs autopilot produces, summarizing what changed over the week. Written by Lily — I ship iOS apps and automate my content stack with Claude Code. Follow along: Portfolio https://bokuwalily.com · X https://x.com/bokuwalily · GitHub https://github.com/bokuwalily