{"slug": "is-your-24-7-automation-actually-alive-checking-it-with-a-single-command", "title": "Is Your 24/7 Automation Actually Alive? Checking It With a Single Command", "summary": "A developer created a single-command health check script, automation-health.sh, to monitor unattended automation jobs that can silently fail. The script inspects launchd jobs, hooks, and logs, returning a GREEN/WARN/FAIL verdict to catch issues like crashed processes or missing execute permissions before they persist unnoticed.", "body_md": "In [my previous autopilot article](https://zenn.dev/bokuwalily/articles/claude-autopilot), I walked through a setup that runs Claude unattended via launchd. This time I want to talk about what happens behind the curtain — **the problem where an unattended job quietly dies and nobody ever notices** — and how to deal with it.\n\nThe truly scary part of unattended automation isn't bugs; it's **silent failure**. A launchd job can crash with exit 78, a hooks script can lose its execute permission, or agentmemory can run in a \"half-alive\" state with the port open but no worker behind it — and on the surface, nothing looks wrong. There's a real case documented in the comments of `automation-health.sh`\n\nwhere exactly this state persisted for six days with nobody noticing.\n\nOnce you build autonomous loops, the number of things to monitor grows. Even in just my own setup:\n\nI have no desire to check by hand whether every one of these is healthy. But hammering `launchctl list`\n\nindividually doesn't give me the \"big picture\" either.\n\nWhat I need is **a single command that inspects all my automation at once and reports back with a three-value verdict: GREEN / WARN / FAIL**.\n\nThe core of `~/.claude/scripts/automation-health.sh`\n\nis simple.\n\n```\nfail=0; warn=0\nok()  { printf \"  ${GRN}✓${RST} %s\\n\" \"$1\"; }\nwn()  { printf \"  ${YEL}⚠${RST} %s\\n\" \"$1\"; warn=$((warn+1)); }\nng()  { printf \"  ${RED}✗${RST} %s\\n\" \"$1\"; fail=$((fail+1)); }\n```\n\nEach check is evaluated with one of `ok`\n\n/ `wn`\n\n/ `ng`\n\n, and at the very end the verdict is decided like this:\n\n```\nif   [ \"$fail\" -gt 0 ]; then\n  printf \"  ${RED}✗ FAIL${RST}  red=%d warn=%d\\n\\n\" \"$fail\" \"$warn\"; exit 1\nelif [ \"$warn\" -gt 0 ]; then\n  printf \"  ${YEL}⚠ WARN${RST}  warn=%d (致命的問題なし)\\n\\n\" \"$warn\"; exit 0\nelse\n  printf \"  ${GRN}✓ ALL GREEN${RST}  全自動化が正常稼働\\n\\n\"; exit 0\nfi\n```\n\nBy separating the exit codes, you can chain follow-up processing from `daily-brief.sh`\n\nor CI with `||`\n\n.\n\nThe script is divided into nine sections. Here are the main checks, taken straight from the real code.\n\n```\nfor job in com.shun.skill-harvest com.shun.skill-curate; do\n  line=$(launchctl list 2>/dev/null | grep -E \"\\b${job}\\b\")\n  if [ -z \"$line\" ]; then\n    ng \"$job: 未ロード (launchctl load し直しが必要)\"\n  else\n    exitc=$(echo \"$line\" | awk '{print $2}')\n    if [ \"$exitc\" = \"0\" ] || [ \"$exitc\" = \"-\" ]; then\n      ok \"$job: ロード済 / last exit=$exitc\"\n    else\n      ng \"$job: last exit=$exitc (前回失敗)\"\n    fi\n  fi\ndone\n```\n\nThe second column of `launchctl list`\n\nis the last exit code. `-`\n\nmeans \"hasn't run yet (normal)\", `0`\n\nmeans success, and anything else means failure.\n\n```\nhooks=(pre_git_guard.sh pre_secrets_check.sh pre_env_guard.sh \\\n       post_audit_log.sh post_format.sh post_tsc_check.sh \\\n       stop_notify.sh user_prompt_submit.sh)\nfor h in \"${hooks[@]}\"; do\n  f=\"$CLAUDE/hooks/$h\"\n  if   [ ! -f \"$f\" ]; then ng \"$h: 不在\"\n  elif [ ! -x \"$f\" ]; then wn \"$h: 実行権限なし (chmod +x 推奨)\"\n  else ok \"$h\"\n  fi\ndone\n```\n\nEven if you wire a hook into `settings.json`\n\n, if the underlying script isn't `chmod +x`\n\n'd it gets silently skipped. This detects that case as a WARN.\n\n```\nhlog=\"$CLAUDE/skills/auto/.harvest.log\"\na=$(age_h \"$hlog\")   # (now - mtime) / 3600 を返すヘルパ\nif [ \"$a\" -ge 0 ] && [ \"$a\" -le 48 ]; then\n  ok \"最終稼働 ${a}h前: ${last##*] }\"\nelse\n  wn \"最終稼働 ${a}h前 (>48h: cron停止の疑い): ${last##*] }\"\nfi\nnskills=$(find \"$CLAUDE/skills/auto\" -maxdepth 2 -name SKILL.md | wc -l | tr -d ' ')\nok \"生成済 auto-skill: ${nskills} 個\"\n```\n\nIf the log's mtime is older than 48h, it's a WARN. \"A log exists\" and \"it ran recently\" are two different things, so I check the mtime.\n\n```\ncnt=$(find \"$CONV\" -name '*.md' ! -name 'INDEX.md' | wc -l | tr -d ' ')\nok \"会話ログ ${cnt} 件 / 最新 ${la}h前\"\nia=$(age_h \"$idx\")\nif [ \"$ia\" -le 24 ]; then ok \"INDEX.md 鮮度 ${ia}h前\"\nelse wn \"INDEX.md が ${ia}h前 (extract が更新していない可能性)\"; fi\nif [ -f \"$hot\" ] && grep -q \"recent:start auto-updated\" \"$hot\"; then\n  ha=$(age_h \"$hot\"); ok \"hot.md 自動更新マーカー有 / ${ha}h前\"\nelse wn \"hot.md の自動更新マーカーが見つからない\"; fi\n# index.md カバレッジ: 実ファイル数 vs 記載ページ数\nreal=$(find \"$VAULT\" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')\nstated=$(grep -oE '総ページ数：[0-9]+' \"$vidx\" | grep -oE '[0-9]+' | head -1)\nif [ \"$stated\" = \"$real\" ]; then ok \"index.md カバレッジ一致 (${real}p)\"\nelse wn \"index.md 記載 ${stated:-?}p ≠ 実 ${real}p\"; fi\nfor f in now.md recent.md archive.md; do\n  [ -f \"$REMEMBER/$f\" ] && ok \"$f 存在\" || wn \"$f 不在 (必須層)\"\ndone\ndup=$(grep -vE '^\\s*$|^##|^#' \"$nowf\" | sort | uniq -c | sort -rn | head -1 | awk '{print $1}')\nif [ \"${dup:-0}\" -ge 3 ]; then\n  wn \"now.md に同一要約が ${dup} 回 (consolidate 遅延の兆候)\"\nelse\n  ok \"now.md 重複なし (consolidate 健全)\"\nfi\n```\n\nWhen consolidation falls behind, `now.md`\n\ngets the same summary appended over and over. Three or more duplicates trigger an early-warning WARN.\n\n```\ncdir_mb=$(( ${cdir_total_mb:-0} - ${disabled_mb:-0} ))\nif   [ \"${cdir_mb:-0}\" -lt 2000 ]; then ok \"~/.claude 実効 ${cdir_mb}MB\"\nelif [ \"${cdir_mb:-0}\" -lt 5000 ]; then wn \"~/.claude 実効 ${cdir_mb}MB (>2GB: 大きめ)\"\nelse ng \"~/.claude 実効 ${cdir_mb}MB (>5GB: 異常肥大)\"; fi\norphan=$(find \"$CLAUDE\" -maxdepth 1 -name 'security_warnings_state_*.json' -mtime +7 | wc -l | tr -d ' ')\nif [ \"${orphan:-0}\" -ge 5 ]; then\n  wn \"security_warnings_state_*.json が ${orphan} 個 (>7日前: backups/ へ退避推奨)\"\nfi\n```\n\nThe reversible cache (`.disabled-cache`\n\n) is excluded from the effective size before judging.\n\n```\ndeclare -a CRON_JOBS=(\n  \"weekly cleanup-misc:$CLAUDE/logs/cleanup-misc.log:192\"\n  \"weekly env-audit:$CLAUDE/logs/env-audit-latest.md:192\"\n  \"monthly plugin-purge:$CLAUDE/logs/plugin-purge.log:744\"\n  \"weekly plugin-auto-disable:$CLAUDE/logs/plugin-auto-disable.log:192\"\n  \"weekly dotfiles-snapshot:$CLAUDE/logs/dotfiles-snapshot.log:192\"\n  \"weekly agents-index:$CLAUDE/logs/agents-index.log:192\"\n  \"daily plugin-usage:$CLAUDE/scripts/plugin-audit-latest.md:48\"\n)\nfor entry in \"${CRON_JOBS[@]}\"; do\n  IFS=\":\" read -r name path budget <<< \"$entry\"\n  a=$(age_h \"$path\")\n  if [ \"$a\" -le \"$budget\" ]; then ok \"$name: ${a}h前 (budget ${budget}h)\"\n  else wn \"$name: ${a}h前 (budget ${budget}h 超過 — launchd 配送失敗の可能性)\"; fi\ndone\n```\n\nFor a weekly job the budget is 192h (8 days), for a monthly one it's 744h (31 days), and anything over budget is a WARN.\n\nOnly the agentmemory server check uses a two-stage verdict.\n\n``` php\nam_line=$(launchctl list 2>/dev/null | awk '$3==\"com.shun.agentmemory\"{print $1\" \"$2}')\nam_pid=${am_line%% *}; am_exit=${am_line##* }\nif [ -z \"$am_line\" ]; then\n  wn \"com.shun.agentmemory が launchd に未ロード\"\nelif [ \"$am_pid\" = \"-\" ] && [ \"$am_exit\" != \"0\" ]; then\n  ng \"com.shun.agentmemory 停止中 (last exit=$am_exit) — クラッシュループの可能性\"\nelse\n  am_code=$(curl -s -o /dev/null -w '%{http_code}' -m 3 \\\n            http://localhost:3111/agentmemory/health 2>/dev/null || echo 000)\n  if   [ \"$am_code\" = \"200\" ]; then ok \"稼働中 (pid=$am_pid / health 200)\"\n  elif [ \"$am_code\" = \"000\" ]; then ng \"プロセスは居るが port 3111 無応答\"\n  else ng \"port 3111 は開くが /agentmemory/health=$am_code — worker 不在の半生状態\"; fi\nfi\n```\n\nThe reason is written in a comment.\n\nA launchd exit 78 crash loop went unnoticed by anyone for six days, and on top of that the \"port is open but there's no worker, so every API returns 404\" half-alive state is invisible to liveness monitoring.\n\nA launchd status check alone cannot detect the \"the process is there but nothing actually works\" case. That's exactly why we confirm HTTP 200 with `curl`\n\n.\n\nNote\n\nUnless you look atbothlaunchd's PID check and the HTTP health endpoint, you'll miss the half-alive state where \"it looks like it started but every API returns 404.\" If you run an MCP server or API server under launchd, always add the extra step of hitting a`/health`\n\nendpoint.\n\nAn easy-to-miss item is the ninth check: double-execution detection.\n\n```\ncron_sh=$(crontab -l 2>/dev/null | grep -vE '^[[:space:]]*#' | \\\n          grep -oE '/[^ ]+\\.sh' | xargs -n1 basename | sort -u)\nlaunchd_sh=$(grep -hoE '/[^<> ]+\\.sh' \\\n             ~/Library/LaunchAgents/com.shun.*.plist | xargs -n1 basename | sort -u)\ndup=$(comm -12 <(printf '%s\\n' \"$cron_sh\") <(printf '%s\\n' \"$launchd_sh\") | \\\n      grep -vE '^[[:space:]]*$')\nif [ -n \"$dup\" ]; then\n  ng \"cron と launchd の両方に登録され二重実行されるスクリプト: $(printf '%s' \"$dup\" | tr '\\n' ' ')\"\nfi\n```\n\nThe comment reads: \"2026-06-01: while migrating from cron to launchd, I forgot to remove the cron entry, so scripts ended up registered in both = double execution every cycle.\" It's the classic pattern of forgetting to clear the old crontab after a migration.\n\n`daily-brief.sh`\n\nis launched by launchd every morning at 8:00, and it embeds a single line about environment health at the top.\n\n```\necho \"## 🏥 環境ヘルス\"\nrun_to 60 ~/.claude/scripts/automation-health.sh 2>&1 | tail -3 | head -1\n```\n\n`tail -3 | head -1`\n\nslices out just the final summary line (the `✓ ALL GREEN`\n\n/ `⚠ WARN`\n\n/ `✗ FAIL`\n\nline).\n\nFurthermore, on the `daily-brief.sh`\n\nside, a `## 🚨 Action needed`\n\nsection appears **only when a live-connectivity probe detects RED**.\n\n```\nRED_FLAGS=\"\"\nflag_red() { RED_FLAGS=\"${RED_FLAGS}\\n- ❌ $1\"; }\n\nif [ -n \"$RED_FLAGS\" ]; then\n  echo \"## 🚨 要対応（ライブ疎通でRED）\"\n  printf '%b\\n' \"$RED_FLAGS\"\nfi\n```\n\nWhen there's no problem, this section doesn't exist. It's designed as **\"a section that only appears while something is failing.\"** The brief is written out to `~/Desktop/Daily Brief/today-brief-YYYYMMDD.md`\n\n, so when you open your desktop, the RED lands right in front of your eyes.\n\nUsing `automation-health.sh`\n\n's exit 1, you can add the same pattern to a launchd wrapper.\n\n```\nbash ~/.claude/scripts/automation-health.sh || touch ~/Desktop/AUTOMATION_FAILED.md\nbash ~/.claude/scripts/automation-health.sh && rm -f ~/Desktop/AUTOMATION_FAILED.md\n```\n\nThe file exists only while something is failing, and disappears once it's fixed — so even without looking at a dashboard, your desktop tells you the state.\n\n`launchctl list`\n\n's last exit codes `0`\n\nand `-`\n\nthe same, you get false WARNs`chmod +x`\n\n'd produces no error from settings.json`/health`\n\n`scripts/`\n\nto `logs/`\n\n, the path on the health-check side needed updating too`.backup`\n\n, you get a permanent WARN`-not -path '*/.*'`\n\n`comm -12`\n\n`exit 1`\n\nas a machine-readable signal, and hook it into a one-line embed in daily-brief or into generating a FAILED fileNext time, I plan to write about a weekly-report auto-generation mechanism that integrates this health check with the improvement logs autopilot produces, summarizing what changed over the week.\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/is-your-24-7-automation-actually-alive-checking-it-with-a-single-command", "canonical_source": "https://dev.to/bokuwalily/is-your-247-automation-actually-alive-checking-it-with-a-single-command-37p2", "published_at": "2026-07-13 11:00:04+00:00", "updated_at": "2026-07-13 11:15:06.713202+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-infrastructure"], "entities": ["Claude", "launchd", "automation-health.sh"], "alternates": {"html": "https://wpnews.pro/news/is-your-24-7-automation-actually-alive-checking-it-with-a-single-command", "markdown": "https://wpnews.pro/news/is-your-24-7-automation-actually-alive-checking-it-with-a-single-command.md", "text": "https://wpnews.pro/news/is-your-24-7-automation-actually-alive-checking-it-with-a-single-command.txt", "jsonld": "https://wpnews.pro/news/is-your-24-7-automation-actually-alive-checking-it-with-a-single-command.jsonld"}}