{"slug": "let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog", "title": "Let Your AI Fix Its Own Broken Automation: Building an Unattended Watchdog", "summary": "A developer built an unattended watchdog system that lets an AI detect, repair, and verify broken automation scripts, pushing changes to production only when verification passes. The system, called self-repair.sh, uses separate health and verification scripts to prevent the AI from committing broken fixes or secrets. The key design principle is to never trust the AI's self-report and instead run an independent verification step before deploying any changes.", "body_md": "If you ship enough side projects, something in your automation is always quietly on fire — and you usually find out days too late. This is the next installment in my \"automation foundation for mass-producing side projects\" series, following \"How I split Claude Code's memory into four layers\" and \"Running Claude Code and Codex together on one machine.\" This time I'll walk through the design and implementation of an **unattended watchdog that lets an AI detect, repair, and verify broken automation scripts, and pushes changes to production without approval only when verification passes**. Using real code as the anchor, I'll explain how to stack guardrails so the AI runs safely instead of \"rampaging without limits.\"\n\nWhen you mass-produce side projects, the number of always-on automation scripts keeps growing. Auto-posting affiliate articles, scrapers, delivering briefs to Discord, checking the review status of iOS apps. Each runs under launchd or cron, so when one breaks, **days can pass without anyone noticing**.\n\nMonitoring and fixing everything by hand hits its limit quickly. But just \"letting the AI fix it\" carries risks: unintended code rewrites, committing secrets, or reporting a repair as done when it's actually still broken and pushing that to production.\n\nSo I designed `self-repair.sh`\n\n. The idea is simple.\n\n```\nDetect → Claude identifies the cause and fixes it → the watchdog independently runs verify\n     → commit/push/deploy only when it passes\n     → roll back the edits and notify a human if it fails\n```\n\n**Don't trust Claude's self-report** is the key point. Even if Claude says \"I fixed it,\" the watchdog runs the verification script itself, and reflects the change only when it exits 0.\n\nThe files are laid out like this.\n\n```\n~/.claude/self-repair/\n├── registry.tsv          # list of monitored projects\n├── checks/\n│   ├── <slug>-health.sh  # health check (exit 0=healthy, non-0=broken)\n│   └── <slug>-verify.sh  # post-repair verification (exit 0=pass, non-0=fail)\n├── state/\n│   └── <slug>-YYYYMMDD.count  # today's attempt count\n└── (the main script lives at ~/.claude/scripts/self-repair.sh)\n\n~/.claude/logs/\n└── self-repair.log\n```\n\nThe main loop simply reads `registry.tsv`\n\nand calls `process()`\n\nfor each project.\n\n```\nwhile IFS=$'\\t' read -r slug dir mode _rest; do\n  case \"$slug\" in ''|\\#*) continue ;; esac\n  [ -n \"$ONLY\" ] && [ \"$slug\" != \"$ONLY\" ] && continue\n  dir=\"${dir/#\\~/$HOME}\"\n  process \"$slug\" \"$dir\" \"${mode:-full}\"\ndone < \"$REG\"\n```\n\nThe `registry.tsv`\n\nformat is three columns: `slug\\tdir\\tmode`\n\n.\n\n```\n# slug          dir                    mode\naffiliate       ~/dev/affiliate-factory full\nbrief-discord   ~/dev/brief             detect\narticle-pub     ~/dev/article-publisher full\n```\n\n`mode`\n\ncan be `full`\n\n(detect + repair) or `detect`\n\n(detect and notify only). Projects that aren't under git management, or where auto-repair is dangerous, should be set to `detect`\n\n.\n\nThe key is to **completely separate health.sh and verify.sh**.\n\n`health.sh`\n\ndecides \"is it broken right now?\" When it detects a problem it exits non-0 and prints the error context to stdout (this output is used in the repair request to Claude).\n\n``` bash\n# Example: affiliate-factory-health.sh\n#!/usr/bin/env bash\n# Broken if no article was generated in the last 24 hours\nLAST=$(find ~/dev/affiliate-factory/output -name '*.md' -newer ~/dev/affiliate-factory/output -mmin -1440 | wc -l | tr -d ' ')\nif [ \"$LAST\" -eq 0 ]; then\n  echo \"過去24時間の記事生成数=0。generate.sh が失敗している可能性\"\n  exit 1\nfi\n```\n\n`verify.sh`\n\ndecides \"does it actually work after the repair?\" It can be stricter than health.sh, or identical. What matters is that **a process independent of Claude's repair runs it**.\n\n``` bash\n# Example: affiliate-factory-verify.sh\n#!/usr/bin/env bash\n# Pass if it actually generates one article and finishes cleanly\ncd ~/dev/affiliate-factory\ntimeout 120 bash generate.sh --dry-run\n```\n\nA comment at the top of the script says \"the real safety is the guardrails.\" By stacking all seven guards, unattended repair becomes viable.\n\nThe most important one. Inside `process()`\n\n, after calling Claude, **the watchdog itself** runs `bash \"$verify\"`\n\n.\n\n```\n# After calling Claude...\nlocal vrc=0\nrun_capped \"$VERIFY_TIMEOUT\" bash \"$verify\" >/tmp/sr-$slug-verify.log 2>&1 || vrc=$?\n\nif [ \"$vrc\" -eq 0 ]; then\n  # fixed → secret scan → commit → push\nelse\n  # ❌ verification failed → roll back the edits\n  restore \"$dir\" \"$baseline\"\nfi\n```\n\n`run_capped`\n\nis a wrapper that runs with a timeout if `gtimeout`\n\nis available. It prevents verification from running forever.\n\n```\nrun_capped() { # run_capped <timeout_sec> <cmd...>\n  local t=\"$1\"; shift\n  if [ -n \"$TIMEOUT_BIN\" ]; then \"$TIMEOUT_BIN\" \"$t\" \"$@\"; else \"$@\"; fi\n}\n```\n\nIf verify is non-0, `restore()`\n\nreturns to the baseline.\n\n```\nrestore() { # restore <dir> <baseline_sha>\n  local dir=\"$1\" base=\"$2\"\n  (cd \"$dir\" && {\n     git reset -q --hard \"${base:-HEAD}\" 2>/dev/null || true\n     git clean -fdq 2>/dev/null || true\n     git stash list 2>/dev/null | grep -q . && git stash drop -q 2>/dev/null || true\n   }) || true\n}\n```\n\n`git reset --hard`\n\nreturns to the pre-commit state, `git clean -fdq`\n\nremoves files Claude newly added, and anything stashed away is dropped too. This **assumes a git repo**, so non-git projects need to be in `detect`\n\nmode (the script also checks with git rev-parse and skips auto-repair if it's not a git repo).\n\nBefore pushing, it scans the staged diff with regular expressions.\n\n```\nsecret_in_staged() { # secret_in_staged <repo>\n  local repo=\"$1\"\n  if (cd \"$repo\" && git ls-files --cached | grep -qE '(^|/)\\.env$'); then\n    echo \".env がコミット対象\"; return 0\n  fi\n  local hits\n  hits=$(cd \"$repo\" && git diff --cached | grep -nE \\\n    'pk_[A-Za-z0-9]{6,}|sk_live_[A-Za-z0-9]|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{20}|xox[baprs]-[A-Za-z0-9]|-----BEGIN [A-Z ]*PRIVATE KEY-----|AIza[0-9A-Za-z_-]{20}' \\\n    2>/dev/null | head -3)\n  if [ -n \"$hits\" ]; then echo \"$hits\"; return 0; fi\n  return 1\n}\n```\n\nIf `.env`\n\nis under tracking, it's an immediate fail. Beyond that it detects Stripe's `pk_`\n\n/`sk_live_`\n\n, AWS's AKIA, GitHub Personal Access Tokens (`ghp_`\n\n), Slack tokens (`xox`\n\n), private key headers, and Google API keys (`AIza`\n\n).\n\nThis secret scan is built into the script itself because\n\nClaude's hooks don't fire when running under launchd. The point is to scan yourself instead of relying on hooks.\n\nWhat actually scans before commit is `secret_in_staged_after_add()`\n\n. It runs `git add -A`\n\n, then scans for secrets, and if clean, commits as-is.\n\n```\nsecret_in_staged_after_add() {\n  local dir=\"$1\" reason\n  (cd \"$dir\" && git add -A) || true\n  if reason=$(secret_in_staged \"$dir\"); then echo \"$reason\"; return 0; fi\n  (cd \"$dir\" && git commit -q -m \"fix(self-repair): $(date +%F) 自己修復による自動修正\" 2>>\"$LOG\") || true\n  return 1\n}\n```\n\nThe same slug can be attempted up to `MAX_ATTEMPTS`\n\n(default 2) times per day. The state file is used as a counter.\n\n```\nMAX_ATTEMPTS=\"${SELF_REPAIR_MAX_ATTEMPTS:-2}\"\n\nattempts_today() { cat \"$STATE/$1-$TODAY.count\" 2>/dev/null || echo 0; }\nbump_attempts() {\n  local n; n=$(attempts_today \"$1\"); n=$((n+1))\n  echo \"$n\" > \"$STATE/$1-$TODAY.count\"; echo \"$n\"\n}\n```\n\nWhen the cap is reached, it skips repair and notifies via Discord that \"a human needs to check this.\" This prevents a repair loop from spinning forever and melting tokens.\n\nIf the 5-hour block's output tokens exceed the threshold, it skips the entire run cycle.\n\n```\nBUDGET_CAP_TOK=\"${SELF_REPAIR_BUDGET_CAP:-380000}\"\n\nbudget_ok() {\n  local tok\n  tok=$(\"$BUDGET_ADVISOR\" 2>/dev/null | python3 -c 'import sys,json\ntry: print(int(json.load(sys.stdin).get(\"5h_output_tokens\",0)))\nexcept Exception: print(0)' 2>/dev/null)\n  tok=\"${tok:-0}\"\n  if [ \"$tok\" -gt \"$BUDGET_CAP_TOK\" ]; then\n    log \"SKIP(budget): 5h_output_tokens=$tok > cap=$BUDGET_CAP_TOK\"\n    return 1\n  fi\n  return 0\n}\n```\n\n`BUDGET_ADVISOR`\n\nis a separate script that returns Claude Code's usage in JSON format. It's the last line of defense against a billing runaway.\n\nClaude is restricted with `--allowedTools`\n\non which tools it can touch, and made to edit only within the project directory.\n\n```\nout=$(cd \"$dir\" && printf '%s' \"$PROMPT\" | run_capped \"$REPAIR_TIMEOUT\" \"$CLAUDE\" -p \\\n      --model \"$MODEL\" --output-format text \\\n      --allowedTools \"Read,Write,Edit,Bash,Grep,Glob\" \\\n      --max-turns 40 2>&1) || rc=$?\n```\n\nI don't use `--skip-permissions`\n\n. The wall is built by narrowing the tools with `--allowedTools`\n\nand making the scope explicit in the prompt.\n\nIt won't push unless the destination GitHub remote is your own account. This prevents unintentionally writing to a third party's fork.\n\n```\nALLOWED_OWNERS=\"bokuwalily\"\n\npush_if_safe() {\n  local repo=\"$1\" url owner\n  url=$(cd \"$repo\" && git remote get-url origin 2>/dev/null || true)\n  [ -z \"$url\" ] && { log \"  push skip: origin未設定（ローカルcommitのみ）\"; return 0; }\n  owner=$(printf '%s' \"$url\" | sed -E 's#.*github.com[:/]+([^/]+)/.*#\\1#')\n  case \" $ALLOWED_OWNERS \" in\n    *\" $owner \"*) ;;\n    *) log \"  push skip: owner=$owner は許可外（第三者repo保護）\"; return 0 ;;\n  esac\n  if (cd \"$repo\" && git push -q origin HEAD 2>>\"$LOG\"); then\n    log \"  pushed → $owner\"\n  else\n    log \"  push失敗（commitは保持）\"\n  fi\n}\n```\n\nThe prompt to Claude is assembled dynamically inside `process()`\n\n. The key is to pass health.sh's output and recent logs as context, and **explicitly state what must not be done as constraints**.\n\n```\nlocal PROMPT=\"あなたは自律修復エージェントです。自動化『${slug}』(ディレクトリ: ${dir})が壊れています。原因を特定し、**最小の変更**で直してください。\n\n# 制約（厳守）\n- 触ってよいのは $dir 配下のファイルのみ。スコープを広げない。新機能を足さない。\n- .env や秘密情報を読み出して出力・コミットしない。\n- 直したら必ず自分で検証コマンド \\`bash $verify\\` を実行し、exit 0 になることを確認してから終了する。\n- 直せない/原因が $dir 外にあるなら、推測で書き換えず『UNFIXABLE: <理由>』とだけ述べて終了する。\n\n# 健全性チェックの出力\n$hctx\n\n# 最近のログ（末尾）\n$rlog\n\n直して、検証まで通してください。\"\n```\n\nThe important part is the `UNFIXABLE:`\n\nexit keyword. In cases that can't or shouldn't be fixed, it prevents the runaway behavior of \"just rewrite something anyway.\" When the cause is outside `$dir`\n\n(a dependency API outage, a cron config, etc.), the design keeps Claude from touching it.\n\nThe health check output is trimmed to `head -40`\n\n, and the logs to `tail -60`\n\n, before being passed. If the context is too large, the judgment wobbles.\n\nBefore repairing, it records the current HEAD (the baseline). Because uncommitted changes would make the restore dirty, it first stashes them with `git stash -u`\n\n.\n\n```\nlocal baseline; baseline=$(cd \"$dir\" && git rev-parse HEAD 2>/dev/null || echo \"\")\n(cd \"$dir\" && git stash -u -q 2>/dev/null) || true\n```\n\n`-u`\n\nis the option that also stashes untracked files. If Claude fails to repair, `restore()`\n\nreturns to `git reset --hard baseline`\n\nand then drops the stash too.\n\nSetting `mode=detect`\n\nmakes it \"just detect that something is broken and notify,\" without calling Claude. To notify only once per day, it creates a flag file under `state/`\n\n.\n\n```\nif [ \"$mode\" = detect ]; then\n  if [ ! -f \"$STATE/$slug-$TODAY.detected\" ]; then\n    local hd; hd=$(head -3 /tmp/sr-$slug-health.log 2>/dev/null | tr '\\n' ' ' | cut -c1-200)\n    notify alerts \"🩺 $slug がサイレント失敗（本日成功の形跡なし）。自動修復対象外＝人間の確認が要ります。$hd\"\n    touch \"$STATE/$slug-$TODAY.detected\"\n  fi\n  return\nfi\n```\n\nProjects not under git management, projects where repair has side effects (requiring writes to external APIs), and projects with no verify.sh in place are safest left as `detect`\n\n. The script also automatically falls back to `detect`\n\n-equivalent behavior (notify only) when `verify.sh`\n\ndoesn't exist.\n\n```\n[ -x \"$verify\" ] || {\n  log \"$slug: verify無し→無人修復は危険なのでskip(人間へ)\"\n  notify alerts \"🛠 $slug が異常だが verify 未整備のため自動修復せず。要確認。\"\n  return\n}\n```\n\nIn actual operation, it runs periodically via a launchd plist. Because of the minimal-PATH problem, the script explicitly sets PATH internally.\n\n``` php\n<!-- ~/Library/LaunchAgents/com.lily.self-repair.plist -->\n<plist version=\"1.0\">\n<dict>\n  <key>Label</key>\n  <string>com.lily.self-repair</string>\n  <key>ProgramArguments</key>\n  <array>\n    <string>/bin/bash</string>\n    <string>/Users/<you>/.claude/scripts/self-repair.sh</string>\n  </array>\n  <key>StartCalendarInterval</key>\n  <dict>\n    <key>Minute</key>\n    <integer>30</integer>\n  </dict>\n  <key>StandardOutPath</key>\n  <string>/Users/<you>/.claude/logs/self-repair-launchd.log</string>\n  <key>StandardErrorPath</key>\n  <string>/Users/<you>/.claude/logs/self-repair-launchd.err</string>\n</dict>\n</plist>\n```\n\nSetting `StartCalendarInterval`\n\nto `Minute: 30`\n\nruns it at 30 minutes past every hour. A 30-minute cadence is enough in most cases.\n\nBecause\n\n`nvm`\n\nisn't loaded when running via launchd, you need to either pass the full path to the`claude`\n\ncommand via the`CLAUDE`\n\nenvironment variable, or explicitly`export PATH`\n\nat the top of the script.\n\nIf you let Claude repair while there are uncommitted work-in-progress files, untracked files remain even after `git reset --hard`\n\n. Unless you follow the flow of stash → repair → drop on failure, you'll create manual cleanup work for yourself.\n\nIf you use the same script for health.sh and verify.sh, even a situation where \"it happened to pass right after the repair\" gets treated as a pass. verify.sh should include a test that **actually runs the processing**, so that even with `--dry-run`\n\nit exercises the full set of code paths.\n\nRelying on GitHub's push protection means **detection happens after the push**. The point is to scan yourself before pushing. Also, since assignments to environment variables don't get caught by ordinary regexes, I scan `git diff --cached`\n\ndirectly to detect dangerous literals.\n\nmacOS's `timeout`\n\ncommand comes in via Homebrew as GNU coreutils' `gtimeout`\n\n. I check for its existence with `command -v gtimeout`\n\n, and if it's absent I make the wrapper a pass-through. Without `gtimeout`\n\n, if Claude stops responding, launchd keeps piling up the next runs.\n\n```\nTIMEOUT_BIN=\"$(command -v gtimeout 2>/dev/null || true)\"\n[ -z \"$TIMEOUT_BIN\" ] && [ -x /opt/homebrew/bin/gtimeout ] && TIMEOUT_BIN=/opt/homebrew/bin/gtimeout\n```\n\nAs the comments note, this watchdog can only take care of **the ways code breaks in projects managed as git repos**.\n\n`.env`\n\nexpired → Claude can't fix it (out of scope)To let Claude correctly judge \"out of reach\" from the prompt, I explicitly provide the escape hatch: \"when you can't fix it, state only `UNFIXABLE:`\n\nand exit.\"\n\nIf you forget the third column in registry.tsv, `mode`\n\nbecomes empty and falls back to full via `${mode:-full}`\n\n. If you forget it on a project you wanted as `detect`\n\n, unintended auto-repair runs, so always state the mode explicitly when writing the tsv.\n\n`registry.tsv`\n\n, and the watchdog monitors everything fully automatically`verify.sh`\n\n`git reset --hard`\n\n`detect`\n\nmode limited to detect + notify`UNFIXABLE:`\n\nand exit\" in the prompt prevents unnecessary rewritesThe more automation you have, the more the \"cost of detecting and repairing when it breaks\" grows in proportion. Once you set up a watchdog, silent failures drop dramatically, and even when things break in the middle of the night, you wake up to a \"self-repaired it\" message in Discord. The cost of writing `health.sh`\n\nand `verify.sh`\n\nfor every project is a one-time thing, and the running cost is nearly zero.\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog", "canonical_source": "https://dev.to/bokuwalily/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog-dlo", "published_at": "2026-07-21 05:00:01+00:00", "updated_at": "2026-07-21 05:29:29.460702+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["Claude"], "alternates": {"html": "https://wpnews.pro/news/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog", "markdown": "https://wpnews.pro/news/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog.md", "text": "https://wpnews.pro/news/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog.txt", "jsonld": "https://wpnews.pro/news/let-your-ai-fix-its-own-broken-automation-building-an-unattended-watchdog.jsonld"}}