{"slug": "15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the", "title": "15 launchd Jobs and One Quota Circuit Breaker: Deciding What to Re-run Once the Circuit Closes", "summary": "A developer built quota-catchup.py, a companion tool to the claude-quota-guard.py quota circuit breaker that decides which of 15 launchd jobs to re-run after a quota-exhaustion circuit closes. The script identifies re-run candidates by checking whether a plist is guarded, whether today's latest marker is a SKIPPED entry, and whether a StartCalendarInterval slot has already passed, avoiding duplicate runs and unnecessary catch-up for StartInterval jobs.", "body_md": "A circuit breaker that halts every job the moment your quota runs dry is only half the story. The harder question shows up afterward: once the circuit closes again, what do you do with everything it skipped? In my [previous post I wrote about fixing the Wiki secret-scan sync](https://zenn.dev/bokuwalily/articles/wiki-improvements-secret-scan-sync). This time I'm switching gears and following up on `claude-quota-guard.py`, the **quota circuit breaker** I built. That script's story ended at \"detect quota exhaustion, stop all jobs.\" In real operation, there's a whole second problem waiting past that point—and along the way it involved a 1200-second timeout that turned out to need 2700, and a load average that climbed past 40 when I got it wrong.\n\n`run_job` in `claude-quota-guard.py` returns exit 0 immediately when a job starts while the circuit is open, leaving nothing but a marker in the log.\n\n```\nprint(\n    \"CLAUDE_QUOTA_JOB_SKIPPED \"\n    f\"job={label} reason={status['reason']} remaining={status['remaining_seconds']}s ts={now()}\",\n    file=sys.stderr,\n)\nreturn 0\n```\n\nThis guard sits in front of 15 of the plists under `~/Library/LaunchAgents/*.plist`. As the comment in the code puts it:\n\n```\n# 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める\n# 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。\n```\n\nThe problem is that after the circuit closes and `open_until` has passed, launchd **does nothing until that job's next `StartCalendarInterval` slot comes around**. If the 9:00 AM job was skipped for quota reasons and the quota recovers at 10:00, but the next slot is 9:00 AM tomorrow, you've lost an entire day's execution. Deciding \"what to re-run, and how much, after recovery\" is the job of `quota-catchup.py`.\n\n`quota-catchup.py` walks every plist in `find_candidates` and only treats a job as a re-run candidate if it passes three conditions ANDed together.\n\n| Condition | Function | Purpose | \n|---|---|---|\n| Is this plist guarded? | `is_guarded` | Don't mix in unrelated jobs | \n| Is today's latest marker SKIPPED? | `latest_marker_is_today_skip` | Exclude jobs that already ran, and stale skips | \n| Has a slot already passed? | `calendar_slot_passed` | Exclude `StartInterval` jobs and jobs with only future slots | \n\n``` php\ndef find_candidates(...) -> list[Candidate]:\n    candidates: list[Candidate] = []\n    for plist_path in sorted(launch_agents.glob(\"*.plist\")):\n        plist = load_plist(plist_path)\n        if not plist or not is_guarded(plist):\n            continue\n        ...\n        if label in already_kicked or not latest_marker_is_today_skip(output_paths, label, today):\n            continue\n        if calendar_slot_passed(plist, now):\n            candidates.append(Candidate(label, plist_path))\n    return candidates\n```\n\nRe-run targets are limited to jobs that are \"launched via `claude-quota-guard.py`.\" The check is nothing more than a string match on `ProgramArguments`.\n\n``` php\ndef is_guarded(plist: dict) -> bool:\n    arguments = plist.get(\"ProgramArguments\")\n    return isinstance(arguments, list) and any(\"claude-quota-guard\" in str(value) for value in arguments)\n```\n\nIf you picked up plists that don't go through the guard, you'd end up kicking ordinary cron-style jobs that have nothing to do with the quota.\n\nThis is the star of the show. A single job can have multiple `StartCalendarInterval` slots. A real example is `com.shun.daily-brief.plist`.\n\n```\n<key>StartCalendarInterval</key>\n<array>\n    <dict><key>Hour</key><integer>8</integer><key>Minute</key><integer>0</integer></dict>\n    <dict><key>Hour</key><integer>10</integer><key>Minute</key><integer>30</integer></dict>\n</array>\n```\n\nEven if the 8:00 slot gets skipped for quota, the same job naturally runs again at 10:30. So the rule is \"if even one slot has passed, it's a re-run candidate,\" not \"wait, because there's still a future slot.\" Conversely, `StartInterval` jobs (e.g., every 30 minutes) will naturally re-run on the next interval if you just leave them alone, so there's no need to make them catch-up targets.\n\n``` php\ndef calendar_slot_passed(plist: dict, now: datetime) -> bool:\n    \"\"\"True only for calendar-only jobs with at least one past slot today.\"\"\"\n    if \"StartInterval\" in plist:\n        return False\n    raw_entries = plist.get(\"StartCalendarInterval\")\n    if isinstance(raw_entries, dict):\n        entries = [raw_entries]\n    elif isinstance(raw_entries, list):\n        entries = raw_entries\n    else:\n        return False\n\n    saw_today_slot = False\n    for entry in entries:\n        if not isinstance(entry, dict) or \"Hour\" not in entry:\n            return False\n        if not runs_today(entry, now):\n            continue\n        try:\n            scheduled = now.replace(\n                hour=int(entry[\"Hour\"]),\n                minute=int(entry.get(\"Minute\", 0)),\n                second=0, microsecond=0,\n            )\n        except (TypeError, ValueError):\n            return False\n        if scheduled > now:\n            continue\n        saw_today_slot = True\n    return saw_today_slot\n```\n\nThe key is walking every slot to the end and accumulating `saw_today_slot` with an OR. If you'd written it as \"return the verdict from the first slot you find,\" you'd hit a bug where the result depends on the order of the slots. For example, with the ordering `[{9:00}, {18:00}]` evaluated at 13:00, the correct answer is \"candidate\" because 9:00 is in the past—but if the loop only judged by the last entry, it would misclassify the job as not a candidate on the grounds that 18:00 is in the future.\n\nA re-run goes through `kick_and_wait`, which runs `launchctl kickstart` and then polls until the job leaves launchd's management (i.e., exits). This timeout started at 1200 seconds, which turned out to be insufficient in practice.\n\n```\n# 実測(2026-08-21): affameba-gen 等の claude 生成レーンは 20 分を超える。\n# 1200s だと「待つのをやめて次を kick」するだけで前のジョブは生き続け、\n# runbook が要求する直列 kick が崩れて重い生成が重なる（load 40 超の二次被害）。\nJOB_TIMEOUT_SECONDS = 2700\n# timeout 時は待つのをやめるだけでなく実際に止める。ここを殺さないと直列性が保てない。\nJOB_KILL_GRACE_SECONDS = 30\n```\n\nThe lesson here: \"stop waiting\" and \"stop the job\" are two different things. With only the former, the timed-out old job kept running in the background while the next candidate got kicked, the generation workloads piled up, and the load average went past 40 as collateral damage. So `terminate_job` actually kills the job—SIGTERM, then SIGKILL—to guarantee serial execution.\n\n``` php\ndef terminate_job(domain_label: str, label: str) -> None:\n    \"\"\"timeout したジョブを実際に止める。次の kick と重ならせないための直列性の担保。\"\"\"\n    for signal_name in (\"SIGTERM\", \"SIGKILL\"):\n        subprocess.run([\"launchctl\", \"kill\", signal_name, domain_label], capture_output=True, check=False)\n        deadline = time.monotonic() + JOB_KILL_GRACE_SECONDS\n        while time.monotonic() < deadline:\n            pid, _ = launchctl_status(label)\n            if pid is None:\n                return\n            time.sleep(2)\n```\n\nThis script has 18 test cases (17 in `unittest`, plus 1 `pytest`-style function). Writing this many tests for a personal automation script might look like overkill, so here are the cases where they actually earned their keep.\n\n**Don't call claude on days with zero candidates**\n\nThis is the one with the biggest real-world cost. The comment in the `run` function explains why.\n\n```\n# 拾うものが無い日に probe を撃つと、30分おきに claude -p を1日48回空撃ちして\n# クォータを削る（このジョブが防ごうとしている事故そのものを起こす）。\n# 候補が出た時だけ回復を確認する。\nif not candidates:\n    return [], 0, 0\n```\n\nThe test that protects this is `test_no_candidates_skips_probe`.\n\n``` python\ndef test_no_candidates_skips_probe(self):\n    self.add_job(\n        error_text=f\"CLAUDE_QUOTA_JOB_RAN job=com.lily.test exit=0 ts={int(self.now.timestamp())}\",\n    )\n    probe = Mock(return_value=True)\n    kicker = Mock()\n    result = quota_catchup.run(\n        dry_run=False, now=self.now, state_path=self.state, catchup_path=self.catchup,\n        launch_agents=self.agents, log_path=self.root / \"result.log\", probe=probe, kicker=kicker,\n    )\n    self.assertEqual(result, ([], 0, 0))\n    probe.assert_not_called()\n    kicker.assert_not_called()\n```\n\n\"The batch that checks whether the quota has recovered burns quota just by checking\" is a self-contradiction that anyone who built a circuit breaker absolutely does not want to step into. That single line, `probe.assert_not_called()`, guarantees it mechanically.\n\n**Today's skip is a candidate; a skip from three days ago is not**\n\n``` python\ndef test_today_skip_is_candidate(self):\n    label = self.add_job()\n    self.assertEqual([item.label for item in self.candidates()], [label])\n\ndef test_old_skip_is_not_candidate(self):\n    self.add_job(mtime=self.now - timedelta(days=3))\n    self.assertEqual(self.candidates(), [])\n```\n\n`latest_marker_is_today_skip` uses a regex to pull `ts=` out of the log and checks whether the date is today. If a SKIPPED log from three days ago got picked up again today, you'd have a zombie state where past failures get re-run every single day.\n\n**A past slot makes it a candidate even if a future slot remains**\n\n``` python\ndef test_past_slot_makes_candidate_even_if_later_slot_is_future(self):\n    self.add_job(schedule=[{\"Hour\": 9, \"Minute\": 0}, {\"Hour\": 18, \"Minute\": 0}])\n    self.assertEqual([item.label for item in self.candidates()], [\"com.lily.test\"])\n```\n\nThis test pins the OR logic in `calendar_slot_passed` described above, using a real `daily-brief`-style schedule (multiple slots).\n\n**Don't get the marker order wrong**\n\nThe log contains both `SKIPPED` and `RAN`. If you misjudge which one is the latest marker, you'll either double-kick a job that actually succeeded, or miss a separate skip that happened after a success.\n\n``` python\ndef test_ran_marker_after_skip_in_same_log_is_not_candidate(self):\n    label = \"com.lily.mixed\"\n    today = int(self.now.timestamp())\n    self.add_job(\n        label,\n        error_text=(\n            f\"CLAUDE_QUOTA_JOB_SKIPPED job={label} reason=quota remaining=1s ts={today - 1}\\n\"\n            f\"CLAUDE_QUOTA_JOB_RAN job={label} exit=0 ts={today}\"\n        ),\n    )\n    self.assertEqual(self.candidates(), [])\n```\n\nIf RAN comes after SKIPPED, the job \"succeeded later after all\" and is not a candidate. `latest_job_marker` guarantees this chronological judgment by scanning `reversed(lines)`.\n\n**Note**\n\nThe common goal of this script's unit tests is not \"prove the clever logic is correct\" but \"**pin down, ahead of time, the boundaries a naive implementation gets wrong**.\" The OR logic in calendar_slot_passed, the old-vs-new marker judgment, suppressing the probe when there are zero candidates—each of these flips its result depending on how a single line is written, and none of them are easy to notice until you actually run it. The value of writing pytest for a personal automation script isn't to convince a reviewer; it's so that **six months from now, when you change the spec, you don't step on the same mistake again**.\n\n`JOB_KILL_GRACE_SECONDS=30` providing the SIGTERM→SIGKILL grace period`terminate_job`, the previous job stays alive while the next one gets kicked, causing the load-over-40 collateral damage`probe_claude()` when there are candidates`StartInterval` is present, `calendar_slot_passed` is unconditionally False`reversed(lines)`\n`--dry-run` turns verification cost into execution cost`is_guarded` / `latest_marker_is_today_skip` / Next time I plan to write about the circuit breaker itself—how `claude-quota-guard.py` distinguishes a genuine \"limit reached\" message from a successful run whose article body just happens to contain the same words.\n\nHow do you handle catch-up for scheduled jobs that got skipped in your own setup—do you re-run them, or just wait for the next slot?\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the", "canonical_source": "https://dev.to/bokuwalily/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the-circuit-closes-3f95", "published_at": "2026-09-18 00:00:03+00:00", "updated_at": "2026-09-18 00:23:05.881455+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["claude-quota-guard.py", "quota-catchup.py", "launchd", "StartCalendarInterval", "com.shun.daily-brief.plist", "Wiki"], "alternates": {"html": "https://wpnews.pro/news/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the", "markdown": "https://wpnews.pro/news/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the.md", "text": "https://wpnews.pro/news/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the.txt", "jsonld": "https://wpnews.pro/news/15-launchd-jobs-and-one-quota-circuit-breaker-deciding-what-to-re-run-once-the.jsonld"}}