15 launchd Jobs and One Quota Circuit Breaker: Deciding What to Re-run Once the Circuit Closes

A developer built quota-catchup.py, a companion tool to the claude-quota-guard.py quota circuit breaker that decides which of 15 launchd jobs to re-run after a quota-exhaustion circuit closes. The script identifies re-run candidates by checking whether a plist is guarded, whether today's latest marker is a SKIPPED entry, and whether a StartCalendarInterval slot has already passed, avoiding duplicate runs and unnecessary catch-up for StartInterval jobs.

A circuit breaker that halts every job the moment your quota runs dry is only half the story. The harder question shows up afterward: once the circuit closes again, what do you do with everything it skipped? In my previous post I wrote about fixing the Wiki secret-scan sync https://zenn.dev/bokuwalily/articles/wiki-improvements-secret-scan-sync . This time I'm switching gears and following up on claude-quota-guard.py , the quota circuit breaker I built. That script's story ended at "detect quota exhaustion, stop all jobs." In real operation, there's a whole second problem waiting past that point—and along the way it involved a 1200-second timeout that turned out to need 2700, and a load average that climbed past 40 when I got it wrong. run job in claude-quota-guard.py returns exit 0 immediately when a job starts while the circuit is open, leaving nothing but a marker in the log. print "CLAUDE QUOTA JOB SKIPPED " f"job={label} reason={status 'reason' } remaining={status 'remaining seconds' }s ts={now }", file=sys.stderr, return 0 This guard sits in front of 15 of the plists under ~/Library/LaunchAgents/ .plist . As the comment in the code puts it: 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。 The problem is that after the circuit closes and open until has passed, launchd does nothing until that job's next StartCalendarInterval slot comes around . If the 9:00 AM job was skipped for quota reasons and the quota recovers at 10:00, but the next slot is 9:00 AM tomorrow, you've lost an entire day's execution. Deciding "what to re-run, and how much, after recovery" is the job of quota-catchup.py . quota-catchup.py walks every plist in find candidates and only treats a job as a re-run candidate if it passes three conditions ANDed together. | Condition | Function | Purpose | |---|---|---| | Is this plist guarded? | is guarded | Don't mix in unrelated jobs | | Is today's latest marker SKIPPED? | latest marker is today skip | Exclude jobs that already ran, and stale skips | | Has a slot already passed? | calendar slot passed | Exclude StartInterval jobs and jobs with only future slots | php def find candidates ... - list Candidate : candidates: list Candidate = for plist path in sorted launch agents.glob " .plist" : plist = load plist plist path if not plist or not is guarded plist : continue ... if label in already kicked or not latest marker is today skip output paths, label, today : continue if calendar slot passed plist, now : candidates.append Candidate label, plist path return candidates Re-run targets are limited to jobs that are "launched via claude-quota-guard.py ." The check is nothing more than a string match on ProgramArguments . php def is guarded plist: dict - bool: arguments = plist.get "ProgramArguments" return isinstance arguments, list and any "claude-quota-guard" in str value for value in arguments If you picked up plists that don't go through the guard, you'd end up kicking ordinary cron-style jobs that have nothing to do with the quota. This is the star of the show. A single job can have multiple StartCalendarInterval slots. A real example is com.shun.daily-brief.plist . <key StartCalendarInterval</key <array <dict <key Hour</key <integer 8</integer <key Minute</key <integer 0</integer </dict <dict <key Hour</key <integer 10</integer <key Minute</key <integer 30</integer </dict </array Even if the 8:00 slot gets skipped for quota, the same job naturally runs again at 10:30. So the rule is "if even one slot has passed, it's a re-run candidate," not "wait, because there's still a future slot." Conversely, StartInterval jobs e.g., every 30 minutes will naturally re-run on the next interval if you just leave them alone, so there's no need to make them catch-up targets. php def calendar slot passed plist: dict, now: datetime - bool: """True only for calendar-only jobs with at least one past slot today.""" if "StartInterval" in plist: return False raw entries = plist.get "StartCalendarInterval" if isinstance raw entries, dict : entries = raw entries elif isinstance raw entries, list : entries = raw entries else: return False saw today slot = False for entry in entries: if not isinstance entry, dict or "Hour" not in entry: return False if not runs today entry, now : continue try: scheduled = now.replace hour=int entry "Hour" , minute=int entry.get "Minute", 0 , second=0, microsecond=0, except TypeError, ValueError : return False if scheduled now: continue saw today slot = True return saw today slot The key is walking every slot to the end and accumulating saw today slot with an OR. If you'd written it as "return the verdict from the first slot you find," you'd hit a bug where the result depends on the order of the slots. For example, with the ordering {9:00}, {18:00} evaluated at 13:00, the correct answer is "candidate" because 9:00 is in the past—but if the loop only judged by the last entry, it would misclassify the job as not a candidate on the grounds that 18:00 is in the future. A re-run goes through kick and wait , which runs launchctl kickstart and then polls until the job leaves launchd's management i.e., exits . This timeout started at 1200 seconds, which turned out to be insufficient in practice. 実測 2026-08-21 : affameba-gen 等の claude 生成レーンは 20 分を超える。 1200s だと「待つのをやめて次を kick」するだけで前のジョブは生き続け、 runbook が要求する直列 kick が崩れて重い生成が重なる（load 40 超の二次被害）。 JOB TIMEOUT SECONDS = 2700 timeout 時は待つのをやめるだけでなく実際に止める。ここを殺さないと直列性が保てない。 JOB KILL GRACE SECONDS = 30 The lesson here: "stop waiting" and "stop the job" are two different things. With only the former, the timed-out old job kept running in the background while the next candidate got kicked, the generation workloads piled up, and the load average went past 40 as collateral damage. So terminate job actually kills the job—SIGTERM, then SIGKILL—to guarantee serial execution. php def terminate job domain label: str, label: str - None: """timeout したジョブを実際に止める。次の kick と重ならせないための直列性の担保。""" for signal name in "SIGTERM", "SIGKILL" : subprocess.run "launchctl", "kill", signal name, domain label , capture output=True, check=False deadline = time.monotonic + JOB KILL GRACE SECONDS while time.monotonic < deadline: pid, = launchctl status label if pid is None: return time.sleep 2 This script has 18 test cases 17 in unittest , plus 1 pytest -style function . Writing this many tests for a personal automation script might look like overkill, so here are the cases where they actually earned their keep. Don't call claude on days with zero candidates This is the one with the biggest real-world cost. The comment in the run function explains why. 拾うものが無い日に probe を撃つと、30分おきに claude -p を1日48回空撃ちして クォータを削る（このジョブが防ごうとしている事故そのものを起こす）。 候補が出た時だけ回復を確認する。 if not candidates: return , 0, 0 The test that protects this is test no candidates skips probe . python def test no candidates skips probe self : self.add job error text=f"CLAUDE QUOTA JOB RAN job=com.lily.test exit=0 ts={int self.now.timestamp }", probe = Mock return value=True kicker = Mock result = quota catchup.run dry run=False, now=self.now, state path=self.state, catchup path=self.catchup, launch agents=self.agents, log path=self.root / "result.log", probe=probe, kicker=kicker, self.assertEqual result, , 0, 0 probe.assert not called kicker.assert not called "The batch that checks whether the quota has recovered burns quota just by checking" is a self-contradiction that anyone who built a circuit breaker absolutely does not want to step into. That single line, probe.assert not called , guarantees it mechanically. Today's skip is a candidate; a skip from three days ago is not python def test today skip is candidate self : label = self.add job self.assertEqual item.label for item in self.candidates , label def test old skip is not candidate self : self.add job mtime=self.now - timedelta days=3 self.assertEqual self.candidates , latest marker is today skip uses a regex to pull ts= out of the log and checks whether the date is today. If a SKIPPED log from three days ago got picked up again today, you'd have a zombie state where past failures get re-run every single day. A past slot makes it a candidate even if a future slot remains python def test past slot makes candidate even if later slot is future self : self.add job schedule= {"Hour": 9, "Minute": 0}, {"Hour": 18, "Minute": 0} self.assertEqual item.label for item in self.candidates , "com.lily.test" This test pins the OR logic in calendar slot passed described above, using a real daily-brief -style schedule multiple slots . Don't get the marker order wrong The log contains both SKIPPED and RAN . If you misjudge which one is the latest marker, you'll either double-kick a job that actually succeeded, or miss a separate skip that happened after a success. python def test ran marker after skip in same log is not candidate self : label = "com.lily.mixed" today = int self.now.timestamp self.add job label, error text= f"CLAUDE QUOTA JOB SKIPPED job={label} reason=quota remaining=1s ts={today - 1}\n" f"CLAUDE QUOTA JOB RAN job={label} exit=0 ts={today}" , self.assertEqual self.candidates , If RAN comes after SKIPPED, the job "succeeded later after all" and is not a candidate. latest job marker guarantees this chronological judgment by scanning reversed lines . Note The common goal of this script's unit tests is not "prove the clever logic is correct" but " pin down, ahead of time, the boundaries a naive implementation gets wrong ." The OR logic in calendar slot passed, the old-vs-new marker judgment, suppressing the probe when there are zero candidates—each of these flips its result depending on how a single line is written, and none of them are easy to notice until you actually run it. The value of writing pytest for a personal automation script isn't to convince a reviewer; it's so that six months from now, when you change the spec, you don't step on the same mistake again . JOB KILL GRACE SECONDS=30 providing the SIGTERM→SIGKILL grace period terminate job , the previous job stays alive while the next one gets kicked, causing the load-over-40 collateral damage probe claude when there are candidates StartInterval is present, calendar slot passed is unconditionally False reversed lines --dry-run turns verification cost into execution cost is guarded / latest marker is today skip / Next time I plan to write about the circuit breaker itself—how claude-quota-guard.py distinguishes a genuine "limit reached" message from a successful run whose article body just happens to contain the same words. How do you handle catch-up for scheduled jobs that got skipped in your own setup—do you re-run them, or just wait for the next slot? Written by Lily — I ship iOS apps and automate my content stack with Claude Code. Follow along: Portfolio https://bokuwalily.com · X https://x.com/bokuwalily · GitHub https://github.com/bokuwalily