2 Pitfalls in Priority Probes: Letting One Real Request Through an Open Circuit Breaker

A developer extended the claude-quota-guard.py circuit breaker so that a single priority job can still send one real request every 30 minutes while the circuit is open, after uniform job blocking dropped the xpilot.autopost posting job to a 43% execution rate (231 runs vs. 308 skips). The fix introduces claim_priority_probe and a --priority flag in run_job, sharing one probe slot across all priority jobs to avoid wasting quota. The developer also reports two pitfalls encountered, one of which silently dropped a daily job for four days from 2026-09-13 to 16.

A circuit breaker that stops everything is easy to reason about — until the one job that can't afford to wait gets stopped along with everything else. In my setup, that job was posting: it fell to a 43% execution rate 231 runs vs. 308 skips because engagement jobs had burned through the quota first. This post is about the fix inside claude-quota-guard.py — claim priority probe and run job , which let a single real request through while the circuit is still open — and the two pitfalls I hit along the way, one of which silently dropped a daily job for four days 2026-09-13 to 16 . Last time, I wrote about how the gate blocks one row, not the whole batch https://zenn.dev/bokuwalily/articles/gate-blocks-one-row-not-batch . Before I get to quota-catchup.py — the script that re-runs everything once it detects quota recovery — I want to look at how the SKIPPED markers it reads are actually produced . When claude-quota-guard.py detects the quota limit, it returns EXIT CIRCUIT OPEN 75 until open until and stops all 15 guarded jobs uniformly. A comment in the code explains why that wasn't enough: 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。 実測 launchd.log 累計 : xpilot.autopost は 231実行/308スキップ＝実行率43%で、 threadspilot.engage 64% より優先度が低い扱いになっていた。投稿はその時間帯を逃すと 二度と埋まらないので、--priority を付けたジョブだけは circuit が開いていても この間隔で1回だけ試行を許す。試行が通ればクォータ回復の早期検知にもなる 従来は open until まで盲目的に待つだけだった 。 claude-quota-guard.py:18-24 The measured result: stopping everything uniformly meant the posting job xpilot.autopost got caught in the crossfire of the quota-hungry engagement jobs and dropped to a 43% execution rate — effectively lower priority than threadspilot.engage 64% . On top of that, open until is determined either by "the reset time parsed from the limit message" or by "a 6-hour cooldown" record claude result , so even if the actual quota comes back earlier, the circuit dutifully stays open until that time. For a job like posting, where "if you miss the time slot, it never gets filled," that's not something you can ignore. --priority jobs one attempt every 30 minutes The fix is: "even while the circuit is open, let priority jobs — and only priority jobs — send one real request at a fixed interval." The caller passes --priority when handing a command to run job . php def run job label: str, command: list str , priority: bool = False - int: if not command: print "claude quota guard: --job requires a command after --", file=sys.stderr return 2 status = circuit status probe claimed = False if status "is open" : if not priority and claim priority probe : print "CLAUDE QUOTA JOB SKIPPED " f"job={label} reason={status 'reason' } remaining={status 'remaining seconds' }s ts={now }", file=sys.stderr, return 0 probe claimed = True print "CLAUDE QUOTA JOB PRIORITY PROBE " f"job={label} reason={status 'reason' } remaining={status 'remaining seconds' }s ts={now }", file=sys.stderr, claude-quota-guard.py:471-490 Even with priority set, whether the job can actually attempt anything is decided by claim priority probe . php def claim priority probe - bool: """circuit が開いている間、優先ジョブに試行権を1つ渡す。 間隔は全優先ジョブで共有する =1本が使ったら次の枠まで他も待つ 。上限に本当に 達している間に何本も叩いてもクォータは戻らないため、叩く回数自体を絞る。 """ if PRIORITY PROBE INTERVAL <= 0: return False with locked state as state: last = int state.get "last priority probe" or 0 current = now if current - last < PRIORITY PROBE INTERVAL: return False state "last priority probe" = current return True claude-quota-guard.py:454-468 ; PRIORITY PROBE INTERVAL defaults to 1800 seconds at claude-quota-guard.py:25 last priority probe is a single timestamp stored in locked state a JSON-persisted state guarded by fcntl.flock . The key point is that this value lives per circuit, not per job : it doesn't care who probed, only "has it been 30 minutes since anyone last probed?" The side that wins the attempt passes CLAUDE QUOTA PRIORITY PROBE=1 in the environment of the child process it launches via subprocess.run . guard = str Path file .resolve env = os.environ.copy env "CLAUDE AUTOMATION GUARD" = "1" env "CLAUDE" = guard env "CLAUDE BIN" = guard if probe claimed: 内側の run claude に「プローブとして走っている」ことを伝える これが無いと circuit で即 75 になる env "CLAUDE QUOTA PRIORITY PROBE" = "1" claude-quota-guard.py:491-498 The receiver of this flag is run claude . When the job command internally invokes the real claude binary, PATH has been rewired so the call goes through the guard itself — which means a second circuit status check runs inside the child process. php def run claude arguments: list str - int: status = circuit status if status "is open" : run job が優先プローブを claim した子プロセスだけは circuit を素通りして実 claude を叩く。 結果は record claude result に入るので、上限文なら circuit が延び、成功なら閉じる。 if os.environ.get "CLAUDE QUOTA PRIORITY PROBE" = "1": print "CLAUDE QUOTA CIRCUIT OPEN " f"reason={status 'reason' } remaining={status 'remaining seconds' }s", file=sys.stderr, return EXIT CIRCUIT OPEN print "CLAUDE QUOTA PRIORITY PROBE PASS " f"reason={status 'reason' } remaining={status 'remaining seconds' }s", file=sys.stderr, claude-quota-guard.py:416-432 If you forget to propagate the environment variable, run job wins the attempt, but the inner run claude checks the circuit again and immediately returns EXIT CIRCUIT OPEN . The one line that gets you through both layers of the gate is env "CLAUDE QUOTA PRIORITY PROBE" = "1" . And the result of the actual call flows into record claude result as usual. If the limit message still comes back, open until is extended; if the call succeeds, the circuit closes. Whether the probe fails or succeeds, that single result directly determines the circuit's next state — which is what makes this "one real request let through." As the docstring on claim priority probe says, PRIORITY PROBE INTERVAL is shared across all jobs, not tracked per job . Even if several --priority jobs are scheduled inside the same 30-minute window, the moment the first one passes claim priority probe , last priority probe is updated. Every subsequent job evaluates to False at priority and claim priority probe and falls through to SKIPPED without ever touching the real client . This is intentional. While the limit is genuinely in effect, hammering it with multiple requests won't bring the quota back, so the design throttles the number of attempts themselves. Operationally, though, if you forget that "having multiple priority jobs does not mean each gets its own 30-minute opportunity," you will lose time wondering "why does this one job never get a turn to verify recovery?" RAN exit≠0 This is the main subject of this post. After subprocess.run , run job checks whether the probe came up empty and emits a different marker accordingly . try: result = subprocess.run command, env=env, check=False except OSError as exc: print f"claude quota guard job={label}: {exc}", file=sys.stderr return 127 if probe claimed and result.returncode = 0 and circuit status "is open" : 優先プローブが上限のまま空振りした。RAN exit≠0 のまま残すと quota-catchup.py （最新マーカー=SKIPPED だけを再実行）から漏れ、復帰後も当日分が欠番になる （2026-09-13〜16 の note2-daily / codex-note-funnel 実測）。SKIPPED として記録する。 print "CLAUDE QUOTA JOB SKIPPED " f"job={label} reason=priority-probe-quota exit={result.returncode} ts={now }", file=sys.stderr, return result.returncode print f"CLAUDE QUOTA JOB RAN job={label} exit={result.returncode} ts={now }", file=sys.stderr, return result.returncode claude-quota-guard.py:499-518 Before this branch existed, the code only looked at the fact that the probe had won the attempt and actually launched a child process, and fell straight through to the trailing CLAUDE QUOTA JOB RAN . Even when the probe hit the limit again and ended with exit≠0 , what remained in the log was CLAUDE QUOTA JOB RAN job=... exit=1 . The problem is that this RAN marker means "already done" as far as quota-catchup.py is concerned. Here is latest job marker , which quota-catchup.py uses to narrow down re-run candidates: php def latest job marker paths: list Path , label: str - Optional Tuple str, int : """Return the newest timestamped skip/run marker for one launchd label.""" marker pattern = re.compile r"CLAUDE QUOTA JOB SKIPPED|RAN \s+job=" + re.escape label + r" ?=\s|$ . \bts= \d+ ?=\s|$ " ... def latest marker is today skip paths: list Path , label: str, today: datetime.date - bool: marker = latest job marker paths, label if marker is None: return False marker type, timestamp = marker return marker type == "SKIPPED" and datetime.fromtimestamp timestamp .astimezone .date == today quota-catchup.py:160-193 As you can see, this check only looks at the marker type SKIPPED or RAN and never inspects the exit code on the RAN side . Even if the probe failed with exit=1 , as long as the last log line is CLAUDE QUOTA JOB RAN , latest marker is today skip returns False and the job quietly drops out of find candidates ' re-run set. The consequence: even after the circuit really closes, the job sits there with the wrong record — "already RAN today" — until its next scheduled time the following morning, for instance . The comment records the real-world impact: note2-daily and codex-note-funnel both lost their daily run through this path between 2026-09-13 and 16. The fix is simple. Check the condition "the probe came up empty while the circuit was open" probe claimed and result.returncode = 0 and circuit status "is open" first, and only in that case emit CLAUDE QUOTA JOB SKIPPED reason=priority-probe-quota instead of CLAUDE QUOTA JOB RAN . The function's return value stays result.returncode , unchanged. launchd's LastExitStatus still records the actual failure correctly, while only the log marker that quota-catchup.py reads gets relabeled as "still needs a retry." That's the separation of concerns. Note: The exit code and "should this be retried?" are separate axes. This bug happened because the two had been crammed into a single RAN marker, and the quota-catchup.py side never anticipated the exit≠0 case. When designing log markers, it's safer not to let "what actually happened" and "what the downstream batch should do next" share the same string. claim priority probe through last priority probe in locked state , and it is CLAUDE QUOTA PRIORITY PROBE=1 . Forget it, and the inner run claude rejects the call at the second circuit check CLAUDE QUOTA JOB RAN exit≠0 . The re-run check in quota-catchup.py only looks at the marker type and never at the exit code Next time, I'll cover how quota-catchup.py picks up these SKIPPED markers and decides how much to re-run after recovery https://zenn.dev/bokuwalily/articles/quota-catchup-slot-selection . If you run a circuit breaker in front of your own scheduled jobs: does your downstream retry logic distinguish "ran and failed" from "never really got a chance"? Written by Lily — I ship iOS apps and automate my content stack with Claude Code. Follow along: Portfolio https://bokuwalily.com · X https://x.com/bokuwalily · GitHub https://github.com/bokuwalily