# 2 Pitfalls in Priority Probes: Letting One Real Request Through an Open Circuit Breaker

> Source: <https://dev.to/bokuwalily/2-pitfalls-in-priority-probes-letting-one-real-request-through-an-open-circuit-breaker-4km5>
> Published: 2026-09-20 00:00:06+00:00

A circuit breaker that stops everything is easy to reason about — until the one job that can't afford to wait gets stopped along with everything else. In my setup, that job was posting: it fell to a 43% execution rate (231 runs vs. 308 skips) because engagement jobs had burned through the quota first. This post is about the fix inside `claude-quota-guard.py` — `claim_priority_probe` and `run_job`, which let a single real request through while the circuit is still open — and the two pitfalls I hit along the way, one of which silently dropped a daily job for four days (2026-09-13 to 16).

Last time, I wrote about [how the gate blocks one row, not the whole batch](https://zenn.dev/bokuwalily/articles/gate-blocks-one-row-not-batch). Before I get to `quota-catchup.py` — the script that re-runs everything once it detects quota recovery — I want to look at **how the SKIPPED markers it reads are actually produced**.

When `claude-quota-guard.py` detects the quota limit, it returns `EXIT_CIRCUIT_OPEN` (75) until `open_until` and stops all 15 guarded jobs uniformly. A comment in the code explains why that wasn't enough:

```
# 🔴 2026-08-21: circuit が開くと全ジョブが一律で止まるため、消費の大半を占める
# 返信/エンゲージ系がクォータを使い切った巻き添えで「投稿」まで停止していた。
# 実測(launchd.log 累計): xpilot.autopost は 231実行/308スキップ＝実行率43%で、
# threadspilot.engage(64%) より優先度が低い扱いになっていた。投稿はその時間帯を逃すと
# 二度と埋まらないので、--priority を付けたジョブだけは circuit が開いていても
# この間隔で1回だけ試行を許す。試行が通ればクォータ回復の早期検知にもなる
# (従来は open_until まで盲目的に待つだけだった)。
```

(`claude-quota-guard.py:18-24`)

The measured result: stopping everything uniformly meant the posting job (`xpilot.autopost`) got caught in the crossfire of the quota-hungry engagement jobs and dropped to a 43% execution rate — effectively lower priority than `threadspilot.engage` (64%). On top of that, `open_until` is determined either by "the reset time parsed from the limit message" or by "a 6-hour cooldown" (`record_claude_result`), so even if the actual quota comes back earlier, the circuit dutifully stays open until that time. For a job like posting, where "if you miss the time slot, it never gets filled," that's not something you can ignore.

`--priority` jobs one attempt every 30 minutes
The fix is: "even while the circuit is open, let priority jobs — and only priority jobs — send one real request at a fixed interval." The caller passes `--priority` when handing a command to `run_job`.

``` php
def run_job(label: str, command: list[str], priority: bool = False) -> int:
    if not command:
        print("claude quota guard: --job requires a command after --", file=sys.stderr)
        return 2
    status = circuit_status()
    probe_claimed = False
    if status["is_open"]:
        if not (priority and claim_priority_probe()):
            print(
                "CLAUDE_QUOTA_JOB_SKIPPED "
                f"job={label} reason={status['reason']} remaining={status['remaining_seconds']}s ts={now()}",
                file=sys.stderr,
            )
            return 0
        probe_claimed = True
        print(
            "CLAUDE_QUOTA_JOB_PRIORITY_PROBE "
            f"job={label} reason={status['reason']} remaining={status['remaining_seconds']}s ts={now()}",
            file=sys.stderr,
        )
```

(`claude-quota-guard.py:471-490`)

Even with `priority` set, whether the job can actually attempt anything is decided by `claim_priority_probe()`.

``` php
def claim_priority_probe() -> bool:
    """circuit が開いている間、優先ジョブに試行権を1つ渡す。

    間隔は全優先ジョブで共有する(=1本が使ったら次の枠まで他も待つ)。上限に本当に
    達している間に何本も叩いてもクォータは戻らないため、叩く回数自体を絞る。
    """
    if PRIORITY_PROBE_INTERVAL <= 0:
        return False
    with locked_state() as state:
        last = int(state.get("last_priority_probe") or 0)
        current = now()
        if current - last < PRIORITY_PROBE_INTERVAL:
            return False
        state["last_priority_probe"] = current
        return True
```

(`claude-quota-guard.py:454-468`; `PRIORITY_PROBE_INTERVAL` defaults to 1800 seconds at `claude-quota-guard.py:25`)

`last_priority_probe` is a single timestamp stored in `locked_state()` (a JSON-persisted state guarded by `fcntl.flock`). The key point is that this value lives **per circuit, not per job**: it doesn't care *who* probed, only "has it been 30 minutes since *anyone* last probed?"

The side that wins the attempt passes `CLAUDE_QUOTA_PRIORITY_PROBE=1` in the environment of the child process it launches via `subprocess.run`.

```
    guard = str(Path(__file__).resolve())
    env = os.environ.copy()
    env["CLAUDE_AUTOMATION_GUARD"] = "1"
    env["CLAUDE"] = guard
    env["CLAUDE_BIN"] = guard
    if probe_claimed:
        # 内側の run_claude に「プローブとして走っている」ことを伝える(これが無いと circuit で即 75 になる)
        env["CLAUDE_QUOTA_PRIORITY_PROBE"] = "1"
```

(`claude-quota-guard.py:491-498`)

The receiver of this flag is `run_claude`. When the job command internally invokes the real `claude` binary, PATH has been rewired so the call goes through the guard itself — which means a second `circuit_status()` check runs inside the child process.

``` php
def run_claude(arguments: list[str]) -> int:
    status = circuit_status()
    if status["is_open"]:
        # run_job が優先プローブを claim した子プロセスだけは circuit を素通りして実 claude を叩く。
        # 結果は record_claude_result に入るので、上限文なら circuit が延び、成功なら閉じる。
        if os.environ.get("CLAUDE_QUOTA_PRIORITY_PROBE") != "1":
            print(
                "CLAUDE_QUOTA_CIRCUIT_OPEN "
                f"reason={status['reason']} remaining={status['remaining_seconds']}s",
                file=sys.stderr,
            )
            return EXIT_CIRCUIT_OPEN
        print(
            "CLAUDE_QUOTA_PRIORITY_PROBE_PASS "
            f"reason={status['reason']} remaining={status['remaining_seconds']}s",
            file=sys.stderr,
        )
```

(`claude-quota-guard.py:416-432`)

If you forget to propagate the environment variable, `run_job` wins the attempt, but the inner `run_claude` checks the circuit again and immediately returns `EXIT_CIRCUIT_OPEN`. The one line that gets you through both layers of the gate is `env["CLAUDE_QUOTA_PRIORITY_PROBE"] = "1"`.

And the result of the actual call flows into `record_claude_result` as usual. If the limit message still comes back, `open_until` is extended; if the call succeeds, the circuit closes. **Whether the probe fails or succeeds, that single result directly determines the circuit's next state** — which is what makes this "one real request let through."

As the docstring on `claim_priority_probe` says, `PRIORITY_PROBE_INTERVAL` is **shared across all jobs, not tracked per job**. Even if several `--priority` jobs are scheduled inside the same 30-minute window, the moment the first one passes `claim_priority_probe()`, `last_priority_probe` is updated. Every subsequent job evaluates to False at `priority and claim_priority_probe()` and falls through to SKIPPED **without ever touching the real client**.

This is intentional. While the limit is genuinely in effect, hammering it with multiple requests won't bring the quota back, so the design throttles the number of attempts themselves. Operationally, though, if you forget that "having multiple priority jobs does not mean each gets its own 30-minute opportunity," you will lose time wondering "why does this one job never get a turn to verify recovery?"

`RAN exit≠0`
This is the main subject of this post. After `subprocess.run`, `run_job` checks whether the probe came up empty and **emits a different marker accordingly**.

```
    try:
        result = subprocess.run(command, env=env, check=False)
    except OSError as exc:
        print(f"claude quota guard job={label}: {exc}", file=sys.stderr)
        return 127
    if probe_claimed and result.returncode != 0 and circuit_status()["is_open"]:
        # 優先プローブが上限のまま空振りした。RAN exit≠0 のまま残すと quota-catchup.py
        # （最新マーカー=SKIPPED だけを再実行）から漏れ、復帰後も当日分が欠番になる
        # （2026-09-13〜16 の note2-daily / codex-note-funnel 実測）。SKIPPED として記録する。
        print(
            "CLAUDE_QUOTA_JOB_SKIPPED "
            f"job={label} reason=priority-probe-quota exit={result.returncode} ts={now()}",
            file=sys.stderr,
        )
        return result.returncode
    print(
        f"CLAUDE_QUOTA_JOB_RAN job={label} exit={result.returncode} ts={now()}",
        file=sys.stderr,
    )
    return result.returncode
```

(`claude-quota-guard.py:499-518`)

Before this branch existed, the code only looked at the fact that the probe had won the attempt and actually launched a child process, and fell straight through to the trailing `CLAUDE_QUOTA_JOB_RAN`. Even when the probe hit the limit again and ended with `exit≠0`, what remained in the log was `CLAUDE_QUOTA_JOB_RAN job=... exit=1`.

The problem is that this `RAN` marker means "already done" as far as `quota-catchup.py` is concerned. Here is `latest_job_marker`, which `quota-catchup.py` uses to narrow down re-run candidates:

``` php
def latest_job_marker(paths: list[Path], label: str) -> Optional[Tuple[str, int]]:
    """Return the newest timestamped skip/run marker for one launchd label."""
    marker_pattern = re.compile(
        r"CLAUDE_QUOTA_JOB_(SKIPPED|RAN)\s+job="
        + re.escape(label)
        + r"(?=\s|$).*\bts=(\d+)(?=\s|$)"
    )
    ...

def latest_marker_is_today_skip(paths: list[Path], label: str, today: datetime.date) -> bool:
    marker = latest_job_marker(paths, label)
    if marker is None:
        return False
    marker_type, timestamp = marker
    return marker_type == "SKIPPED" and datetime.fromtimestamp(timestamp).astimezone().date() == today
```

(`quota-catchup.py:160-193`)

As you can see, this check **only looks at the marker type (`SKIPPED` or `RAN`) and never inspects the `exit` code on the `RAN` side**. Even if the probe failed with `exit=1`, as long as the last log line is `CLAUDE_QUOTA_JOB_RAN`, `latest_marker_is_today_skip` returns False and the job quietly drops out of `find_candidates`' re-run set.

The consequence: even after the circuit really closes, the job sits there with the wrong record — "already `RAN` today" — until its next scheduled time (the following morning, for instance). The comment records the real-world impact: `note2-daily` and `codex-note-funnel` both lost their daily run through this path between 2026-09-13 and 16.

The fix is simple. Check the condition "the probe came up empty while the circuit was open" (`probe_claimed and result.returncode != 0 and circuit_status()["is_open"]`) first, and only in that case emit `CLAUDE_QUOTA_JOB_SKIPPED reason=priority-probe-quota` instead of `CLAUDE_QUOTA_JOB_RAN`. The function's return value stays `result.returncode`, unchanged. launchd's `LastExitStatus` still records the actual failure correctly, while only the log marker that `quota-catchup.py` reads gets relabeled as "still needs a retry." That's the separation of concerns.

**Note:** The exit code and "should this be retried?" are separate axes. This bug happened because the two had been crammed into a single `RAN` marker, and the `quota-catchup.py` side never anticipated the `exit≠0` case. When designing log markers, it's safer not to let "what actually happened" and "what the downstream batch should do next" share the same string.

`claim_priority_probe` through `last_priority_probe` in `locked_state`, and it is `CLAUDE_QUOTA_PRIORITY_PROBE=1`. Forget it, and the inner `run_claude` rejects the call at the second circuit check`CLAUDE_QUOTA_JOB_RAN exit≠0`. The re-run check in `quota-catchup.py` only looks at the marker type and never at the `exit` code
Next time, I'll cover how `quota-catchup.py` picks up these `SKIPPED` markers and [decides how much to re-run after recovery](https://zenn.dev/bokuwalily/articles/quota-catchup-slot-selection).

If you run a circuit breaker in front of your own scheduled jobs: does your downstream retry logic distinguish "ran and failed" from "never really got a chance"?

*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.

Follow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*