One Line of mv Kills the 4x Duplicate Run After Recovery: Making launchd Queue Processing Idempotent

A developer running an autonomous side business on Claude Code documented how a day-long Codex outage caused 57 queued JSON files to pile up, and when the service recovered, five launchd lanes fired simultaneously and processed the same file four times. The engineer traced the duplicate runs to launchd's lack of mutual exclusion over a shared file queue, noting that a check-then-act lock file is not atomic and that scheduler-plus-shared-directory designs will always collide unless the pickup script implements explicit locking.

Six months ago my side business made ¥0 a month. This month it's ¥1.2M, running on an autonomous setup built with Claude Code. And here's the thing about automation: the moment it breaks is exactly when the sloppiness in your design gets exposed. When automation breaks, the reflex is to reach for a procedural fix: "next time I'll do this instead." Check the logs more often. Retry by hand. Space things out a bit more. All of these are bandages meant to shrink the wound the next time the problem recurs. They don't remove the cause, so once the same conditions line up, it will happen again. After six months of running an automated system, one thing I'm certain of: a scheduler combined with a shared directory will always collide unless you explicitly implement mutual exclusion. Not "it might collide if you're unlucky," but "it will collide whenever the conditions for collision line up." This isn't a probability problem; it's a structural one. On September 17, 2026, Codex was down for an entire day with usage limit exceeded . Under ~/.codex/sessions/2026/09/17/ there were 54 sessions piled up, every one of them terminated with a limit error. Throughout that time, three workflows—note-autolike, ai-portraits-fragments, and social-autolike—kept firing jobs, and unprocessed JSON files accumulated in ~/dev/note-autolike/done/ . Measured the next morning at 05:20, the JSON files flagged needs imagegen thumbnail had swelled from 43 to 57 . Fourteen new files had stacked up, all of them left untouched. The instant Codex came back, five launchd lanes fired at once. Each lane's pickup script grabs the JSON files under done/ with ls or a glob and starts processing. With no mutual exclusion, multiple processes look at the same JSON at the same time. The result: the funnel-pm lane processed the same JSON 4 times, one minute apart . As the numbers show—8 of 11 files were hit 2 or more times—this wasn't a freak accident. It was an inevitable consequence of the design. Look at StartCalendarInterval in com.lily.codex-note-funnel.plist and you'll see two entries: 10:40 and 16:40. <key StartCalendarInterval</key <array <dict <key Hour</key <integer 10</integer <key Minute</key <integer 40</integer </dict <dict <key Hour</key <integer 16</integer <key Minute</key <integer 40</integer </dict </array launchd simply invokes run-codex-funnel.sh according to this schedule. Whether the previous run finished, or whether another lane is currently working on the same file—launchd has no interest in any of it. And that's correct, by-design behavior. launchd is a scheduler, not a mutex. Mutual exclusion over a file queue is the application layer's responsibility—in other words, it's a feature the pickup script itself has to implement. In normal operation, each lane's timing is slightly offset, which naturally keeps collisions rare. Target JSON files also arrive one at a time in sequence, so there are few opportunities for multiple lanes to see the same file simultaneously. But a long outage like Codex's usage limit exceeded changes the situation completely. All lanes start at once with 57 JSON files backed up from the outage period. 通常運転: レーンA → job 001 処理 → 完了 レーンB → job 002 処理 → 完了 （タイミングがズレており競合しない） Codex 復帰直後: レーンA ─┐ レーンB ─┤──→ job 001 を同時取得 → 4連射 funnel-pm ─┘ （57本が溜まっており、全レーンが先頭ファイルに殺到） A procedural fix like "adjust the intervals" can't handle this "burst release" pattern. No matter how tightly you tune the intervals, the instant after recovery, every lane will fire simultaneously. "Just create a lock file" is the natural idea. But look closely at the implementation and there's a problem. NG: check-then-act は原子的でない LOCKFILE=~/dev/note-autolike/.processing.lock if -f "$LOCKFILE" ; then touch "$LOCKFILE" process file "$json" rm "$LOCKFILE" fi Between the if -f check and the touch , there's a "Time of Check to Time of Use" TOCTOU window where another process can slip in. Two processes both observe "no file" and both execute touch —that's the race. Under high load and high frequency, this window really does open. On top of that, if the process crashes mid-run, the lock file stays behind. Every subsequent run sees the lock and exits immediately, so processing stalls completely. You need a separate crash-recovery procedure, and operational cost goes up. File locking via the flock command is another option, but since file descriptors aren't inherited each time launchd spawns a new process, it doesn't work for exclusion between lanes. The existing ~/.claude/scripts/autolike-plist-reconcile.sh contains a guard built with the same concern in mind. That script does a reload when the AUTOLIKE TIMEOUT SEC value in the plist has drifted from the value launchd has loaded, but the core is this code: php pid=$ launchctl list | awk -v l="$L" '$3==l{print $1}' if -n "$pid" && "$pid" = "-" ; then echo " $ date '+%F %T' $L 実行中 pid=$pid のため見送り $have - $want " "$LOG" continue fi launchctl bootout "$D/$L" 2 /dev/null The comment at the top of the script says that booting out a running job wipes out that run's likes entirely, so it must always wait. It fetches the PID via launchctl list and skips the bootout unless the value is - not running . Before starting work, check "is someone else already processing this?"—and if so, exit immediately. That's the same idea as the mv pattern in this article. That said, a PID check has limits. You can't reduce to zero the chance that a new process starts between the "check" and the "bootout." In the context where this script runs, that's acceptable, but for claiming exclusive ownership of a file queue, it's insufficient. You need a stronger atomic operation. Codex usage limit exceeded から復帰 │ ▼ ~/dev/note-autolike/done/ に 57 本が滞留 ┌────────────────────────────────────────┐ │ job 20260917 001.json │ │ job 20260917 002.json │ │ ... │ │ job 20260917 057.json │ └────────────────────────────────────────┘ │ 5 レーンが launchd により一斉起動 ┌────────────────────────────────────────┐ │ com.lily.codex-note-funnel 10:40 │ │ com.lily.autolike.note1 │ │ com.lily.autolike.note2 │ │ com.lily.autolike.funnel-pm ← 4連射 │ │ com.lily.autolike.social │ └────────────────────────────────────────┘ │ 【修正前】各レーンが done/ を glob → 先頭ファイルを取得 │ 排他なし → 全レーンが job 001 を同時に read ↓ funnel-pm が 1分差で同一 JSON を 4 回処理 ───────────────────────────────────────── 【修正後】mv による原子的所有権取得 ┌────────────────────────────────────────┐ │ mv done/job 001.json processing/ │ │ ├─ 成功: 自プロセスのみが処理を継続 │ │ └─ 失敗: 他プロセスが取得済み → exit 0│ └────────────────────────────────────────┘ │ 「1ファイル = 1プロセスのみ処理」が保証される On the same filesystem, mv executes a single POSIX rename 2 system call. Because the "rename this file" operation completes indivisibly at the kernel level, even if two processes simultaneously run mv done/job 001.json processing/job 001.json , exactly one succeeds and the other is guaranteed to fail. No window opens between "check" and "acquire." This is the fundamental difference between a PID check and mv . With a PID check, "check" and "start processing" are separate operations. With mv , "check" and "acquire" are completed inside one system call. 2プロセスが同時に実行した場合の挙動 プロセスA: mv done/job 001.json processing/job 001.json → 成功（終了コード 0） プロセスB: mv done/job 001.json processing/job 001.json → 失敗（No such file or directory, 終了コード 1） Here's the implementation to insert at the top of run-codex-funnel.sh . bash /bin/bash set -uo pipefail DONE DIR=~/dev/note-autolike/done PROCESSING DIR=~/dev/note-autolike/processing mkdir -p "$PROCESSING DIR" done/ から処理対象を1本選ぶ TARGET=$ ls "$DONE DIR"/ .json 2 /dev/null | head -1 -z "$TARGET" && exit 0 対象なし → 正常終了 BASENAME=$ basename "$TARGET" アトミックに所有権を取得。失敗 = 他プロセスが取得済み mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null || exit 0 WORKING="$PROCESSING DIR/$BASENAME" ここから先は自プロセスだけが処理する権利を持つ There's effectively only one situation in which mv fails: another process has already mv 'd the file and it no longer exists. That is precisely the race condition itself, so exiting immediately with exit 0 is the correct response. You don't even need to write an error log. A cross-device mv spanning different mount points is not atomic, so be careful—but in this setup, both done/ and processing/ live under ~/dev/note-autolike/ , which guarantees the same filesystem. The ProgramArguments in com.lily.codex-note-funnel.plist are structured like this: <key ProgramArguments</key <array <string ~/.claude/scripts/claude-quota-guard.py</string <string --job</string <string com.lily.codex-note-funnel</string <string --priority</string <string --</string <string ~/dev/note-autolike/run-codex-funnel.sh</string </array claude-quota-guard.py runs first as a quota-control wrapper, and if there's headroom, it launches run-codex-funnel.sh . Since the pickup logic lives inside run-codex-funnel.sh , inserting the mv ownership claim into the first few lines of the script is all it takes. No plist changes required. launchd will keep calling run-codex-funnel.sh at 10:40 and 16:40. Even with 57 JSON files backed up right after Codex recovers from usage limit exceeded , every time run-codex-funnel.sh starts it repeats the same behavior: "first claim one file with mv ; if that fails, exit immediately." Even if five processes, including the funnel-pm lane, run at the same time, exactly one process handles each file. The first half showed only the single mv line, but to actually run it in production you need layers before and after. Looking at ProgramArguments in com.lily.codex-note-funnel.plist , claude-quota-guard.py runs as the leading wrapper, and then run-codex-funnel.sh is called. The full script looks like this: bash /bin/bash launchd 環境の PATH は最小限のため先頭で上書きする PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin export PATH set -uo pipefail DONE DIR=~/dev/note-autolike/done PROCESSING DIR=~/dev/note-autolike/processing PROCESSED DIR=~/dev/note-autolike/processed LOG=~/dev/note-autolike/logs/codex-funnel.log mkdir -p "$PROCESSING DIR" "$PROCESSED DIR" 起動時リカバリ: 前回クラッシュで processing/ に残ったファイルを done/ へ戻す for stale in "$PROCESSING DIR"/ .json; do -f "$stale" || break age=$ $ date +%s - $ stat -f %m "$stale" if "$age" -gt 3600 ; then mv "$stale" "$DONE DIR/" 2 /dev/null echo " $ date '+%F %T' recovered stale: $ basename "$stale" " "$LOG" fi done done/ から処理対象を1本選ぶ TARGET=$ ls "$DONE DIR"/ .json 2 /dev/null | head -1 -z "$TARGET" && exit 0 BASENAME=$ basename "$TARGET" ─── ここが核心 ───────────────────────────────────────── アトミックに所有権を取得。失敗 = 他プロセスが取得済み → 即退出 mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null || exit 0 ──────────────────────────────────────────────────────── WORKING="$PROCESSING DIR/$BASENAME" クラッシュ時に processing/ へファイルが残るのを防ぐ EXIT トラップ 正常完了時は processed/ へ移動、異常時は done/ へ差し戻す cleanup { local code=$? if $code -eq 0 ; then mv "$WORKING" "$PROCESSED DIR/$BASENAME" 2 /dev/null else mv "$WORKING" "$DONE DIR/$BASENAME" 2 /dev/null echo " $ date '+%F %T' ERROR exit=$code, returned: $BASENAME" "$LOG" fi } trap ' cleanup' EXIT ── ここから先は自プロセスだけが $WORKING を処理する ── echo " $ date '+%F %T' processing: $BASENAME" "$LOG" ... 本体処理 ... Look at EnvironmentVariables in com.lily.codex-note-funnel.plist : the key exists, but the content is an empty <dict/ . <key EnvironmentVariables</key <dict </dict When launchd starts a process, its PATH is only the minimal set /usr/bin:/bin:/usr/sbin:/sbin . If you echo $PATH in a terminal you'll see /usr/local/bin included, but that's added by .zshrc or the nvm initialization script. launchd doesn't run any such shell initialization. Tools installed in /usr/local/bin like jq , node , and python3 fail silently as "command not found" in scripts launched from a plist. The fix is one of two choices: write it in the plist's EnvironmentVariables , or hardcode it at the top of the script. The latter is more convenient because you can verify the script's behavior on its own, so I've standardized on export PATH at the top. set -uo pipefail and || exit 0 Must Go Together When set -uo pipefail is active, the whole script aborts the moment a command returns a non-zero exit code. mv returns exit code 1 when the source file doesn't exist. That maps exactly onto the normal race scenario of "another process already claimed it." NG: set -uo pipefail 環境では mv 失敗でスクリプトが非ゼロ終了する mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null OK: 失敗を明示的に「正常な退出」に変換する mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null || exit 0 2 /dev/null is needed as its counterpart. With || exit 0 alone, "No such file or directory" is printed to stderr . Since StandardErrorPath in com.lily.codex-note-funnel.plist points to ~/dev/note-autolike/logs/codex-funnel.error.log , every race piles a line that isn't actually an error into the error log. When you monitor logs, this becomes the cause of false alarms like "the error log is growing." The PID-check portion of autolike-plist-reconcile.sh has a subtlety that's easy to miss. php pid=$ launchctl list | awk -v l="$L" '$3==l{print $1}' if -n "$pid" && "$pid" = "-" ; then The first column returned by launchctl list is either a PID or - . - means the job is registered but not currently running. With -n "$pid" alone, a pid of - would also be judged as "has a PID." That's why "$pid" = "-" is joined with AND. This idiom of checking "the variable is non-empty" and "the value is meaningful" separately can be applied to the mv pattern too. If you want to add a defensive layer that verifies the WORKING file actually exists rather than looking only at the exit code of mv , write it like this: mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null || exit 0 念のため: mv が成功したはずなのにファイルがない（同一FSでない等） -f "$WORKING" || { echo "FATAL: mv succeeded but file missing" "$LOG"; exit 1; } This extra check is normally never reached, but it makes the symptom explicit if you accidentally run a cross-device mv . In an environment with set -uo pipefail active, having the cause in the log beats dying silently—debugging afterward is dramatically faster. The step at the top of the script that "returns processing/ files older than one hour to done/ " is a dead-letter countermeasure. age=$ $ date +%s - $ stat -f %m "$stale" if "$age" -gt 3600 ; then mv "$stale" "$DONE DIR/" 2 /dev/null fi If the process crashes after claiming ownership with mv , the file stays in processing/ . The EXIT trap is designed to return it to done/ , but the trap won't run in cases like a forced kill with signal 9. The symptom of "I check done/ the next morning and it's empty, yet nothing got processed" is caused by this pattern. By running recovery at startup, the file automatically re-enters the queue on the next launchd invocation. The threshold is 3600 seconds one hour so that legitimately long-running jobs don't get returned by mistake. Since StartCalendarInterval in com.lily.codex-note-funnel.plist has the two entries 10:40 and 16:40, the maximum schedule gap is six hours. One hour is a sufficient safety margin within that range. The day after implementing mv || exit 0 , I looked at codex-funnel.error.log and the line count had gone from 1 to 32. The processed count was increasing normally. The 4x duplicates were gone. And yet the error log was growing. At first I assumed "there must be an exception somewhere" and started debugging the main processing. After about 30 minutes of chasing it, I realized 2 /dev/null was missing. The code I first wrote was this: mv "$TARGET" "$PROCESSING DIR/$BASENAME" || exit 0 || only handles the exit code. It doesn't stop output to stderr . launchd keeps writing stderr to StandardErrorPath . As a result, "No such file or directory" accumulated at every moment of contention. The correct form is this: mv "$TARGET" "$PROCESSING DIR/$BASENAME" 2 /dev/null || exit 0 The cause was that I hadn't built the habit of always writing these two together. My preconception that "error log growing = bug in the main body" sent my debugging in the wrong direction. When the error log grows, the first thing to check is "am I silencing the normal failures?"—I should have made that order routine from the start. This happened in the version before I implemented the EXIT trap. Codex crashed overnight several times in a row, and when I checked the next morning, done/ was empty but processing wasn't happening. Tracing the symptom, five JSON files had accumulated in processing/ . Each time the script started, it checked done/ , found TARGET empty, and did exit 0 . But the "files that should be processed" were stuck in processing/ . done/ ← 空（次の仕事はここに来る） processing/ ← job 001.json, job 002.json ... ← ゾンビ processed/ ← 完了済み This state does double damage. First, processing stops. Second, "the fact that it's stuck isn't visible from outside." Because done/ is empty, the automation looks healthy. But even when new files arrive in done/ , nobody ever touches the zombies in processing/ . The fix combined two things: returning files via the EXIT trap and recovering stale processing/ files at startup. Either one alone is insufficient; to handle forced kills where the trap doesn't run, startup recovery is required. After implementing this, I only need to periodically ls -la the processing/ directory to confirm nothing is lingering, and this type of stall resolves itself by the next morning. This was the failure that burned the most time. Running run-codex-funnel.sh directly in the terminal processes JSON files normally. Via launchd, "processing: ..." never appears in codex-funnel.log . The error log is empty too. The script is starting, but doing nothing. What I used to pin down the cause was reproducing the shell launchd starts as faithfully as possible. env -i HOME="$HOME" PATH=/usr/bin:/bin:/usr/sbin:/sbin bash -l ~/dev/note-autolike/run-codex-funnel.sh Running it with env -i to minimize the environment variables, jq: command not found appeared partway through. The main processing used jq to read the needs imagegen thumbnail flag from the JSON, but jq was installed at /usr/local/bin/jq . With set -uo pipefail active, the script exits non-zero the moment jq isn't found. The exit code should have been written to codex-funnel.error.log , but since the error message itself never went to stderr no jq means no error text either , the log was empty. Had I read that EnvironmentVariables in com.lily.codex-note-funnel.plist was empty, I would have noticed sooner. The cause was skipping verification on the assumption that "PATH is probably fine" without actually reading the file. Since then, writing PATH= at the top has been an absolute rule for every script run via launchd. PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin export PATH Whether this one line is present is now the first thing I check in code review. ls .json Returned Exit Code 2 on "No Files" and Crashed the Script When done/ is empty, ls "$DONE DIR"/ .json fails shell glob expansion and returns exit code 2. In a set -uo pipefail environment, this causes the script to terminate abnormally. NG: done/ が空のとき ls が exit 2 → pipefail でスクリプト全体が落ちる TARGET=$ ls "$DONE DIR"/ .json | head -1 OK: 2 /dev/null でエラーを捨て、変数が空かどうかで判断する TARGET=$ ls "$DONE DIR"/ .json 2 /dev/null | head -1 -z "$TARGET" && exit 0 This problem wouldn't occur if the design treated an empty done/ as "abnormal." But this setup is designed so that "in normal operation, done/ is empty more often than not," so empty is the normal path. To "treat empty as normal," you need a line that explicitly absorbs the command's failure. Looking at how autolike-plist-reconcile.sh is written, there are three places where it uses continue to skip out of the loop. want=$ $PB -c "Print :EnvironmentVariables:AUTOLIKE TIMEOUT SEC" "$P" 2 /dev/null || continue have=$ launchctl print "$D/$L" 2 /dev/null | ... | head -1 -n "$have" || continue "$want" = "$have" && continue || continue appears throughout. The decision "if we can't get this lane's info, skip it" is written as moving on to the next iteration without stopping on the error. This idea of "converting a failure into an instruction for the next step" is the same philosophy as mv ... || exit 0 . Errors aren't something to leave in the log— they're a signal for the next action. Once I switched to this viewpoint, my scripts became dramatically clearer. funnel-pm Was Firing 4x The last "stuck" is less about technology and more about observability. I only noticed the 4x duplicate on September 17 when I counted the files in processed/ at 05:20 the next morning, on the 18th. Looking at the log, there were four entries for the same JSON, one minute apart. For the 17 hours the problem was happening, I knew nothing. The logs accumulate in ~/dev/note-autolike/logs/codex-funnel.log , but nobody monitors them in real time. The fact that "processing was duplicated" could only be detected from duplicate filenames in processed/ . I now use this duplicate-detection one-liner in my morning check: ls ~/dev/note-autolike/processed/ | sed 's/ 0-9 $//' | sort | uniq -d It strips the timestamp portion of the filenames and checks for duplicate base names. If it returns anything, that day's processing had a double execution. Structural improvement that drives duplicates to zero the mv pattern and, independently, monitoring that lets you notice duplicates happened—these are problems on different layers. Build a design that doesn't break, and also have a mechanism that lets you notice a break the same day rather than the next. In an environment where automation feeds real revenue, this two-tier approach is indispensable. launchd does not inherit PATH. Running with an empty EnvironmentVariables block will get you stuck. EnvironmentVariables in com.lily.codex-note-funnel.plist is empty, and the effective PATH at launchd startup is only /usr/bin:/bin:/usr/sbin:/sbin . The codex installed via nvm is not on this PATH. Even though run-codex-funnel.sh has set -uo pipefail , in a "command not found" situation there are cases where -u reacts late which codex returns an empty string . The fastest diagnosis is to start it manually with launchctl start com.lily.codex-note-funnel and check the logs. Nice: 10 and LowPriorityIO: true conflict with burst processing. Both plist settings deprioritize the script's execution when the system is under heavy load. When you want to churn through 57 JSON files right after Codex recovers, Nice: 10 is a "yield CPU to other user operations" setting, so processing is delayed. "Run quietly in the background" and "process at full speed after recovery" are in conflict. Decide which to prioritize and write the intent in a comment. The current plist chooses the former. The default ThrottleInterval depends on the launchd version. Accumulated non-zero exits can cause StartCalendarInterval scheduled starts to be skipped. This ties directly into failure 3 in part 2 throttling on exit 1 , but to add: if 10:40 is skipped due to throttling, the next scheduled run is 16:40, six hours later. The reliable diagnosis is to go back through days where nothing was written to codex-funnel.error.log StandardErrorPath . If you consistently use exit 0 , the default ThrottleInterval setting is fine, but if you're "leaving it to the default," record that intent in a plist comment or documentation. An mv failure due to a missing claimed/ directory and an mv failure due to race avoidance both return the same exit 0 . I touched on this in failure 5 of part 2, but let me restate it as the underlying design trade-off. Giving "normal race avoidance" and "abnormal environment misconfiguration" the same exit code prevents launchd's throttle, but delays discovery of problems. Since you're paying the price of this silence, external monitoring is mandatory. Cross-device mv is not atomic. POSIX rename 2 is serialized by the kernel only "on the same filesystem." If queue/ and claimed/ span different mount points e.g., NFS, Docker volumes, external drives , the operation decomposes internally into copy + delete, and a TOCTOU window opens. This setup has both directories under ~/dev/note-autolike/ , so there's no issue, but whenever you change the directory layout, verify the same filesystem with df -h . Filename sort order becomes unstable when files are created in the same second. The lexicographic sort in ls -1 "$QUEUE DIR"/ .json | sort | head -1 assumes the timestamp portions of the filenames differ. When batch submission creates multiple files within one second, same-name prefixes collide, and five lanes grab different files as the "first." As a result, no claim contention occurs, all lanes process simultaneously, and five Codex sessions start at once. The workaround is to include epoch seconds + PID suffix in the filename: job $ date +%s $$ $ uuidgen | cut -d- -f1 .json . Leftover files in claimed/ become a silent dead-letter queue. If processing crashes midway, the file stays in claimed/ . The next launchd start looks at queue/ , so files in claimed/ are never picked up again. In fact, after Codex recovered on September 17, 2026, there was a day when several files were stuck in claimed/ . Since adding monitoring that alerts when a file in claimed/ has gone more than an hour without an update, the lag before noticing this state has dropped to zero. The claude-quota-guard.py fd-inheritance issue remains even after the mv claim. The first entry in ProgramArguments in com.lily.codex-note-funnel.plist is ~/.claude/scripts/claude-quota-guard.py . While this Python wrapper launches run-codex-funnel.sh via subprocess.Popen , the job appears "running" to launchd. The PID check in autolike-plist-reconcile.sh verifying via launchctl list | awk -v l="$L" '$3==l{print $1}' that the PID is not - , which skips reload on the judgment "codex-funnel is running," correctly captures this "wrapper is alive" state as well. The mv claim eliminated the race, but you need to stay aware of how PIDs are read given that the wrapper launch is a precondition. The glob in reconcile.sh excludes codex-note-funnel . autolike-plist-reconcile.sh targets com.lily.autolike. .plist . com.lily.codex-note-funnel.plist is a separate file and doesn't match this glob. Since the automatic timeout-value reload that reconcile performs doesn't apply to codex-funnel, changing codex-funnel's timeout setting requires a manual launchctl bootout + bootstrap . If this is intentional, state it explicitly in a comment so that six months from now you won't be confused. No log rotation configured for codex-funnel.log . The plist's StandardOutPath points to ~/dev/note-autolike/logs/codex-funnel.log . launchd doesn't rotate logs. With dozens of lines per run accumulating continuously, it reaches tens of MB in a year. You need explicit rotation via newsyslog or logrotate , or line-count cap management inside the script. Without a mechanism to automatically detect "log not updating," you won't notice a stall for a week. I've separately added monitoring that fires an alert if the update timestamp of claimed/ hasn't changed within 15 minutes after the 10:40 start. The exit 0 design maximizes compatibility with launchd, but in exchange carries the silent risk that "normal exit and silent failure are indistinguishable." This monitoring script isn't optional—it should be installed as a pair with the mv claim pattern. Here are the rules that solidified after six months of running an autonomous environment at the ¥1.2M/month scale, each grounded in real code. 1. Put the mv ownership claim at the very top of the script Run mv before argument validation, log initialization, or directory checks. If you can't get the file, there's no need to run any subsequent processing at all. This minimizes CPU and memory usage per lane while resolving contention as fast as possible. TARGET=$ ls -1 "$QUEUE DIR"/ .json 2 /dev/null | sort | head -1 -z "$TARGET" && exit 0 BASENAME=$ basename "$TARGET" mv "${QUEUE DIR}/${BASENAME}" "${CLAIMED DIR}/${BASENAME}" 2 /dev/null || exit 0 ここより下は自プロセスだけが実行する 2. Log successful claims; keep failures race avoidance silent With "success = logged, race avoidance = silent," grep "claimed:" codex-funnel.log | wc -l becomes today's processed count. A count log that needs no filtering dramatically reduces the implementation cost of periodic monitoring scripts. Pairing 2 /dev/null with exit 0 is a consistent pattern. 3. Return exit 0 only for "normal race avoidance" and "empty queue" For "directory doesn't exist," "permission error," and "unexpected mv error," return exit 1 . The more silent exits you have, the later you discover failures. Set up the environment with an initial deploy script, then write the script body on the premise that "only normal exits are exit 0 ," and maintenance cost goes down. 4. Make state visible with the three directories queue/ , claimed/ , done/ When the directory itself represents state, ls ~/dev/note-autolike/claimed/ | wc -l tells you "the number of stuck files" at a glance. No flag management or DB needed— ls alone is your monitoring tool. 5. Put epoch seconds + PID in filenames to stabilize sort order FILENAME="job $ date +%s $$ $ uuidgen | cut -d- -f1 .json" $$ is the PID of the generating process. Even when multiple files are created in the same second, differing PIDs make the lexicographic order deterministic. This stabilizes the premise of the mv claim pattern: all five lanes select the same file as first. 6. Standardize the "record the reason for skipping" pattern from reconcile.sh across all scripts Lines 18–22 of autolike-plist-reconcile.sh log the reason and continue when a running job is found: php pid=$ launchctl list | awk -v l="$L" '$3==l{print $1}' if -n "$pid" && "$pid" = "-" ; then echo " $ date '+%F %T' $L 実行中 pid=$pid のため見送り $have - $want " "$LOG" continue fi The pattern of logging "why nothing was done" makes later failure tracing dramatically easier. Keep this habit even after the mv claim eliminates the race. 7. Separate post-crash recovery from the main script If you build the "return leftover claimed/ files to queue/ " step into the main script, a new race reappears where two processes simultaneously "return the leftover file → try to re-claim it." Stick to a design where recovery is delegated to an independent monitoring script that runs at low frequency. 8. Record the start time in a sidecar file right after claiming mv "${QUEUE DIR}/${BASENAME}" "${CLAIMED DIR}/${BASENAME}" 2 /dev/null || exit 0 echo "{\"claimed at\":\"$ date -u +%Y-%m-%dT%H:%M:%SZ \"}" \ "${CLAIMED DIR}/${BASENAME}.claim" Writing the start time to a .claim sidecar lets you write a recovery script that "returns files whose .claim is older than one hour to queue/ ." Aging detection for leftover files becomes precise. 9. Make the initial deploy script idempotent and re-runnable mkdir -p ~/dev/note-autolike/{queue,claimed,done,logs} Prepare a script that runs this before calling launchd's bootstrap . This structurally prevents a recurrence of failure 5 from part 2, where mv fails silently because a directory doesn't exist. 10. Separate log files between lanes to prevent interleaved lines The plist settings StandardOutPath: codex-funnel.log and StandardErrorPath: codex-funnel.error.log point to separate files per lane, and that's correct. In the post-recovery situation where five lanes start at once, if a shared log gets interleaved, reconstructing what happened becomes impossible. Including the label name com.lily.codex-note-funnel in the log filename is the minimum bar. 11. Be aware of ThrottleInterval behavior and make the setting's intent explicit If you consistently use exit 0 , launchd's default ThrottleInterval causes no problems. But consciously choose to "leave it at the default" and write the reason in the plist or operations docs. This prevents the confusion of "I didn't change any settings, so why?" when trouble hits six months later. 12. Verify the same filesystem with df -h before placing queue/ and claimed/ The biggest risk to the atomicity premise is a cross-device mv . Build the habit of checking whenever you change directories into the deploy script: queue dev=$ df -P "$QUEUE DIR" | awk 'NR==2{print $1}' claimed dev=$ df -P "$CLAIMED DIR" | awk 'NR==2{print $1}' "$queue dev" = "$claimed dev" || { echo "ERROR: cross-device mv"; exit 1; } The problem that occurred on September 17, 2026, after Codex recovered from usage limit exceeded —"the funnel-pm lane processed the same JSON 4 times, one minute apart"—was an inevitability of the design. When five lanes fire at once with 57 JSON files backed up, a pickup script without mutual exclusion collides structurally, not probabilistically. Because launchd is a scheduler, not a mutex. The fix was one line. mv "$TARGET" "$CLAIMED DIR/$BASENAME" 2 /dev/null || exit 0 POSIX rename 2 is serialized by the kernel on the same filesystem. Fundamentally unlike a "check, then acquire" PID check, the check and the acquisition complete within a single system call. Unlike flock with its fd-inheritance problem, the lock is decoupled from the life and death of the process. Since the file itself becomes the claim token, a leftover file after a crash remains in a state that's human-readable. The "do nothing if running" PID-check pattern in autolike-plist-reconcile.sh and the mv claim pattern here were born from the same philosophy. The only difference is "whether there's a TOCTOU window." Both are designs that make the choice "exit if someone else is processing" in the very first operation. To keep an autonomous environment stable over the long term, the only way is to keep stacking up these kinds of "structures that don't permit contention in the first place." Procedural fixes shrink the wound; environmental fixes eliminate the event. The reason I could go from ¥0 to ¥1.2M a month in six months is that on the days things broke, I repeatedly chose "fix the structure" over "next time I'll do it differently." The full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day setup guide are compiled in a paid note Japanese . 📕 How to actually earn with a Claude Code autonomous environment — the system, real examples, getting started, and support https://note.com/bokuwalily/n/n849b3a07784a Written by Lily — I ship iOS apps and automate my content stack with Claude Code. Follow along: Portfolio https://bokuwalily.com · X https://x.com/bokuwalily · GitHub https://github.com/bokuwalily