{"slug": "one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue", "title": "One Line of mv Kills the 4x Duplicate Run After Recovery: Making launchd Queue Processing Idempotent", "summary": "A developer running an autonomous side business on Claude Code documented how a day-long Codex outage caused 57 queued JSON files to pile up, and when the service recovered, five launchd lanes fired simultaneously and processed the same file four times. The engineer traced the duplicate runs to launchd's lack of mutual exclusion over a shared file queue, noting that a check-then-act lock file is not atomic and that scheduler-plus-shared-directory designs will always collide unless the pickup script implements explicit locking.", "body_md": "Six months ago my side business made ¥0 a month. This month it's ¥1.2M, running on an autonomous setup built with Claude Code. And here's the thing about automation: the moment it breaks is exactly when the sloppiness in your design gets exposed.\n\nWhen automation breaks, the reflex is to reach for a procedural fix: \"next time I'll do this instead.\" Check the logs more often. Retry by hand. Space things out a bit more. All of these are bandages meant to shrink the wound the next time the problem recurs. They don't remove the cause, so once the same conditions line up, it will happen again.\n\nAfter six months of running an automated system, one thing I'm certain of: **a scheduler combined with a shared directory will always collide unless you explicitly implement mutual exclusion.** Not \"it might collide if you're unlucky,\" but \"it will collide whenever the conditions for collision line up.\" This isn't a probability problem; it's a structural one.\n\nOn September 17, 2026, Codex was down for an entire day with `usage_limit_exceeded`. Under `~/.codex/sessions/2026/09/17/` there were 54 sessions piled up, every one of them terminated with a limit error. Throughout that time, three workflows—note-autolike, ai-portraits-fragments, and social-autolike—kept firing jobs, and unprocessed JSON files accumulated in `~/dev/note-autolike/done/`. Measured the next morning at 05:20, the JSON files flagged `needs_imagegen_thumbnail` had swelled **from 43 to 57**. Fourteen new files had stacked up, all of them left untouched.\n\nThe instant Codex came back, five launchd lanes fired at once. Each lane's pickup script grabs the JSON files under `done/` with `ls` or a glob and starts processing. With no mutual exclusion, multiple processes look at the same JSON at the same time. The result: the `funnel-pm` lane processed the same JSON **4 times, one minute apart**. As the numbers show—8 of 11 files were hit 2 or more times—this wasn't a freak accident. It was an inevitable consequence of the design.\n\nLook at `StartCalendarInterval` in `com.lily.codex-note-funnel.plist` and you'll see two entries: 10:40 and 16:40.\n\n```\n<key>StartCalendarInterval</key>\n<array>\n    <dict>\n        <key>Hour</key>\n        <integer>10</integer>\n        <key>Minute</key>\n        <integer>40</integer>\n    </dict>\n    <dict>\n        <key>Hour</key>\n        <integer>16</integer>\n        <key>Minute</key>\n        <integer>40</integer>\n    </dict>\n</array>\n```\n\nlaunchd simply invokes `run-codex-funnel.sh` according to this schedule. Whether the previous run finished, or whether another lane is currently working on the same file—launchd has no interest in any of it. And that's correct, by-design behavior. launchd is a scheduler, not a mutex. Mutual exclusion over a file queue is the application layer's responsibility—in other words, it's a feature the pickup script itself has to implement.\n\nIn normal operation, each lane's timing is slightly offset, which naturally keeps collisions rare. Target JSON files also arrive one at a time in sequence, so there are few opportunities for multiple lanes to see the same file simultaneously.\n\nBut a long outage like Codex's `usage_limit_exceeded` changes the situation completely. All lanes start at once with 57 JSON files backed up from the outage period.\n\n```\n通常運転:\n  レーンA → job_001 処理 → 完了\n  レーンB → job_002 処理 → 完了  （タイミングがズレており競合しない）\n\nCodex 復帰直後:\n  レーンA ─┐\n  レーンB ─┤──→ job_001 を同時取得 → 4連射\n  funnel-pm ─┘\n  （57本が溜まっており、全レーンが先頭ファイルに殺到）\n```\n\nA procedural fix like \"adjust the intervals\" can't handle this \"burst release\" pattern. No matter how tightly you tune the intervals, the instant after recovery, every lane will fire simultaneously.\n\n\"Just create a lock file\" is the natural idea. But look closely at the implementation and there's a problem.\n\n```\n# NG: check-then-act は原子的でない\nLOCKFILE=~/dev/note-autolike/.processing.lock\nif [ ! -f \"$LOCKFILE\" ]; then\n    touch \"$LOCKFILE\"\n    process_file \"$json\"\n    rm \"$LOCKFILE\"\nfi\n```\n\nBetween the `if [ ! -f ]` check and the `touch`, there's a \"Time of Check to Time of Use\" (TOCTOU) window where another process can slip in. Two processes both observe \"no file\" and both execute `touch`—that's the race. Under high load and high frequency, this window really does open.\n\nOn top of that, if the process crashes mid-run, the lock file stays behind. Every subsequent run sees the lock and exits immediately, so processing stalls completely. You need a separate crash-recovery procedure, and operational cost goes up.\n\nFile locking via the `flock` command is another option, but since file descriptors aren't inherited each time launchd spawns a new process, it doesn't work for exclusion between lanes.\n\nThe existing `~/.claude/scripts/autolike-plist-reconcile.sh` contains a guard built with the same concern in mind. That script does a `reload` when the `AUTOLIKE_TIMEOUT_SEC` value in the plist has drifted from the value launchd has loaded, but the core is this code:\n\n``` php\npid=$(launchctl list | awk -v l=\"$L\" '$3==l{print $1}')\nif [ -n \"$pid\" ] && [ \"$pid\" != \"-\" ]; then\n    echo \"[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)\" >>\"$LOG\"\n    continue\nfi\nlaunchctl bootout \"$D/$L\" 2>/dev/null\n```\n\nThe comment at the top of the script says that booting out a running job wipes out that run's likes entirely, so it must always wait. It fetches the PID via `launchctl list` and skips the bootout unless the value is `-` (not running). Before starting work, check \"is someone else already processing this?\"—and if so, exit immediately. That's the same idea as the `mv` pattern in this article.\n\nThat said, a PID check has limits. You can't reduce to zero the chance that a new process starts between the \"check\" and the \"bootout.\" In the context where this script runs, that's acceptable, but for claiming exclusive ownership of a file queue, it's insufficient. You need a stronger atomic operation.\n\n```\nCodex usage_limit_exceeded から復帰\n              │\n              ▼\n    ~/dev/note-autolike/done/ に 57 本が滞留\n    ┌────────────────────────────────────────┐\n    │  job_20260917_001.json                │\n    │  job_20260917_002.json                │\n    │  ...                                  │\n    │  job_20260917_057.json                │\n    └────────────────────────────────────────┘\n              │\n    5 レーンが launchd により一斉起動\n    ┌────────────────────────────────────────┐\n    │ com.lily.codex-note-funnel   (10:40)  │\n    │ com.lily.autolike.note1               │\n    │ com.lily.autolike.note2               │\n    │ com.lily.autolike.funnel-pm  ← 4連射  │\n    │ com.lily.autolike.social              │\n    └────────────────────────────────────────┘\n              │\n    【修正前】各レーンが done/ を glob → 先頭ファイルを取得\n              │  排他なし → 全レーンが job_001 を同時に read\n              ↓\n    funnel-pm が 1分差で同一 JSON を 4 回処理\n    ─────────────────────────────────────────\n    【修正後】mv による原子的所有権取得\n    ┌────────────────────────────────────────┐\n    │  mv done/job_001.json processing/     │\n    │  ├─ 成功: 自プロセスのみが処理を継続   │\n    │  └─ 失敗: 他プロセスが取得済み → exit 0│\n    └────────────────────────────────────────┘\n              │\n    「1ファイル = 1プロセスのみ処理」が保証される\n```\n\nOn the same filesystem, `mv` executes a single POSIX `rename(2)` system call. Because the \"rename this file\" operation completes indivisibly at the kernel level, even if two processes simultaneously run `mv done/job_001.json processing/job_001.json`, **exactly one succeeds and the other is guaranteed to fail.** No window opens between \"check\" and \"acquire.\"\n\nThis is the fundamental difference between a PID check and `mv`. With a PID check, \"check\" and \"start processing\" are separate operations. With `mv`, \"check\" and \"acquire\" are completed inside one system call.\n\n```\n# 2プロセスが同時に実行した場合の挙動\n# プロセスA: mv done/job_001.json processing/job_001.json → 成功（終了コード 0）\n# プロセスB: mv done/job_001.json processing/job_001.json → 失敗（No such file or directory, 終了コード 1）\n```\n\nHere's the implementation to insert at the top of `run-codex-funnel.sh`.\n\n``` bash\n#!/bin/bash\nset -uo pipefail\n\nDONE_DIR=~/dev/note-autolike/done\nPROCESSING_DIR=~/dev/note-autolike/processing\nmkdir -p \"$PROCESSING_DIR\"\n\n# done/ から処理対象を1本選ぶ\nTARGET=$(ls \"$DONE_DIR\"/*.json 2>/dev/null | head -1)\n[ -z \"$TARGET\" ] && exit 0          # 対象なし → 正常終了\n\nBASENAME=$(basename \"$TARGET\")\n\n# アトミックに所有権を取得。失敗 = 他プロセスが取得済み\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null || exit 0\n\nWORKING=\"$PROCESSING_DIR/$BASENAME\"\n# ここから先は自プロセスだけが処理する権利を持つ\n```\n\nThere's effectively only one situation in which `mv` fails: another process has already `mv`'d the file and it no longer exists. That is precisely the race condition itself, so exiting immediately with `exit 0` is the correct response. You don't even need to write an error log.\n\nA cross-device `mv` (spanning different mount points) is not atomic, so be careful—but in this setup, both `done/` and `processing/` live under `~/dev/note-autolike/`, which guarantees the same filesystem.\n\nThe `ProgramArguments` in `com.lily.codex-note-funnel.plist` are structured like this:\n\n```\n<key>ProgramArguments</key>\n<array>\n    <string>~/.claude/scripts/claude-quota-guard.py</string>\n    <string>--job</string>\n    <string>com.lily.codex-note-funnel</string>\n    <string>--priority</string>\n    <string>--</string>\n    <string>~/dev/note-autolike/run-codex-funnel.sh</string>\n</array>\n```\n\n`claude-quota-guard.py` runs first as a quota-control wrapper, and if there's headroom, it launches `run-codex-funnel.sh`. Since the pickup logic lives inside `run-codex-funnel.sh`, inserting the `mv` ownership claim into the first few lines of the script is all it takes. No plist changes required.\n\nlaunchd will keep calling `run-codex-funnel.sh` at 10:40 and 16:40. Even with 57 JSON files backed up right after Codex recovers from `usage_limit_exceeded`, every time `run-codex-funnel.sh` starts it repeats the same behavior: \"first claim one file with `mv`; if that fails, exit immediately.\" Even if five processes, including the funnel-pm lane, run at the same time, exactly one process handles each file.\n\nThe first half showed only the single `mv` line, but to actually run it in production you need layers before and after. Looking at `ProgramArguments` in `com.lily.codex-note-funnel.plist`, `claude-quota-guard.py` runs as the leading wrapper, and then `run-codex-funnel.sh` is called. The full script looks like this:\n\n``` bash\n#!/bin/bash\n# launchd 環境の PATH は最小限のため先頭で上書きする\nPATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin\nexport PATH\n\nset -uo pipefail\n\nDONE_DIR=~/dev/note-autolike/done\nPROCESSING_DIR=~/dev/note-autolike/processing\nPROCESSED_DIR=~/dev/note-autolike/processed\nLOG=~/dev/note-autolike/logs/codex-funnel.log\n\nmkdir -p \"$PROCESSING_DIR\" \"$PROCESSED_DIR\"\n\n# 起動時リカバリ: 前回クラッシュで processing/ に残ったファイルを done/ へ戻す\nfor stale in \"$PROCESSING_DIR\"/*.json; do\n    [ -f \"$stale\" ] || break\n    age=$(( $(date +%s) - $(stat -f %m \"$stale\") ))\n    if [ \"$age\" -gt 3600 ]; then\n        mv \"$stale\" \"$DONE_DIR/\" 2>/dev/null\n        echo \"[$(date '+%F %T')] recovered stale: $(basename \"$stale\")\" >> \"$LOG\"\n    fi\ndone\n\n# done/ から処理対象を1本選ぶ\nTARGET=$(ls \"$DONE_DIR\"/*.json 2>/dev/null | head -1)\n[ -z \"$TARGET\" ] && exit 0\n\nBASENAME=$(basename \"$TARGET\")\n\n# ─── ここが核心 ─────────────────────────────────────────\n# アトミックに所有権を取得。失敗 = 他プロセスが取得済み → 即退出\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null || exit 0\n# ────────────────────────────────────────────────────────\n\nWORKING=\"$PROCESSING_DIR/$BASENAME\"\n\n# クラッシュ時に processing/ へファイルが残るのを防ぐ EXIT トラップ\n# 正常完了時は processed/ へ移動、異常時は done/ へ差し戻す\n_cleanup() {\n    local code=$?\n    if [ $code -eq 0 ]; then\n        mv \"$WORKING\" \"$PROCESSED_DIR/$BASENAME\" 2>/dev/null\n    else\n        mv \"$WORKING\" \"$DONE_DIR/$BASENAME\" 2>/dev/null\n        echo \"[$(date '+%F %T')] ERROR exit=$code, returned: $BASENAME\" >> \"$LOG\"\n    fi\n}\ntrap '_cleanup' EXIT\n\n# ── ここから先は自プロセスだけが $WORKING を処理する ──\necho \"[$(date '+%F %T')] processing: $BASENAME\" >> \"$LOG\"\n# ... 本体処理 ...\n```\n\nLook at `EnvironmentVariables` in `com.lily.codex-note-funnel.plist`: the key exists, but the content is an empty `<dict/>`.\n\n```\n<key>EnvironmentVariables</key>\n<dict>\n</dict>\n```\n\nWhen launchd starts a process, its PATH is only the minimal set `/usr/bin:/bin:/usr/sbin:/sbin`. If you `echo $PATH` in a terminal you'll see `/usr/local/bin` included, but that's added by `.zshrc` or the nvm initialization script. launchd doesn't run any such shell initialization. Tools installed in `/usr/local/bin` like `jq`, `node`, and `python3` fail silently as \"command not found\" in scripts launched from a plist. **The fix is one of two choices: write it in the plist's `EnvironmentVariables`, or hardcode it at the top of the script.** The latter is more convenient because you can verify the script's behavior on its own, so I've standardized on `export PATH` at the top.\n\n`set -uo pipefail` and `|| exit 0` Must Go Together\nWhen `set -uo pipefail` is active, the whole script aborts the moment a command returns a non-zero exit code. `mv` returns exit code 1 when the source file doesn't exist. That maps exactly onto the normal race scenario of \"another process already claimed it.\"\n\n```\n# NG: set -uo pipefail 環境では mv 失敗でスクリプトが非ゼロ終了する\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null\n\n# OK: 失敗を明示的に「正常な退出」に変換する\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null || exit 0\n```\n\n`2>/dev/null` is needed as its counterpart. With `|| exit 0` alone, \"No such file or directory\" is printed to `stderr`. Since `StandardErrorPath` in `com.lily.codex-note-funnel.plist` points to `~/dev/note-autolike/logs/codex-funnel.error.log`, every race piles a line that isn't actually an error into the error log. When you monitor logs, this becomes the cause of false alarms like \"the error log is growing.\"\n\nThe PID-check portion of `autolike-plist-reconcile.sh` has a subtlety that's easy to miss.\n\n``` php\npid=$(launchctl list | awk -v l=\"$L\" '$3==l{print $1}')\nif [ -n \"$pid\" ] && [ \"$pid\" != \"-\" ]; then\n```\n\nThe first column returned by `launchctl list` is either a PID or `-`. `-` means the job is registered but not currently running. With `[ -n \"$pid\" ]` alone, a `pid` of `-` would also be judged as \"has a PID.\" That's why `[ \"$pid\" != \"-\" ]` is joined with AND.\n\nThis idiom of checking \"the variable is non-empty\" and \"the value is meaningful\" separately can be applied to the `mv` pattern too. If you want to add a defensive layer that verifies the `WORKING` file actually exists rather than looking only at the exit code of `mv`, write it like this:\n\n```\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null || exit 0\n# 念のため: mv が成功したはずなのにファイルがない（同一FSでない等）\n[ -f \"$WORKING\" ] || { echo \"FATAL: mv succeeded but file missing\" >> \"$LOG\"; exit 1; }\n```\n\nThis extra check is normally never reached, but it makes the symptom explicit if you accidentally run a cross-device `mv`. In an environment with `set -uo pipefail` active, having the cause in the log beats dying silently—debugging afterward is dramatically faster.\n\nThe step at the top of the script that \"returns `processing/` files older than one hour to `done/`\" is a dead-letter countermeasure.\n\n```\nage=$(( $(date +%s) - $(stat -f %m \"$stale\") ))\nif [ \"$age\" -gt 3600 ]; then\n    mv \"$stale\" \"$DONE_DIR/\" 2>/dev/null\nfi\n```\n\nIf the process crashes after claiming ownership with `mv`, the file stays in `processing/`. The EXIT trap is designed to return it to `done/`, but the trap won't run in cases like a forced kill with signal 9. The symptom of \"I check `done/` the next morning and it's empty, yet nothing got processed\" is caused by this pattern. By running recovery at startup, the file automatically re-enters the queue on the next launchd invocation. The threshold is 3600 seconds (one hour) so that legitimately long-running jobs don't get returned by mistake. Since `StartCalendarInterval` in `com.lily.codex-note-funnel.plist` has the two entries 10:40 and 16:40, the maximum schedule gap is six hours. One hour is a sufficient safety margin within that range.\n\nThe day after implementing `mv || exit 0`, I looked at `codex-funnel.error.log` and the line count had gone from 1 to 32. The processed count was increasing normally. The 4x duplicates were gone. And yet the error log was growing.\n\nAt first I assumed \"there must be an exception somewhere\" and started debugging the main processing. After about 30 minutes of chasing it, I realized `2>/dev/null` was missing. The code I first wrote was this:\n\n```\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" || exit 0\n```\n\n`||` only handles the exit code. It doesn't stop output to `stderr`. launchd keeps writing `stderr` to `StandardErrorPath`. As a result, \"No such file or directory\" accumulated at every moment of contention.\n\nThe correct form is this:\n\n```\nmv \"$TARGET\" \"$PROCESSING_DIR/$BASENAME\" 2>/dev/null || exit 0\n```\n\nThe cause was that I hadn't built the habit of always writing these two together. My preconception that \"error log growing = bug in the main body\" sent my debugging in the wrong direction. **When the error log grows, the first thing to check is \"am I silencing the normal failures?\"—I should have made that order routine from the start.**\n\nThis happened in the version before I implemented the EXIT trap. Codex crashed overnight several times in a row, and when I checked the next morning, `done/` was empty but processing wasn't happening.\n\nTracing the symptom, five JSON files had accumulated in `processing/`. Each time the script started, it checked `done/`, found `TARGET` empty, and did `exit 0`. But the \"files that should be processed\" were stuck in `processing/`.\n\n```\ndone/       ← 空（次の仕事はここに来る）\nprocessing/ ← job_001.json, job_002.json ...  ← ゾンビ\nprocessed/  ← 完了済み\n```\n\nThis state does double damage. First, processing stops. Second, \"the fact that it's stuck isn't visible from outside.\" Because `done/` is empty, the automation looks healthy. But even when new files arrive in `done/`, nobody ever touches the zombies in `processing/`.\n\nThe fix combined two things: **returning files via the EXIT trap** and **recovering stale `processing/` files at startup.** Either one alone is insufficient; to handle forced kills where the trap doesn't run, startup recovery is required. After implementing this, I only need to periodically `ls -la` the `processing/` directory to confirm nothing is lingering, and this type of stall resolves itself by the next morning.\n\nThis was the failure that burned the most time. Running `run-codex-funnel.sh` directly in the terminal processes JSON files normally. Via launchd, \"processing: ...\" never appears in `codex-funnel.log`. The error log is empty too. The script is starting, but doing nothing.\n\nWhat I used to pin down the cause was reproducing the shell launchd starts as faithfully as possible.\n\n```\nenv -i HOME=\"$HOME\" PATH=/usr/bin:/bin:/usr/sbin:/sbin bash -l ~/dev/note-autolike/run-codex-funnel.sh\n```\n\nRunning it with `env -i` to minimize the environment variables, `jq: command not found` appeared partway through. The main processing used `jq` to read the `needs_imagegen_thumbnail` flag from the JSON, but `jq` was installed at `/usr/local/bin/jq`. With `set -uo pipefail` active, the script exits non-zero the moment `jq` isn't found. The exit code should have been written to `codex-funnel.error.log`, but since the error message itself never went to `stderr` (no `jq` means no error text either), the log was empty.\n\nHad I read that `EnvironmentVariables` in `com.lily.codex-note-funnel.plist` was empty, I would have noticed sooner. The cause was skipping verification on the assumption that \"PATH is probably fine\" without actually reading the file. Since then, **writing `PATH=` at the top** has been an absolute rule for every script run via launchd.\n\n```\nPATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin\nexport PATH\n```\n\nWhether this one line is present is now the first thing I check in code review.\n\n`ls *.json` Returned Exit Code 2 on \"No Files\" and Crashed the Script\nWhen `done/` is empty, `ls \"$DONE_DIR\"/*.json` fails shell glob expansion and returns exit code 2. In a `set -uo pipefail` environment, this causes the script to terminate abnormally.\n\n```\n# NG: done/ が空のとき ls が exit 2 → pipefail でスクリプト全体が落ちる\nTARGET=$(ls \"$DONE_DIR\"/*.json | head -1)\n\n# OK: 2>/dev/null でエラーを捨て、変数が空かどうかで判断する\nTARGET=$(ls \"$DONE_DIR\"/*.json 2>/dev/null | head -1)\n[ -z \"$TARGET\" ] && exit 0\n```\n\nThis problem wouldn't occur if the design treated an empty `done/` as \"abnormal.\" But this setup is designed so that \"in normal operation, `done/` is empty more often than not,\" so empty is the normal path. **To \"treat empty as normal,\" you need a line that explicitly absorbs the command's failure.**\n\nLooking at how `autolike-plist-reconcile.sh` is written, there are three places where it uses `continue` to skip out of the loop.\n\n```\nwant=$($PB -c \"Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC\" \"$P\" 2>/dev/null) || continue\nhave=$(launchctl print \"$D/$L\" 2>/dev/null | ...) | head -1)\n[ -n \"$have\" ] || continue\n[ \"$want\" = \"$have\" ] && continue\n```\n\n`|| continue` appears throughout. The decision \"if we can't get this lane's info, skip it\" is written as moving on to the next iteration without stopping on the error. This idea of \"converting a failure into an instruction for the next step\" is the same philosophy as `mv ... || exit 0`. Errors aren't something to leave in the log—**they're a signal for the next action.** Once I switched to this viewpoint, my scripts became dramatically clearer.\n\n`funnel-pm` Was Firing 4x\nThe last \"stuck\" is less about technology and more about observability. I only noticed the 4x duplicate on September 17 when I counted the files in `processed/` at 05:20 the next morning, on the 18th. Looking at the log, there were four entries for the same JSON, one minute apart.\n\nFor the 17 hours the problem was happening, I knew nothing. The logs accumulate in `~/dev/note-autolike/logs/codex-funnel.log`, but nobody monitors them in real time. The fact that \"processing was duplicated\" could only be detected from duplicate filenames in `processed/`.\n\nI now use this **duplicate-detection one-liner** in my morning check:\n\n```\nls ~/dev/note-autolike/processed/ | sed 's/_[0-9]*$//' | sort | uniq -d\n```\n\nIt strips the timestamp portion of the filenames and checks for duplicate base names. If it returns anything, that day's processing had a double execution.\n\nStructural improvement that drives duplicates to zero (the `mv` pattern) and, independently, monitoring that lets you notice duplicates happened—these are problems on different layers. Build a design that doesn't break, and also have a mechanism that lets you notice a break the same day rather than the next. In an environment where automation feeds real revenue, this two-tier approach is indispensable.\n\n**launchd does not inherit PATH. Running with an empty `EnvironmentVariables` block will get you stuck.** `EnvironmentVariables` in `com.lily.codex-note-funnel.plist` is empty, and the effective PATH at launchd startup is only `/usr/bin:/bin:/usr/sbin:/sbin`. The codex installed via nvm is not on this PATH. Even though `run-codex-funnel.sh` has `set -uo pipefail`, in a \"command not found\" situation there are cases where `-u` reacts late (`which codex` returns an empty string). The fastest diagnosis is to start it manually with `launchctl start com.lily.codex-note-funnel` and check the logs.\n\n**`Nice: 10` and `LowPriorityIO: true` conflict with burst processing.** Both plist settings deprioritize the script's execution when the system is under heavy load. When you want to churn through 57 JSON files right after Codex recovers, `Nice: 10` is a \"yield CPU to other user operations\" setting, so processing is delayed. \"Run quietly in the background\" and \"process at full speed after recovery\" are in conflict. Decide which to prioritize and write the intent in a comment. The current plist chooses the former.\n\n**The default `ThrottleInterval` depends on the launchd version. Accumulated non-zero exits can cause `StartCalendarInterval` scheduled starts to be skipped.** This ties directly into failure #3 in part 2 (throttling on `exit 1`), but to add: if 10:40 is skipped due to throttling, the next scheduled run is 16:40, six hours later. The reliable diagnosis is to go back through days where nothing was written to `codex-funnel.error.log` (`StandardErrorPath`). If you consistently use `exit 0`, the default `ThrottleInterval` setting is fine, but if you're \"leaving it to the default,\" record that intent in a plist comment or documentation.\n\n**An `mv` failure due to a missing `claimed/` directory and an `mv` failure due to race avoidance both return the same `exit 0`.** I touched on this in failure #5 of part 2, but let me restate it as the underlying design trade-off. Giving \"normal race avoidance\" and \"abnormal environment misconfiguration\" the same exit code prevents launchd's throttle, but delays discovery of problems. Since you're paying the price of this silence, external monitoring is mandatory.\n\n**Cross-device `mv` is not atomic.** `POSIX rename(2)` is serialized by the kernel only \"on the same filesystem.\" If `queue/` and `claimed/` span different mount points (e.g., NFS, Docker volumes, external drives), the operation decomposes internally into copy + delete, and a TOCTOU window opens. This setup has both directories under `~/dev/note-autolike/`, so there's no issue, but whenever you change the directory layout, verify the same filesystem with `df -h`.\n\n**Filename sort order becomes unstable when files are created in the same second.** The lexicographic sort in `ls -1 \"$QUEUE_DIR\"/*.json | sort | head -1` assumes the timestamp portions of the filenames differ. When batch submission creates multiple files within one second, same-name prefixes collide, and five lanes grab different files as the \"first.\" As a result, no claim contention occurs, all lanes process simultaneously, and five Codex sessions start at once. The workaround is to include epoch seconds + PID suffix in the filename: `job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json`.\n\n**Leftover files in `claimed/` become a silent dead-letter queue.** If processing crashes midway, the file stays in `claimed/`. The next launchd start looks at `queue/`, so files in `claimed/` are never picked up again. In fact, after Codex recovered on September 17, 2026, there was a day when several files were stuck in `claimed/`. Since adding monitoring that alerts when a file in `claimed/` has gone more than an hour without an update, the lag before noticing this state has dropped to zero.\n\n**The `claude-quota-guard.py` fd-inheritance issue remains even after the `mv` claim.** The first entry in `ProgramArguments` in `com.lily.codex-note-funnel.plist` is `~/.claude/scripts/claude-quota-guard.py`. While this Python wrapper launches `run-codex-funnel.sh` via `subprocess.Popen`, the job appears \"running\" to launchd. The PID check in `autolike-plist-reconcile.sh` (verifying via `launchctl list | awk -v l=\"$L\" '$3==l{print $1}'` that the PID is not `-`), which skips reload on the judgment \"codex-funnel is running,\" correctly captures this \"wrapper is alive\" state as well. The `mv` claim eliminated the race, but you need to stay aware of how PIDs are read given that the wrapper launch is a precondition.\n\n**The glob in `reconcile.sh` excludes `codex-note-funnel`.** `autolike-plist-reconcile.sh` targets `com.lily.autolike.*.plist`. `com.lily.codex-note-funnel.plist` is a separate file and doesn't match this glob. Since the automatic timeout-value reload that reconcile performs doesn't apply to codex-funnel, changing codex-funnel's timeout setting requires a manual `launchctl bootout` + `bootstrap`. If this is intentional, state it explicitly in a comment so that six months from now you won't be confused.\n\n**No log rotation configured for `codex-funnel.log`.** The plist's `StandardOutPath` points to `~/dev/note-autolike/logs/codex-funnel.log`. launchd doesn't rotate logs. With dozens of lines per run accumulating continuously, it reaches tens of MB in a year. You need explicit rotation via `newsyslog` or `logrotate`, or line-count cap management inside the script.\n\n**Without a mechanism to automatically detect \"log not updating,\" you won't notice a stall for a week.** I've separately added monitoring that fires an alert if the update timestamp of `claimed/` hasn't changed within 15 minutes after the 10:40 start. The `exit 0` design maximizes compatibility with launchd, but in exchange carries the silent risk that \"normal exit and silent failure are indistinguishable.\" This monitoring script isn't optional—it should be installed as a pair with the mv claim pattern.\n\nHere are the rules that solidified after six months of running an autonomous environment at the ¥1.2M/month scale, each grounded in real code.\n\n**1. Put the `mv` ownership claim at the very top of the script**\n\nRun `mv` before argument validation, log initialization, or directory checks. If you can't get the file, there's no need to run any subsequent processing at all. This minimizes CPU and memory usage per lane while resolving contention as fast as possible.\n\n```\nTARGET=$(ls -1 \"$QUEUE_DIR\"/*.json 2>/dev/null | sort | head -1)\n[ -z \"$TARGET\" ] && exit 0\nBASENAME=$(basename \"$TARGET\")\nmv \"${QUEUE_DIR}/${BASENAME}\" \"${CLAIMED_DIR}/${BASENAME}\" 2>/dev/null || exit 0\n# ここより下は自プロセスだけが実行する\n```\n\n**2. Log successful claims; keep failures (race avoidance) silent**\n\nWith \"success = logged, race avoidance = silent,\" `grep \"claimed:\" codex-funnel.log | wc -l` becomes today's processed count. A count log that needs no filtering dramatically reduces the implementation cost of periodic monitoring scripts. Pairing `2>/dev/null` with `exit 0` is a consistent pattern.\n\n**3. Return `exit 0` only for \"normal race avoidance\" and \"empty queue\"**\n\nFor \"directory doesn't exist,\" \"permission error,\" and \"unexpected mv error,\" return `exit 1`. The more silent exits you have, the later you discover failures. Set up the environment with an initial deploy script, then write the script body on the premise that \"only normal exits are `exit 0`,\" and maintenance cost goes down.\n\n**4. Make state visible with the three directories `queue/`, `claimed/`, `done/`**\n\nWhen the directory itself represents state, `ls ~/dev/note-autolike/claimed/ | wc -l` tells you \"the number of stuck files\" at a glance. No flag management or DB needed—`ls` alone is your monitoring tool.\n\n**5. Put epoch seconds + PID in filenames to stabilize sort order**\n\n```\nFILENAME=\"job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json\"\n```\n\n`$$` is the PID of the generating process. Even when multiple files are created in the same second, differing PIDs make the lexicographic order deterministic. This stabilizes the premise of the mv claim pattern: all five lanes select the same file as first.\n\n**6. Standardize the \"record the reason for skipping\" pattern from `reconcile.sh` across all scripts**\n\nLines 18–22 of `autolike-plist-reconcile.sh` log the reason and `continue` when a running job is found:\n\n``` php\npid=$(launchctl list | awk -v l=\"$L\" '$3==l{print $1}')\nif [ -n \"$pid\" ] && [ \"$pid\" != \"-\" ]; then\n  echo \"[$(date '+%F %T')] $L 実行中(pid=$pid) のため見送り ($have -> $want)\" >>\"$LOG\"\n  continue\nfi\n```\n\nThe pattern of logging \"why nothing was done\" makes later failure tracing dramatically easier. Keep this habit even after the mv claim eliminates the race.\n\n**7. Separate post-crash recovery from the main script**\n\nIf you build the \"return leftover `claimed/` files to `queue/`\" step into the main script, a new race reappears where two processes simultaneously \"return the leftover file → try to re-claim it.\" Stick to a design where recovery is delegated to an independent monitoring script that runs at low frequency.\n\n**8. Record the start time in a sidecar file right after claiming**\n\n```\nmv \"${QUEUE_DIR}/${BASENAME}\" \"${CLAIMED_DIR}/${BASENAME}\" 2>/dev/null || exit 0\necho \"{\\\"claimed_at\\\":\\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\"}\" \\\n  > \"${CLAIMED_DIR}/${BASENAME}.claim\"\n```\n\nWriting the start time to a `.claim` sidecar lets you write a recovery script that \"returns files whose `.claim` is older than one hour to `queue/`.\" Aging detection for leftover files becomes precise.\n\n**9. Make the initial deploy script idempotent and re-runnable**\n\n```\nmkdir -p ~/dev/note-autolike/{queue,claimed,done,logs}\n```\n\nPrepare a script that runs this before calling launchd's `bootstrap`. This structurally prevents a recurrence of failure #5 from part 2, where `mv` fails silently because a directory doesn't exist.\n\n**10. Separate log files between lanes to prevent interleaved lines**\n\nThe plist settings `StandardOutPath: codex-funnel.log` and `StandardErrorPath: codex-funnel.error.log` point to separate files per lane, and that's correct. In the post-recovery situation where five lanes start at once, if a shared log gets interleaved, reconstructing what happened becomes impossible. Including the label name (`com.lily.codex-note-funnel`) in the log filename is the minimum bar.\n\n**11. Be aware of `ThrottleInterval` behavior and make the setting's intent explicit**\n\nIf you consistently use `exit 0`, launchd's default `ThrottleInterval` causes no problems. But consciously choose to \"leave it at the default\" and write the reason in the plist or operations docs. This prevents the confusion of \"I didn't change any settings, so why?\" when trouble hits six months later.\n\n**12. Verify the same filesystem with `df -h` before placing `queue/` and `claimed/`**\n\nThe biggest risk to the atomicity premise is a cross-device `mv`. Build the habit of checking whenever you change directories into the deploy script:\n\n```\nqueue_dev=$(df -P \"$QUEUE_DIR\" | awk 'NR==2{print $1}')\nclaimed_dev=$(df -P \"$CLAIMED_DIR\" | awk 'NR==2{print $1}')\n[ \"$queue_dev\" = \"$claimed_dev\" ] || { echo \"ERROR: cross-device mv\"; exit 1; }\n```\n\nThe problem that occurred on September 17, 2026, after Codex recovered from `usage_limit_exceeded`—\"the funnel-pm lane processed the same JSON 4 times, one minute apart\"—was an inevitability of the design. When five lanes fire at once with 57 JSON files backed up, a pickup script without mutual exclusion collides structurally, not probabilistically. Because launchd is a scheduler, not a mutex.\n\nThe fix was one line.\n\n```\nmv \"$TARGET\" \"$CLAIMED_DIR/$BASENAME\" 2>/dev/null || exit 0\n```\n\nPOSIX `rename(2)` is serialized by the kernel on the same filesystem. Fundamentally unlike a \"check, then acquire\" PID check, the check and the acquisition complete within a single system call. Unlike `flock` with its fd-inheritance problem, the lock is decoupled from the life and death of the process. Since the file itself becomes the claim token, a leftover file after a crash remains in a state that's human-readable.\n\nThe \"do nothing if running\" PID-check pattern in `autolike-plist-reconcile.sh` and the mv claim pattern here were born from the same philosophy. The only difference is \"whether there's a TOCTOU window.\" Both are designs that make the choice \"exit if someone else is processing\" in the very first operation.\n\nTo keep an autonomous environment stable over the long term, the only way is to keep stacking up these kinds of \"structures that don't permit contention in the first place.\" Procedural fixes shrink the wound; environmental fixes eliminate the event. The reason I could go from ¥0 to ¥1.2M a month in six months is that on the days things broke, I repeatedly chose \"fix the structure\" over \"next time I'll do it differently.\"\n\nThe full picture of the system, the breakdown of the ¥1.2M/month, and the 30-day setup guide are compiled in a paid note (Japanese).\n\n📕 [How to actually earn with a Claude Code autonomous environment — the system, real examples, getting started, and support](https://note.com/bokuwalily/n/n849b3a07784a)\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue", "canonical_source": "https://dev.to/bokuwalily/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue-processing-idempotent-1jl9", "published_at": "2026-09-21 00:00:06+00:00", "updated_at": "2026-09-21 00:22:52.333141+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Claude Code", "Codex", "launchd", "note-autolike", "ai-portraits-fragments", "social-autolike"], "alternates": {"html": "https://wpnews.pro/news/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue", "markdown": "https://wpnews.pro/news/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue.md", "text": "https://wpnews.pro/news/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue.txt", "jsonld": "https://wpnews.pro/news/one-line-of-mv-kills-the-4x-duplicate-run-after-recovery-making-launchd-queue.jsonld"}}