Six months ago my side business made ยฅ0 a month. This month it's ยฅ1.2M, running on an autonomous setup built with Claude Code. And here's the thing about automation: the moment it breaks is exactly when the sloppiness in your design gets exposed.
When automation breaks, the reflex is to reach for a procedural fix: "next time I'll do this instead." Check the logs more often. Retry by hand. Space things out a bit more. All of these are bandages meant to shrink the wound the next time the problem recurs. They don't remove the cause, so once the same conditions line up, it will happen again.
After six months of running an automated system, one thing I'm certain of: a scheduler combined with a shared directory will always collide unless you explicitly implement mutual exclusion. Not "it might collide if you're unlucky," but "it will collide whenever the conditions for collision line up." This isn't a probability problem; it's a structural one.
On September 17, 2026, Codex was down for an entire day with usage_limit_exceeded. Under ~/.codex/sessions/2026/09/17/ there were 54 sessions piled up, every one of them terminated with a limit error. Throughout that time, three workflowsโnote-autolike, ai-portraits-fragments, and social-autolikeโkept firing jobs, and unprocessed JSON files accumulated in ~/dev/note-autolike/done/. Measured the next morning at 05:20, the JSON files flagged needs_imagegen_thumbnail had swelled from 43 to 57. Fourteen new files had stacked up, all of them left untouched.
The instant Codex came back, five launchd lanes fired at once. Each lane's pickup script grabs the JSON files under done/ with ls or a glob and starts processing. With no mutual exclusion, multiple processes look at the same JSON at the same time. The result: the funnel-pm lane processed the same JSON 4 times, one minute apart. As the numbers showโ8 of 11 files were hit 2 or more timesโthis wasn't a freak accident. It was an inevitable consequence of the design.
Look at StartCalendarInterval in com.lily.codex-note-funnel.plist and you'll see two entries: 10:40 and 16:40.
<key>StartCalendarInterval</key>
<array>
<dict>
<key>Hour</key>
<integer>10</integer>
<key>Minute</key>
<integer>40</integer>
</dict>
<dict>
<key>Hour</key>
<integer>16</integer>
<key>Minute</key>
<integer>40</integer>
</dict>
</array>
launchd simply invokes run-codex-funnel.sh according to this schedule. Whether the previous run finished, or whether another lane is currently working on the same fileโlaunchd has no interest in any of it. And that's correct, by-design behavior. launchd is a scheduler, not a mutex. Mutual exclusion over a file queue is the application layer's responsibilityโin other words, it's a feature the pickup script itself has to implement.
In normal operation, each lane's timing is slightly offset, which naturally keeps collisions rare. Target JSON files also arrive one at a time in sequence, so there are few opportunities for multiple lanes to see the same file simultaneously.
But a long outage like Codex's usage_limit_exceeded changes the situation completely. All lanes start at once with 57 JSON files backed up from the outage period.
้ๅธธ้่ปข:
ใฌใผใณA โ job_001 ๅฆ็ โ ๅฎไบ
ใฌใผใณB โ job_002 ๅฆ็ โ ๅฎไบ ๏ผใฟใคใใณใฐใใบใฌใฆใใ็ซถๅใใชใ๏ผ
Codex ๅพฉๅธฐ็ดๅพ:
ใฌใผใณA โโ
ใฌใผใณB โโคโโโ job_001 ใๅๆๅๅพ โ 4้ฃๅฐ
funnel-pm โโ
๏ผ57ๆฌใๆบใพใฃใฆใใใๅ
จใฌใผใณใๅ
้ ญใใกใคใซใซๆฎบๅฐ๏ผ
A procedural fix like "adjust the intervals" can't handle this "burst release" pattern. No matter how tightly you tune the intervals, the instant after recovery, every lane will fire simultaneously.
"Just create a lock file" is the natural idea. But look closely at the implementation and there's a problem.
LOCKFILE=~/dev/note-autolike/.processing.lock
if [ ! -f "$LOCKFILE" ]; then
touch "$LOCKFILE"
process_file "$json"
rm "$LOCKFILE"
fi
Between the if [ ! -f ] check and the touch, there's a "Time of Check to Time of Use" (TOCTOU) window where another process can slip in. Two processes both observe "no file" and both execute touchโthat's the race. Under high load and high frequency, this window really does open.
On top of that, if the process crashes mid-run, the lock file stays behind. Every subsequent run sees the lock and exits immediately, so processing stalls completely. You need a separate crash-recovery procedure, and operational cost goes up.
File locking via the flock command is another option, but since file descriptors aren't inherited each time launchd spawns a new process, it doesn't work for exclusion between lanes.
The existing ~/.claude/scripts/autolike-plist-reconcile.sh contains a guard built with the same concern in mind. That script does a reload when the AUTOLIKE_TIMEOUT_SEC value in the plist has drifted from the value launchd has loaded, but the core is this code:
pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
echo "[$(date '+%F %T')] $L ๅฎ่กไธญ(pid=$pid) ใฎใใ่ฆ้ใ ($have -> $want)" >>"$LOG"
continue
fi
launchctl bootout "$D/$L" 2>/dev/null
The comment at the top of the script says that booting out a running job wipes out that run's likes entirely, so it must always wait. It fetches the PID via launchctl list and skips the bootout unless the value is - (not running). Before starting work, check "is someone else already processing this?"โand if so, exit immediately. That's the same idea as the mv pattern in this article.
That said, a PID check has limits. You can't reduce to zero the chance that a new process starts between the "check" and the "bootout." In the context where this script runs, that's acceptable, but for claiming exclusive ownership of a file queue, it's insufficient. You need a stronger atomic operation.
Codex usage_limit_exceeded ใใๅพฉๅธฐ
โ
โผ
~/dev/note-autolike/done/ ใซ 57 ๆฌใๆป็
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ job_20260917_001.json โ
โ job_20260917_002.json โ
โ ... โ
โ job_20260917_057.json โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
5 ใฌใผใณใ launchd ใซใใไธๆ่ตทๅ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ com.lily.codex-note-funnel (10:40) โ
โ com.lily.autolike.note1 โ
โ com.lily.autolike.note2 โ
โ com.lily.autolike.funnel-pm โ 4้ฃๅฐ โ
โ com.lily.autolike.social โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
ใไฟฎๆญฃๅใๅใฌใผใณใ done/ ใ glob โ ๅ
้ ญใใกใคใซใๅๅพ
โ ๆไปใชใ โ ๅ
จใฌใผใณใ job_001 ใๅๆใซ read
โ
funnel-pm ใ 1ๅๅทฎใงๅไธ JSON ใ 4 ๅๅฆ็
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
ใไฟฎๆญฃๅพใmv ใซใใๅๅญ็ๆๆๆจฉๅๅพ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ mv done/job_001.json processing/ โ
โ โโ ๆๅ: ่ชใใญใปในใฎใฟใๅฆ็ใ็ถ็ถ โ
โ โโ ๅคฑๆ: ไปใใญใปในใๅๅพๆธใฟ โ exit 0โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
ใ1ใใกใคใซ = 1ใใญใปในใฎใฟๅฆ็ใใไฟ่จผใใใ
On the same filesystem, mv executes a single POSIX rename(2) system call. Because the "rename this file" operation completes indivisibly at the kernel level, even if two processes simultaneously run mv done/job_001.json processing/job_001.json, exactly one succeeds and the other is guaranteed to fail. No window opens between "check" and "acquire."
This is the fundamental difference between a PID check and mv. With a PID check, "check" and "start processing" are separate operations. With mv, "check" and "acquire" are completed inside one system call.
Here's the implementation to insert at the top of run-codex-funnel.sh.
#!/bin/bash
set -uo pipefail
DONE_DIR=~/dev/note-autolike/done
PROCESSING_DIR=~/dev/note-autolike/processing
mkdir -p "$PROCESSING_DIR"
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0 # ๅฏพ่ฑกใชใ โ ๆญฃๅธธ็ตไบ
BASENAME=$(basename "$TARGET")
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
WORKING="$PROCESSING_DIR/$BASENAME"
There's effectively only one situation in which mv fails: another process has already mv'd the file and it no longer exists. That is precisely the race condition itself, so exiting immediately with exit 0 is the correct response. You don't even need to write an error log.
A cross-device mv (spanning different mount points) is not atomic, so be carefulโbut in this setup, both done/ and processing/ live under ~/dev/note-autolike/, which guarantees the same filesystem.
The ProgramArguments in com.lily.codex-note-funnel.plist are structured like this:
<key>ProgramArguments</key>
<array>
<string>~/.claude/scripts/claude-quota-guard.py</string>
<string>--job</string>
<string>com.lily.codex-note-funnel</string>
<string>--priority</string>
<string>--</string>
<string>~/dev/note-autolike/run-codex-funnel.sh</string>
</array>
claude-quota-guard.py runs first as a quota-control wrapper, and if there's headroom, it launches run-codex-funnel.sh. Since the pickup logic lives inside run-codex-funnel.sh, inserting the mv ownership claim into the first few lines of the script is all it takes. No plist changes required.
launchd will keep calling run-codex-funnel.sh at 10:40 and 16:40. Even with 57 JSON files backed up right after Codex recovers from usage_limit_exceeded, every time run-codex-funnel.sh starts it repeats the same behavior: "first claim one file with mv; if that fails, exit immediately." Even if five processes, including the funnel-pm lane, run at the same time, exactly one process handles each file.
The first half showed only the single mv line, but to actually run it in production you need layers before and after. Looking at ProgramArguments in com.lily.codex-note-funnel.plist, claude-quota-guard.py runs as the leading wrapper, and then run-codex-funnel.sh is called. The full script looks like this:
#!/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
export PATH
set -uo pipefail
DONE_DIR=~/dev/note-autolike/done
PROCESSING_DIR=~/dev/note-autolike/processing
PROCESSED_DIR=~/dev/note-autolike/processed
LOG=~/dev/note-autolike/logs/codex-funnel.log
mkdir -p "$PROCESSING_DIR" "$PROCESSED_DIR"
for stale in "$PROCESSING_DIR"/*.json; do
[ -f "$stale" ] || break
age=$(( $(date +%s) - $(stat -f %m "$stale") ))
if [ "$age" -gt 3600 ]; then
mv "$stale" "$DONE_DIR/" 2>/dev/null
echo "[$(date '+%F %T')] recovered stale: $(basename "$stale")" >> "$LOG"
fi
done
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0
BASENAME=$(basename "$TARGET")
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
WORKING="$PROCESSING_DIR/$BASENAME"
_cleanup() {
local code=$?
if [ $code -eq 0 ]; then
mv "$WORKING" "$PROCESSED_DIR/$BASENAME" 2>/dev/null
else
mv "$WORKING" "$DONE_DIR/$BASENAME" 2>/dev/null
echo "[$(date '+%F %T')] ERROR exit=$code, returned: $BASENAME" >> "$LOG"
fi
}
trap '_cleanup' EXIT
echo "[$(date '+%F %T')] processing: $BASENAME" >> "$LOG"
Look at EnvironmentVariables in com.lily.codex-note-funnel.plist: the key exists, but the content is an empty <dict/>.
<key>EnvironmentVariables</key>
<dict>
</dict>
When launchd starts a process, its PATH is only the minimal set /usr/bin:/bin:/usr/sbin:/sbin. If you echo $PATH in a terminal you'll see /usr/local/bin included, but that's added by .zshrc or the nvm initialization script. launchd doesn't run any such shell initialization. Tools installed in /usr/local/bin like jq, node, and python3 fail silently as "command not found" in scripts launched from a plist. The fix is one of two choices: write it in the plist's EnvironmentVariables, or hardcode it at the top of the script. The latter is more convenient because you can verify the script's behavior on its own, so I've standardized on export PATH at the top.
set -uo pipefail and || exit 0 Must Go Together
When set -uo pipefail is active, the whole script aborts the moment a command returns a non-zero exit code. mv returns exit code 1 when the source file doesn't exist. That maps exactly onto the normal race scenario of "another process already claimed it."
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
2>/dev/null is needed as its counterpart. With || exit 0 alone, "No such file or directory" is printed to stderr. Since StandardErrorPath in com.lily.codex-note-funnel.plist points to ~/dev/note-autolike/logs/codex-funnel.error.log, every race piles a line that isn't actually an error into the error log. When you monitor logs, this becomes the cause of false alarms like "the error log is growing."
The PID-check portion of autolike-plist-reconcile.sh has a subtlety that's easy to miss.
pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
The first column returned by launchctl list is either a PID or -. - means the job is registered but not currently running. With [ -n "$pid" ] alone, a pid of - would also be judged as "has a PID." That's why [ "$pid" != "-" ] is joined with AND.
This idiom of checking "the variable is non-empty" and "the value is meaningful" separately can be applied to the mv pattern too. If you want to add a defensive layer that verifies the WORKING file actually exists rather than looking only at the exit code of mv, write it like this:
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
[ -f "$WORKING" ] || { echo "FATAL: mv succeeded but file missing" >> "$LOG"; exit 1; }
This extra check is normally never reached, but it makes the symptom explicit if you accidentally run a cross-device mv. In an environment with set -uo pipefail active, having the cause in the log beats dying silentlyโdebugging afterward is dramatically faster.
The step at the top of the script that "returns processing/ files older than one hour to done/" is a dead-letter countermeasure.
age=$(( $(date +%s) - $(stat -f %m "$stale") ))
if [ "$age" -gt 3600 ]; then
mv "$stale" "$DONE_DIR/" 2>/dev/null
fi
If the process crashes after claiming ownership with mv, the file stays in processing/. The EXIT trap is designed to return it to done/, but the trap won't run in cases like a forced kill with signal 9. The symptom of "I check done/ the next morning and it's empty, yet nothing got processed" is caused by this pattern. By running recovery at startup, the file automatically re-enters the queue on the next launchd invocation. The threshold is 3600 seconds (one hour) so that legitimately long-running jobs don't get returned by mistake. Since StartCalendarInterval in com.lily.codex-note-funnel.plist has the two entries 10:40 and 16:40, the maximum schedule gap is six hours. One hour is a sufficient safety margin within that range.
The day after implementing mv || exit 0, I looked at codex-funnel.error.log and the line count had gone from 1 to 32. The processed count was increasing normally. The 4x duplicates were gone. And yet the error log was growing.
At first I assumed "there must be an exception somewhere" and started debugging the main processing. After about 30 minutes of chasing it, I realized 2>/dev/null was missing. The code I first wrote was this:
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" || exit 0
|| only handles the exit code. It doesn't stop output to stderr. launchd keeps writing stderr to StandardErrorPath. As a result, "No such file or directory" accumulated at every moment of contention.
The correct form is this:
mv "$TARGET" "$PROCESSING_DIR/$BASENAME" 2>/dev/null || exit 0
The cause was that I hadn't built the habit of always writing these two together. My preconception that "error log growing = bug in the main body" sent my debugging in the wrong direction. When the error log grows, the first thing to check is "am I silencing the normal failures?"โI should have made that order routine from the start.
This happened in the version before I implemented the EXIT trap. Codex crashed overnight several times in a row, and when I checked the next morning, done/ was empty but processing wasn't happening.
Tracing the symptom, five JSON files had accumulated in processing/. Each time the script started, it checked done/, found TARGET empty, and did exit 0. But the "files that should be processed" were stuck in processing/.
done/ โ ็ฉบ๏ผๆฌกใฎไปไบใฏใใใซๆฅใ๏ผ
processing/ โ job_001.json, job_002.json ... โ ใพใณใ
processed/ โ ๅฎไบๆธใฟ
This state does double damage. First, processing stops. Second, "the fact that it's stuck isn't visible from outside." Because done/ is empty, the automation looks healthy. But even when new files arrive in done/, nobody ever touches the zombies in processing/.
The fix combined two things: returning files via the EXIT trap and recovering stale processing/ files at startup. Either one alone is insufficient; to handle forced kills where the trap doesn't run, startup recovery is required. After implementing this, I only need to periodically ls -la the processing/ directory to confirm nothing is lingering, and this type of stall resolves itself by the next morning.
This was the failure that burned the most time. Running run-codex-funnel.sh directly in the terminal processes JSON files normally. Via launchd, "processing: ..." never appears in codex-funnel.log. The error log is empty too. The script is starting, but doing nothing.
What I used to pin down the cause was reproducing the shell launchd starts as faithfully as possible.
env -i HOME="$HOME" PATH=/usr/bin:/bin:/usr/sbin:/sbin bash -l ~/dev/note-autolike/run-codex-funnel.sh
Running it with env -i to minimize the environment variables, jq: command not found appeared partway through. The main processing used jq to read the needs_imagegen_thumbnail flag from the JSON, but jq was installed at /usr/local/bin/jq. With set -uo pipefail active, the script exits non-zero the moment jq isn't found. The exit code should have been written to codex-funnel.error.log, but since the error message itself never went to stderr (no jq means no error text either), the log was empty.
Had I read that EnvironmentVariables in com.lily.codex-note-funnel.plist was empty, I would have noticed sooner. The cause was skipping verification on the assumption that "PATH is probably fine" without actually reading the file. Since then, writing PATH= at the top has been an absolute rule for every script run via launchd.
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
export PATH
Whether this one line is present is now the first thing I check in code review.
ls *.json Returned Exit Code 2 on "No Files" and Crashed the Script
When done/ is empty, ls "$DONE_DIR"/*.json fails shell glob expansion and returns exit code 2. In a set -uo pipefail environment, this causes the script to terminate abnormally.
TARGET=$(ls "$DONE_DIR"/*.json | head -1)
TARGET=$(ls "$DONE_DIR"/*.json 2>/dev/null | head -1)
[ -z "$TARGET" ] && exit 0
This problem wouldn't occur if the design treated an empty done/ as "abnormal." But this setup is designed so that "in normal operation, done/ is empty more often than not," so empty is the normal path. To "treat empty as normal," you need a line that explicitly absorbs the command's failure.
Looking at how autolike-plist-reconcile.sh is written, there are three places where it uses continue to skip out of the loop.
want=$($PB -c "Print :EnvironmentVariables:AUTOLIKE_TIMEOUT_SEC" "$P" 2>/dev/null) || continue
have=$(launchctl print "$D/$L" 2>/dev/null | ...) | head -1)
[ -n "$have" ] || continue
[ "$want" = "$have" ] && continue
|| continue appears throughout. The decision "if we can't get this lane's info, skip it" is written as moving on to the next iteration without stopping on the error. This idea of "converting a failure into an instruction for the next step" is the same philosophy as mv ... || exit 0. Errors aren't something to leave in the logโthey're a signal for the next action. Once I switched to this viewpoint, my scripts became dramatically clearer.
funnel-pm Was Firing 4x
The last "stuck" is less about technology and more about observability. I only noticed the 4x duplicate on September 17 when I counted the files in processed/ at 05:20 the next morning, on the 18th. Looking at the log, there were four entries for the same JSON, one minute apart.
For the 17 hours the problem was happening, I knew nothing. The logs accumulate in ~/dev/note-autolike/logs/codex-funnel.log, but nobody monitors them in real time. The fact that "processing was duplicated" could only be detected from duplicate filenames in processed/.
I now use this duplicate-detection one-liner in my morning check:
ls ~/dev/note-autolike/processed/ | sed 's/_[0-9]*$//' | sort | uniq -d
It strips the timestamp portion of the filenames and checks for duplicate base names. If it returns anything, that day's processing had a double execution.
Structural improvement that drives duplicates to zero (the mv pattern) and, independently, monitoring that lets you notice duplicates happenedโthese are problems on different layers. Build a design that doesn't break, and also have a mechanism that lets you notice a break the same day rather than the next. In an environment where automation feeds real revenue, this two-tier approach is indispensable.
launchd does not inherit PATH. Running with an empty EnvironmentVariables block will get you stuck. EnvironmentVariables in com.lily.codex-note-funnel.plist is empty, and the effective PATH at launchd startup is only /usr/bin:/bin:/usr/sbin:/sbin. The codex installed via nvm is not on this PATH. Even though run-codex-funnel.sh has set -uo pipefail, in a "command not found" situation there are cases where -u reacts late (which codex returns an empty string). The fastest diagnosis is to start it manually with launchctl start com.lily.codex-note-funnel and check the logs.
Nice: 10 and LowPriorityIO: true conflict with burst processing. Both plist settings deprioritize the script's execution when the system is under heavy load. When you want to churn through 57 JSON files right after Codex recovers, Nice: 10 is a "yield CPU to other user operations" setting, so processing is delayed. "Run quietly in the background" and "process at full speed after recovery" are in conflict. Decide which to prioritize and write the intent in a comment. The current plist chooses the former.
The default ThrottleInterval depends on the launchd version. Accumulated non-zero exits can cause StartCalendarInterval scheduled starts to be skipped. This ties directly into failure #3 in part 2 (throttling on exit 1), but to add: if 10:40 is skipped due to throttling, the next scheduled run is 16:40, six hours later. The reliable diagnosis is to go back through days where nothing was written to codex-funnel.error.log (StandardErrorPath). If you consistently use exit 0, the default ThrottleInterval setting is fine, but if you're "leaving it to the default," record that intent in a plist comment or documentation.
An mv failure due to a missing claimed/ directory and an mv failure due to race avoidance both return the same exit 0. I touched on this in failure #5 of part 2, but let me restate it as the underlying design trade-off. Giving "normal race avoidance" and "abnormal environment misconfiguration" the same exit code prevents launchd's throttle, but delays discovery of problems. Since you're paying the price of this silence, external monitoring is mandatory.
Cross-device mv is not atomic. POSIX rename(2) is serialized by the kernel only "on the same filesystem." If queue/ and claimed/ span different mount points (e.g., NFS, Docker volumes, external drives), the operation decomposes internally into copy + delete, and a TOCTOU window opens. This setup has both directories under ~/dev/note-autolike/, so there's no issue, but whenever you change the directory layout, verify the same filesystem with df -h.
Filename sort order becomes unstable when files are created in the same second. The lexicographic sort in ls -1 "$QUEUE_DIR"/*.json | sort | head -1 assumes the timestamp portions of the filenames differ. When batch submission creates multiple files within one second, same-name prefixes collide, and five lanes grab different files as the "first." As a result, no claim contention occurs, all lanes process simultaneously, and five Codex sessions start at once. The workaround is to include epoch seconds + PID suffix in the filename: job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json.
Leftover files in claimed/ become a silent dead-letter queue. If processing crashes midway, the file stays in claimed/. The next launchd start looks at queue/, so files in claimed/ are never picked up again. In fact, after Codex recovered on September 17, 2026, there was a day when several files were stuck in claimed/. Since adding monitoring that alerts when a file in claimed/ has gone more than an hour without an update, the lag before noticing this state has dropped to zero.
The claude-quota-guard.py fd-inheritance issue remains even after the mv claim. The first entry in ProgramArguments in com.lily.codex-note-funnel.plist is ~/.claude/scripts/claude-quota-guard.py. While this Python wrapper launches run-codex-funnel.sh via subprocess.Popen, the job appears "running" to launchd. The PID check in autolike-plist-reconcile.sh (verifying via launchctl list | awk -v l="$L" '$3==l{print $1}' that the PID is not -), which skips reload on the judgment "codex-funnel is running," correctly captures this "wrapper is alive" state as well. The mv claim eliminated the race, but you need to stay aware of how PIDs are read given that the wrapper launch is a precondition.
The glob in reconcile.sh excludes codex-note-funnel. autolike-plist-reconcile.sh targets com.lily.autolike.*.plist. com.lily.codex-note-funnel.plist is a separate file and doesn't match this glob. Since the automatic timeout-value reload that reconcile performs doesn't apply to codex-funnel, changing codex-funnel's timeout setting requires a manual launchctl bootout + bootstrap. If this is intentional, state it explicitly in a comment so that six months from now you won't be confused.
No log rotation configured for codex-funnel.log. The plist's StandardOutPath points to ~/dev/note-autolike/logs/codex-funnel.log. launchd doesn't rotate logs. With dozens of lines per run accumulating continuously, it reaches tens of MB in a year. You need explicit rotation via newsyslog or logrotate, or line-count cap management inside the script.
Without a mechanism to automatically detect "log not updating," you won't notice a stall for a week. I've separately added monitoring that fires an alert if the update timestamp of claimed/ hasn't changed within 15 minutes after the 10:40 start. The exit 0 design maximizes compatibility with launchd, but in exchange carries the silent risk that "normal exit and silent failure are indistinguishable." This monitoring script isn't optionalโit should be installed as a pair with the mv claim pattern.
Here are the rules that solidified after six months of running an autonomous environment at the ยฅ1.2M/month scale, each grounded in real code.
1. Put the mv ownership claim at the very top of the script
Run mv before argument validation, log initialization, or directory checks. If you can't get the file, there's no need to run any subsequent processing at all. This minimizes CPU and memory usage per lane while resolving contention as fast as possible.
TARGET=$(ls -1 "$QUEUE_DIR"/*.json 2>/dev/null | sort | head -1)
[ -z "$TARGET" ] && exit 0
BASENAME=$(basename "$TARGET")
mv "${QUEUE_DIR}/${BASENAME}" "${CLAIMED_DIR}/${BASENAME}" 2>/dev/null || exit 0
2. Log successful claims; keep failures (race avoidance) silent
With "success = logged, race avoidance = silent," grep "claimed:" codex-funnel.log | wc -l becomes today's processed count. A count log that needs no filtering dramatically reduces the implementation cost of periodic monitoring scripts. Pairing 2>/dev/null with exit 0 is a consistent pattern.
3. Return exit 0 only for "normal race avoidance" and "empty queue"
For "directory doesn't exist," "permission error," and "unexpected mv error," return exit 1. The more silent exits you have, the later you discover failures. Set up the environment with an initial deploy script, then write the script body on the premise that "only normal exits are exit 0," and maintenance cost goes down.
4. Make state visible with the three directories queue/, claimed/, done/
When the directory itself represents state, ls ~/dev/note-autolike/claimed/ | wc -l tells you "the number of stuck files" at a glance. No flag management or DB neededโls alone is your monitoring tool.
5. Put epoch seconds + PID in filenames to stabilize sort order
FILENAME="job_$(date +%s)_$$_$(uuidgen | cut -d- -f1).json"
$$ is the PID of the generating process. Even when multiple files are created in the same second, differing PIDs make the lexicographic order deterministic. This stabilizes the premise of the mv claim pattern: all five lanes select the same file as first.
6. Standardize the "record the reason for skipping" pattern from reconcile.sh across all scripts
Lines 18โ22 of autolike-plist-reconcile.sh log the reason and continue when a running job is found:
pid=$(launchctl list | awk -v l="$L" '$3==l{print $1}')
if [ -n "$pid" ] && [ "$pid" != "-" ]; then
echo "[$(date '+%F %T')] $L ๅฎ่กไธญ(pid=$pid) ใฎใใ่ฆ้ใ ($have -> $want)" >>"$LOG"
continue
fi
The pattern of logging "why nothing was done" makes later failure tracing dramatically easier. Keep this habit even after the mv claim eliminates the race.
7. Separate post-crash recovery from the main script
If you build the "return leftover claimed/ files to queue/" step into the main script, a new race reappears where two processes simultaneously "return the leftover file โ try to re-claim it." Stick to a design where recovery is delegated to an independent monitoring script that runs at low frequency.
8. Record the start time in a sidecar file right after claiming
mv "${QUEUE_DIR}/${BASENAME}" "${CLAIMED_DIR}/${BASENAME}" 2>/dev/null || exit 0
echo "{\"claimed_at\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" \
> "${CLAIMED_DIR}/${BASENAME}.claim"
Writing the start time to a .claim sidecar lets you write a recovery script that "returns files whose .claim is older than one hour to queue/." Aging detection for leftover files becomes precise.
9. Make the initial deploy script idempotent and re-runnable
mkdir -p ~/dev/note-autolike/{queue,claimed,done,logs}
Prepare a script that runs this before calling launchd's bootstrap. This structurally prevents a recurrence of failure #5 from part 2, where mv fails silently because a directory doesn't exist.
10. Separate log files between lanes to prevent interleaved lines
The plist settings StandardOutPath: codex-funnel.log and StandardErrorPath: codex-funnel.error.log point to separate files per lane, and that's correct. In the post-recovery situation where five lanes start at once, if a shared log gets interleaved, reconstructing what happened becomes impossible. Including the label name (com.lily.codex-note-funnel) in the log filename is the minimum bar.
11. Be aware of ThrottleInterval behavior and make the setting's intent explicit
If you consistently use exit 0, launchd's default ThrottleInterval causes no problems. But consciously choose to "leave it at the default" and write the reason in the plist or operations docs. This prevents the confusion of "I didn't change any settings, so why?" when trouble hits six months later.
12. Verify the same filesystem with df -h before placing queue/ and claimed/
The biggest risk to the atomicity premise is a cross-device mv. Build the habit of checking whenever you change directories into the deploy script:
queue_dev=$(df -P "$QUEUE_DIR" | awk 'NR==2{print $1}')
claimed_dev=$(df -P "$CLAIMED_DIR" | awk 'NR==2{print $1}')
[ "$queue_dev" = "$claimed_dev" ] || { echo "ERROR: cross-device mv"; exit 1; }
The problem that occurred on September 17, 2026, after Codex recovered from usage_limit_exceededโ"the funnel-pm lane processed the same JSON 4 times, one minute apart"โwas an inevitability of the design. When five lanes fire at once with 57 JSON files backed up, a pickup script without mutual exclusion collides structurally, not probabilistically. Because launchd is a scheduler, not a mutex.
The fix was one line.
mv "$TARGET" "$CLAIMED_DIR/$BASENAME" 2>/dev/null || exit 0
POSIX rename(2) is serialized by the kernel on the same filesystem. Fundamentally unlike a "check, then acquire" PID check, the check and the acquisition complete within a single system call. Unlike flock with its fd-inheritance problem, the lock is decoupled from the life and death of the process. Since the file itself becomes the claim token, a leftover file after a crash remains in a state that's human-readable.
The "do nothing if running" PID-check pattern in autolike-plist-reconcile.sh and the mv claim pattern here were born from the same philosophy. The only difference is "whether there's a TOCTOU window." Both are designs that make the choice "exit if someone else is processing" in the very first operation.
To keep an autonomous environment stable over the long term, the only way is to keep stacking up these kinds of "structures that don't permit contention in the first place." Procedural fixes shrink the wound; environmental fixes eliminate the event. The reason I could go from ยฅ0 to ยฅ1.2M a month in six months is that on the days things broke, I repeatedly chose "fix the structure" over "next time I'll do it differently."
The full picture of the system, the breakdown of the ยฅ1.2M/month, and the 30-day setup guide are compiled in a paid note (Japanese).
*Written by Lily โ I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio ยท X ยท GitHub*