Everyone has met the bug that only shows up in production. This one was its exact mirror image: it passed every single scheduled run and died only when I typed the command myself. That inversion is why it stayed invisible for five days โ five days in which my posting pipeline was down and my phone stayed completely quiet.
Some context on why that pipeline matters to me: I went from ยฅ100k/month as a university student to ยฅ600k/month juggling gigs, got laid off and dropped back to zero, then spent six months building an autonomous Claude Code setup. Revenue is now ยฅ1.2M/month, and the foundation under all of it is a system that grows articles while I sleep. This is the story of that system failing quietly โ failing without telling anyone.
Most automation write-ups end at "build the system and life gets easier." The biggest thing I learned in the last six months is the opposite. A system starts rotting the moment you build it, and it will not tell you that it is rotting.
article-daily-stock.sh
(~/.claude/scripts/article-daily-stock.sh
) runs from launchd in two slots โ 8:00 and 10:35 every morning โ generates one article, and stocks it in ~/content/article/
. The core of the design is written in a comment.
Pushing to Zenn and generating stock are decoupled. Even if the deploy jams, the generation buffer keeps stacking up. That "separate generation from publishing" split is what supports a stable output of 30 articles a month โ or so it was supposed to.
The problem wasn't that this script exited successfully. The problem was that the failure notification died along with the failure.
What's the nastiest class of bug in software development? "It breaks only in production and never reproduces locally" โ the demon everyone meets at least once. What I ran into this time was the perfect flip side of that.
launchd does not set LANG. In other words, scripts under launchd run in the C locale. When you open a terminal and run the script by hand, the shell inherits
LANG=ja_JP.UTF-8
. That difference produces a failure pattern that is exactly backwards from normal.ja_JP.UTF-8
) โ instant death with exit 127The scheduled run passes every day, so nobody suspects it. It only dies when you run it manually, so you shrug it off as "I must have invoked it wrong." And this time โ the place where it died was the failure-notification line itself.
When a post to Threads failed, the script that was supposed to fire a Discord notification (daily_post.sh
in scent-media
) crashed on the notification line, and the very fact that it had failed got swallowed. For five days posting was stopped, and nothing reached my phone.
ๆ็จฟใๅคฑๆใใ
โ ๅคฑๆ้็ฅในใฏใชใใใ่ตทๅ
โ ้็ฅ่กใงใฏใฉใใทใฅ๏ผexit 127๏ผ
โ ใขใฉใผใใๅบใชใ
โ 5ๆฅ้่ชฐใๆฐใฅใใชใ
By its nature, a bug in notification code is "only ever hit when something fails" โ so while things are succeeding, it is undetectable, always. Any script that writes logs and notifications in Japanese โ which is to say nearly every piece of automation I write โ was structurally capable of stepping on this mine.
To understand where the bug lives, let's first walk the whole shape of article-daily-stock.sh
. The script is 528 lines, but the skeleton splits into 12 phases.
launchd (com.shun.article-daily)
8:00 JST โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
10:35 JST (catch-up) โโโโโโโโโโโโโโโโโโโโโโโโโค
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 0: ๆฌๆฅ็ๆๆธใฟใใงใใฏ โ
โ $DONE_MARKER ใๅญๅจ โ SKIP_GEN=1 โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 1: ็ฐๅข่ฃๅฎ โ
โ PATH += nvm / homebrew โ
โ caffeinate -i -s โ
โ ไบ้ๅฎ่กใญใใฏ (lockdir) โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ audit_repair() โ
โ ๅ
จในใใใฏใฎ็ฃๆป + ่ชๅทฑไฟฎๅพฉ โ
โ ใตใ ใๆฌ ๆ โ ่ชๅๅ็ๆ โ
โ ๆฌๆ็ ดๆ โ QUEUEใธๅๆๅ
ฅ โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
SKIP_GEN=1? โโโโโ exit 0 (audit only)
โ No
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 2: ไบ็ฎใใงใใฏ โ
โ token-budget-advisor.sh โ
โ ๐ด critical โ exit 0 โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 3: ใญใฅใผ็ฉบใชใ่ชๅ็ซๆก โ
โ claude -p ใงใใฟ1ๆฌใ็ๆ โ
โ slug้่คใใงใใฏ โ QUEUE่ฟฝๅ โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 4-5: ่จไบๅท็ญ โ
โ QUEUEๅ
้ ญใฎtopicใๅๅพ โ
โ claude -p --max-turns 40 โ
โ ๅฎใใกใคใซใRead/Grepใใฆๅผ็จ โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 6: ๅคๆฎตๆค่จผ โ
โ frontmatter.title โค 70ๅญ โ
โ ๆฌๆ โฅ 1200 bytes โ
โ ็ฆๆญข่ช (ใฟใคใ ใขใฆใ็ญ) ในใญใฃใณโ
โ ็งๅฏในใญใฃใณ + ๅฎใในๆญฃ่ฆๅ โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 7-8: ในใใใฏๅ โ
โ ~/content/article/articles/ โ
โ gen_note_thumbs.py โ
โ ~/content/article/thumbnailsโ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Phase 9-12: ๅพๅฆ็ โ
โ coverage.json upsert โ
โ QUEUE pop โ done queue โ
โ DONE_MARKER touch โ
โ git push (best-effort) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Because this pipeline drives itself twice every morning, the article buffer keeps growing without me touching a keyboard.
Look at what happens in the very first phase.
NODE_BIN=$(ls -d "$HOME"/.nvm/versions/node/*/bin 2>/dev/null | sort -V | tail -1)
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"
[ -n "$NODE_BIN" ] && export PATH="${NODE_BIN}:$PATH"
The launchd plist (com.shun.article-daily
) does not read shell profiles. It starts in an environment where .zshrc
and .bash_profile
may as well not exist. So the nvm path and the Homebrew path have to be filled in manually. The same root cause is why daily_generate.sh
in scent-media
โ discovered in the same window โ had been wiped out by claude: No such file or directory
: .local/bin
was not on PATH
.
Scripts under launchd cannot assume environment variables. The same applies to the locale. LANG=ja_JP.UTF-8
is set by the terminal. launchd does not set it. So you land in the C locale. That fact โ "the terminal and launchd run in different environments" โ is the direct reason this bug went undiscovered for five days.
caffeinate
if [ -z "${CAFFEINATED:-}" ]; then
exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@"
fi
LOCKDIR="$HOME/.claude/locks/article-daily.lock"
if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
log "ๅฅใคใณในใฟใณในๅฎ่กไธญ(pid=$oldpid) โ skip"; exit 0
fi
rm -rf "$LOCKDIR"; /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi
echo $$ > "$LOCKDIR/pid"
trap 'rm -rf "$LOCKDIR"' EXIT INT TERM
caffeinate -i -s
keeps the MacBook from sleeping, and the CAFFEINATED
environment variable tells the script whether it has already re-launched itself via exec
. If the 10:35 slot fires while the 8:00 slot is still generating an article, and the pid
in lockdir
is alive (kill -0
), it backs out immediately with exit 0
. If a stale lock is left behind, it does rm -rf
and re-acquires โ riding this decision on the atomicity of mkdir
is what prevents races.
audit_repair()
โ the self-repair that runs every slot
Even on days where article generation is skipped (SKIP_GEN=1
), audit_repair()
always runs.
audit_repair
if [ "$MODE" = "audit" ] || [ "$SKIP_GEN" = "1" ]; then
log "===== article-daily done(audit$([ "$SKIP_GEN" = "1" ] && echo '+gen-skipped')) ====="
exit 0
fi
Inside audit_repair()
it sweeps every file in ~/content/article/articles/
and runs body-quality checks (at least 1200 bytes, frontmatter title present, no forbidden words) plus a thumbnail width check (at least 2000px via sips -g pixelWidth
). If a thumbnail is missing, it regenerates it on the spot with gen_note_thumbs.py
; if the body is broken, it digs the meta out of done-queue
and pushes it back to the head of topic-queue
โ up to two times.
article_ok() {
local f="$1"
[ -s "$f" ] || return 1
[ "$(stat -f%z "$f" 2>/dev/null || echo 0)" -ge "$MIN_ARTICLE_BYTES" ] || return 1
grep -qE '^title:' "$f" || return 1
frontmatter_title_ok "$f" || return 1
grep -qiE 'request timed out|ไธๆใชๅๅ|TODO: *ๆฌๆ|\(็ๆๅคฑๆ\)' "$f" && return 1
return 0
}
This article_ok()
function is the gatekeeper for "stub detection." It explicitly rejects the string "request timed out" that the Claude API emits on timeout, and placeholder-ish text like "TODO: ๆฌๆ". Because it runs every slot, a state where yesterday's generation was actually broken is guaranteed to be detected the next morning.
When the topic queue (~/zenn-articles/.topic-queue.json
) empties out, claude itself plans the next topic and adds it to the queue.
REPLENISH_PROMPT=$(cat <<EOF
ใใชใใฏ Lily ใฎใClaude Code็ฐๅขใๆ่กใทใชใผใบใฎ็ทจ้่
ใๆฌกใซๆธใ่จไบใใฟใ1ๆฌใ ใ็ซๆกใใใ
ใใฟใฏใ่ใๅฎ้ใซใใฃใ่ชๅๅใป็ฐๅขๆง็ฏใปClaude Code้็จใฎๅทฅๅคซใใใ้ธใถใๆ้ ็ฆๆญข๏ผๅฟ
ใๅฎๅจใใใใกใคใซใในใฏใชใใใๆ นๆ ใซใใใ
...
EOF
)
TOPIC_JSON=$(run_to 600 "$CLAUDE" -p "$REPLENISH_PROMPT" \
--model "${ARTICLE_MODEL:-sonnet}" --effort high \
--allowedTools "Read,Grep,Glob,Bash" --max-turns 20 ...)
The model is allowed only Read,Grep,Glob,Bash
, forcing it to pick topics grounded in files that actually exist. The output is received under a strict JSON schema (with presence checks for slug
, title
, sources
, thumb_title
), and even slug-duplication is judged mechanically. On a duplicate slug it immediately does exit 0
and retries in the next slot โ a design that seals off fabrication and duplication at the API level.
Once a topic is pulled off the queue, a separate claude -p
session is launched with --max-turns 40
to actually write the article.
run_to 1500 "$CLAUDE" -p "$PROMPT" \
--model "${ARTICLE_MODEL:-sonnet}" --effort high \
--output-format text --allowedTools "Read,Grep,Glob,Write,Bash" --max-turns 40 >> "$LOG" 2>&1
The timeout is 1500 seconds (25 minutes). On failure it just leaves a line in log
and exit 0
s โ article_ok()
will detect the failure and force a retry in the next slot, so there is no reason to halt here.
The post-generation pipeline is meticulous.
frontmatter_title_chars()
(an inline Python3 heredoc) โ discard if over 70 charactersarticle_ok()
โ discard if NGpublished: false
with sed
lily-footer.py
~/
with sed -E 's#/Users/[A-Za-z0-9._-]+/#~/#g'
api_key =
patterns, etc.) โ discard immediately on a hitFinally it stocks the file at ~/content/article/articles/$NO-$SLUG.md
, generates the thumbnail with gen_note_thumbs.py
, upserts into coverage.json
, then pops the QUEUE and moves the entry to the done-queue. The git push is best-effort โ if it fails, "generation succeeded" still stands. Because DONE_MARKER
is touch
ed before git push
, a push failure followed by the next slot restarting will not produce duplicate generation.
This pipeline was built from the design stage on the assumption that things fail. Timeouts, insufficient generation quality, broken thumbnails, network drops โ each has a self-repair path, and the audit runs every slot. Even when I'm away from the keyboard, if something is broken, ~/content/article/_NEEDS-FIX.txt
will be standing there the next morning and a macOS notification will fly in.
It was supposed to fly in.
"$VAR๏ผๅ
จ่ง๏ผ"
dies depending on locale
First, confirm the behavior locally. The reproduction code recorded in ~/Documents/claude-obsidian/wiki/learning/locale-dependent-shell-bugs.md
can be used as-is.
LC_ALL=ja_JP.UTF-8 bash -c 'set -u; ID=test; echo "โ
ๅฎไบ: $ID๏ผ่ฆ็ขบ่ช๏ผ"'
#=> bash: ID๏ผ่ฆ็ขบ่ช๏ผ: unbound variable (exit 127)
LC_ALL=ja_JP.UTF-8 bash -c 'set -u; ID=test; echo "โ
ๅฎไบ: ${ID}๏ผ่ฆ็ขบ่ช๏ผ"'
#=> โ
ๅฎไบ: test๏ผ่ฆ็ขบ่ช๏ผ (exit 0)
When you write $ID๏ผ่ฆ็ขบ่ช๏ผ
, bash under the ja_JP.UTF-8
locale tries to read full-width characters as part of the variable name when determining where the name ends. ๏ผ
is not ASCII (
but U+FF08 (FULLWIDTH LEFT PARENTHESIS). With set -u
enabled, it decides "there is no variable named ID๏ผ่ฆ็ขบ่ช๏ผ
" and dies immediately with exit 127.
Under the C locale (LC_ALL=C
), full-width characters are not interpreted as part of a variable name, so expansion stops at $ID
and everything works. launchd doesn't set LANG, so you get the C locale โ hence the inversion where the scheduled run passes and the manual run dies.
๏ผ
isn't the only dangerous character. Full-width UTF-8 characters in general โ ใ
, ใ
, ใ
, ๏ผ
, ใป
, ๏ผ
, โ
and friends โ can all become the same trap. Any automation script that writes logs or notifications in Japanese can step on this mine. As long as ASCII characters follow, there's no problem โ which means a script written entirely in English never encounters this bug. The more carefully you write in Japanese, the higher your odds of hitting it. An ironic property.
article-daily-stock.sh
's line of defense โ the fixed form
If you look inside article-daily-stock.sh
(~/.claude/scripts/article-daily-stock.sh
), the current code already has braces.
notify "title้ทใใ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ"
notify "่จไบ็ๆใไธๅฎๅ
จ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ"
Before the fix these read $SLUG๏ผๅ่ฉฆ่ก๏ผ
. Since ๏ผ
is full-width, running it by hand from the terminal produces an undefined-variable error for SLUG๏ผๅ่ฉฆ่ก๏ผ
, and with set -uo pipefail
it dies instantly. Under the scheduled run (C locale) it passes normally, so no matter how many days go by, it is never found.
What was easy to overlook this time is the danger of the notification line. You pay attention to core logic like article_ok()
and audit_repair()
. But the argument to notify
that's called on failure โ the string ${SLUG}๏ผๅ่ฉฆ่ก๏ผ
โ sits on a path that is never reached on success. Testing only the success path means it is never hit, ever.
One more thing: the claude
binary detection code in Phase 1 of article-daily-stock.sh
is a line of defense in the same vein.
CLAUDE="${CLAUDE_BIN:-$(command -v claude 2>/dev/null)}"
[ -z "$CLAUDE" ] && [ -x "$HOME/.local/bin/claude" ] && CLAUDE="$HOME/.local/bin/claude"
[ -z "$CLAUDE" ] && CLAUDE=$(ls -t "$HOME"/.nvm/versions/node/*/bin/claude 2>/dev/null | head -1)
[ -x "$CLAUDE" ] || { log "ABORT: claude binary not found"; notify "claude binaryใ็กใ"; exit 0; }
It's a three-stage fallback: command -v claude
โ ~/.local/bin/claude
โ under nvm's bin โ ABORT if still not found. Because the launchd plist doesn't read shell profiles, ~/.local/bin
isn't on PATH. Unless you write with that knowledge in hand, what you end up with is a script that silently gets wiped out every morning by claude: command not found
.
rg
The bug pattern is unambiguous. Just collect everything where "a non-ASCII character immediately follows $VAR, and it isn't already in ${VAR} form."
rg -n --no-heading -g '*.sh' -g '!node_modules' \
'\$[A-Za-z_][A-Za-z0-9_]*[^\x00-\x7F]' ~/dev ~/.claude/scripts ~/bin | rg -v '\$\{'
There's a reason it's a two-stage pipe. The first regex \$[A-Za-z_][A-Za-z0-9_]*[^\x00-\x7F]
matches everything that "starts with $
, continues with alphanumerics and underscores, and is immediately followed by a non-ASCII character." That hits both $VAR
and ${VAR}
. The second stage, rg -v '\$\{'
, excludes the already-braced ${VAR}
form. Already-fixed occurrences aren't picked up as noise, and only what needs fixing remains.
When I ran this, out came 5 repos, 8 files, 12 sites. When you mass-produce scripts solo, habits spread horizontally by copy-paste. Past judgments like "this repo doesn't have any anymore" go stale. Don't hardcode the target list โ always run detection against the current code.
notify "่จไบ็ๆใไธๅฎๅ
จ: $SLUG๏ผๅ่ฉฆ่ก๏ผ"
echo "ๅฆ็ๆธใฟ: $ID๏ผ${DATE}๏ผ"
notify "่จไบ็ๆใไธๅฎๅ
จ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ"
echo "ๅฆ็ๆธใฟ: ${ID}๏ผ${DATE}๏ผ"
The only change is $VAR
โ ${VAR}
. Not a single character of the Japanese log wording changes. With braces, bash interprets only what's inside {}
as the variable name and is no longer confused by the full-width character that follows.
There is one caveat, though. article-daily-stock.sh
contains several inline Python scripts, all written as single-quoted heredocs.
frontmatter_title_chars() {
python3 - "$1" <<'PY'
import sys
PY
}
Inside <<'PY'
(with single quotes) the shell does not expand anything, so the $
in there is interpreted by Python. The scope of the fix is strictly limited to contexts the shell expands โ check the heredoc's quoting before touching anything.
After fixing, I always demonstrated it before closing out. Confirm that the old form exits 127 under ja_JP.UTF-8
, and that the new form prints the same wording and exits 0. Don't stop at "I think I fixed it." The result of actually running it is the evidence.
The first sign of trouble was noticed by accident. "Threads engagement feels thin lately" โ I opened the dashboard on that hunch and found posting had been stopped for five days.
I opened the log for daily_post.sh
(the Threads auto-posting script) in the scent-media
project. The post-failure error lines were there. But the "sending Discord notification" log line that should have followed was not. There was no trace whatsoever of the notification function being called.
At first I suspected a changed Discord webhook URL or a rate limit. But the webhook worked fine when hit by hand. Then, the moment I ran the script itself directly from the terminal, the error appeared.
bash: ๆ็จฟID๏ผ2026-08-05๏ผ: unbound variable
The ๏ผ
in $ๅคๆฐๅ๏ผ
was full-width. When a Threads post failed, the script tried to send a notification to Discord โ but the very line assembling that notification died with exit 127, so the information that it had failed reached nowhere.
ๆ็จฟใๅคฑๆใใ
โ ๅคฑๆ้็ฅใฎๅฆ็ใซๅ
ฅใ
โ ้็ฅๆๅญๅใฎ็ตใฟ็ซใฆใง $VAR๏ผๅ
จ่ง๏ผใๅบ็พ โ exit 127
โ ้็ฅใ้ไฟกใใใชใ
โ ใญใฐใธใฎๆธใ่พผใฟใ้็ฅใใๅพใใซใใฃใใใๆฎใใชใ
โ 5ๆฅ้่ชฐใๆฐใฅใใชใ
The very fact that "it only dies when run by hand" is what produced the five-day delay. The scheduled run (C locale) passes every day, so you don't suspect it. If a manual run fails, you write it off as "a problem with how I invoked it." This time I happened to open the dashboard and notice; if I hadn't, it could have continued for weeks.
That's where the horror of notification code lies. It sits on a path that's only reached when the core logic fails. No matter how many times you test the success path, the quality of the notification code is guaranteed by nothing at all. No amount of confirming the success path guarantees anything about the quality of the failure path โ a lesson I also recorded in locale-dependent-shell-bugs.md
as [[silent-success-antipattern]]
.
daily_generate.sh
was wiped out for a different reason
When I checked another script in the same scent-media
project โ daily_generate.sh
, the one that generates content using the Claude API โ it wasn't running either. But the cause wasn't the locale.
~/.claude/scripts/daily_generate.sh: line 12: claude: command not found
The claude
binary was not found. Because the launchd plist doesn't read shell profiles, ~/.local/bin
isn't on PATH. It's the very problem that article-daily-stock.sh
explicitly solves in Phase 1.
The reason two different bugs surfaced in the same project at the same time is the same reason. The person who wrote the scripts had forgotten that scripts under launchd "run in a different environment than the terminal."
Locale differences and PATH differences โ both come down to the single fact that "launchd does not inherit the terminal's shell environment." Even when you hold that fact as knowledge, you forget it at the moment you write the script. Because when you test locally, running it from the terminal works. It worked โ and you don't dig further. Unless you deliberately reproduce the environment as launchd sees it, your test is nothing more than a check of the success path.
When applying the cross-cutting fix to ~/.claude/scripts/
, I had assumed this directory was "a junk drawer of scripts not managed by git." So after the fix I decided "no commit needed" and moved on.
Then, just in case, I ran git status
after the fix โ and changes came out.
On branch main
Changes not staged for commit:
modified: article-daily-stock.sh
modified: token-budget-advisor.sh
~/.claude/scripts/
was a git repository. And two files were managed as tracked files. Proceeding without verifying the assumption "this shouldn't be under git" created rework after the fact.
The lesson is simple. Running git status before you start is faster. Verifying with one command beats deciding in your head โ it's quicker and more reliable.
After the fix, I staged only those two files with git add article-daily-stock.sh token-budget-advisor.sh
so unrelated diffs wouldn't get swept in, and committed (2b65662
). The commit hashes across all 5 repos are recorded in my post-verification notes โ lily-line-funnel
is 6596c00
, autopilot
is 56288fa
, brand-404
is f0d37f5
, and metrics-hub
is f9b1a18
.
Even while saying I'd swept all 12 sites, exactly one site inside ~/.claude/scripts/article-daily-stock.sh
was left in a state that couldn't be committed.
That site was inside a block of the script that hadn't been committed yet. The fix target was contained within a few dozen lines of uncommitted diff from a half-written new feature.
log "ๅฆ็ในใญใใ: $SLUG๏ผ้่ค๏ผ" # โ ใใใไฟฎๆญฃๅฏพ่ฑก
There were two options โ (a) commit the whole uncommitted block, or (b) fix just that one line locally and not commit it. Option (a) would "sweep in diffs unrelated to this work," making the commit's intent ambiguous. Option (b) means "leaving it in a safe form in the worktree, uncommitted."
I chose (b), explicitly noting that the only unmet completion criterion is "the repository as a whole is clean." In the local worktree all 12 sites are in ${VAR}
form, and I closed it out in a state that won't die under either the scheduled launchd run (C locale) or a manual run (ja_JP.UTF-8
). The remaining one site will be closed together with the commit that tidies up the uncommitted block โ I noted that explicitly and moved on. Not hiding "part of the completion criteria is unmet" is what helps you later.
Until I chased this bug down, I already knew that "launchd and the terminal have different locales." But it never connected to "therefore the notification line must be protected too." Knowledge and implementation are different things. When you write a script, you're careful with the success path. The failure notification line, you wave off assuming it works โ and that assumption caused five days of silent death.
When the notification line dies, "it failed" never arrives. And "never arrived" is indistinguishable from "it succeeded." In automation, notification code has to be more robust than the core code. Because when the notification code goes down, it doesn't even tell you the core went down.
Just as article-daily-stock.sh
has "separate generation from publishing" in its design, the notification path needs a design of its own: "don't let a failure of the notification itself slip by." After this incident, I picked up the habit of always checking for ${VAR}
form before a notification call. One pair of braces is the difference between five days of silence and same-day detection.
Here are the mines I actually stepped on, plus the traps that are easy to miss. This is the "why you get stuck there" behind the symptoms covered above.
โ I wasn't conscious that single-quoted heredocs are out of scope
article-daily-stock.sh
has several places that call Python inline, like frontmatter_title_chars()
.
frontmatter_title_chars() {
python3 - "$1" <<'PY'
import sys
PY
}
The inside of <<'PY'
(with single quotes) is not expanded by the shell. Running the detection command will also hit Python code containing $
, but no fix is needed there. Not knowing this at first, I hesitated over "should I add braces here too?" The single criterion is "is it inside a single-quoted heredoc or not?" Double-quoted or unquoted heredocs (<<PY
) are expanded, so those are in scope.
โก >/dev/null 2>&1 hides notification failures
The script's notify()
is defined like this.
notify() { /usr/bin/osascript -e "display notification \"$1\" with title \"Article daily\"" >/dev/null 2>&1; }
Because both stdout and stderr are thrown away, nothing remains even if osascript
fails. If the notification argument has a full-width expansion problem, the line itself can die with exit 127 and 2>&1
swallows it. That is exactly the direct cause of this five-day silence. Something I realized after the fix: notification failures in particular should have been left in $LOG
.
โข Get the DONE_MARKER timing wrong and you get duplicate generation
In article-daily-stock.sh
, the touch
of DONE_MARKER
happens before the git push.
touch "$DONE_MARKER"
touch "$DONE_MARKER" # pushใฎๅพใซใๅฟตใฎใใ
Even if git push
fails, DONE_MARKER is up, so the 10:35 catch-up slot exits with SKIP_GEN=1
after audit only. Put DONE_MARKER after the push, and a push failure means the next slot regenerates the same topic โ duplicate generation. It's an easy point to overlook at design time.
โฃ Ignore stale lockdirs and you tie your own hands
Inside the lock mechanism built on mkdir
's atomicity, there's a stale check.
if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
log "ๅฅใคใณในใฟใณในๅฎ่กไธญ(pid=$oldpid) โ skip"; exit 0
fi
rm -rf "$LOCKDIR"; /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi
kill -0
checks whether the process is alive; if it isn't, the lock is treated as stale, rm -rf
'd, and re-acquired. In an early implementation without this check, after the script was force-killed with SIGKILL (e.g. timing out on budget overrun), the lockdir stuck around and the script never ran again from the next day on.
โค The pipefail part of set -uo pipefail kills you in unexpected places
The pipefail
in set -uo pipefail
means "if any one command in the pipe returns non-zero, the whole thing is non-zero." With combinations like jq ... | grep -q ...
, the script can die even in the normal case where grep returns exit 1 for "no match."
jq -r '.[].slug' "$QUEUE" | grep -qx "$NEW_SLUG"
used_slugs | grep -qx "$NEW_SLUG"
used_slugs()
in article-daily-stock.sh
handles this pattern. "Not found" from grep -q
isn't an error but the normal case โ yet under pipefail
it means something different. A bash-specific pitfall.
โฅ I put off detecting the claude binary
In the first implementation, I deferred the handling for "the claude
command isn't found" and moved on. When launched from launchd, ~/.local/bin
isn't on PATH, so in reality it was being silently wiped out every morning with claude: command not found
. The current three-stage fallback was born from that experience.
CLAUDE="${CLAUDE_BIN:-$(command -v claude 2>/dev/null)}"
[ -z "$CLAUDE" ] && [ -x "$HOME/.local/bin/claude" ] && CLAUDE="$HOME/.local/bin/claude"
[ -z "$CLAUDE" ] && CLAUDE=$(ls -t "$HOME"/.nvm/versions/node/*/bin/claude 2>/dev/null | head -1)
[ -x "$CLAUDE" ] || { log "ABORT: claude binary not found"; notify "claude binaryใ็กใ"; exit 0; }
Only after adding this detection did the "claude not found" ABORT log appear and let me grasp the problem. "It should be running" is not running.
โฆ I forgot the ALLOWED_OWNER check and pushed to a fork
An OWNER
check sits immediately before the git push.
OWNER=$(printf '%s' "$URL" | sed -nE 's#.*github\.com[:/]+([^/]+)/.*#\1#p')
if [ "$OWNER" = "$ALLOWED_OWNER" ]; then
fi
During the period when I didn't have this, an automated run fired while origin was still pointed at a different repository during development, and commits piled up in an unrelated repo. Pinning ALLOWED_OWNER=bokuwalily
prevents the worst case even if origin gets changed by mistake.
โง I doubted MIN_ARTICLE_BYTES=1200 without knowing where it came from
The body check in article_ok()
discards anything under 1200 bytes via stat -f%z
. At first I doubted it โ "why 1200 bytes?" โ and tried to change the value. But these are bytes. In UTF-8 a Japanese character is 3 bytes, so 1200 bytes โ 400 characters. It's a floor set at a value that even the opening of a halfway decent technical article should exceed. stat -f%z
is a macOS-specific option (the Linux version is stat -c%s
), so porting to Linux requires a rewrite.
Practical rules derived from actually stepping on this bug and sweeping 5 repos / 8 files / 12 sites.
1. If a full-width character follows a variable expansion, always use ${VAR} form
notify "็ๆๅคฑๆ: $SLUG๏ผๅ่ฉฆ่ก๏ผ"
notify "็ๆๅคฑๆ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ"
One pair of braces. Not a single character of the Japanese log wording has to change.
2. Keep the cross-repo detection command on hand
rg -n --no-heading -g '*.sh' -g '!node_modules' \
'\$[A-Za-z_][A-Za-z0-9_]*[^\x00-\x7F]' ~/dev ~/.claude/scripts ~/bin | rg -v '\$\{'
Don't stop at "I fixed one." The same habit spreads horizontally by copy-paste. This time the same pattern was scattered across 5 repos. Don't hardcode the list of targets โ always run it against the current code.
3. Reproduce launchd's runtime in the terminal and test there
env -i HOME="$HOME" PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" \
CAFFEINATED=1 /bin/bash ~/path/to/script.sh dry
Starting from a nearly empty environment with env -i
lets you check launchd-side behavior locally. "It passes in the terminal" is not "it passes under launchd."
4. Set LANG explicitly in the launchd plist to align the environments
<key>EnvironmentVariables</key>
<dict>
<key>LANG</key>
<string>ja_JP.UTF-8</string>
</dict>
Declaring LANG in the plist makes scheduled runs use ja_JP.UTF-8
too. By "running in the same environment as the terminal," locale-difference bugs become detectable in local testing. This time I chose to fix the code, but aligning the runtime environment is also an option.
5. Do PATH completion before set -u, in the script's first phase
NODE_BIN=$(ls -d "$HOME"/.nvm/versions/node/*/bin 2>/dev/null | sort -V | tail -1)
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"
[ -n "$NODE_BIN" ] && export PATH="${NODE_BIN}:$PATH"
The launchd plist doesn't read shell profiles. Treat every tool that lives under nvm, Homebrew, or ~/.local/bin
as "invisible" and add them explicitly.
6. Resolve the claude binary with a three-stage fallback, and ABORT if not found
Search in the order command -v
โ ~/.local/bin
โ under nvm's bin, and if it's nowhere, exit 0
(retry next slot). I use exit 0
and leave only a log entry, because exit 1
can blow away the notification too.
7. Write notification code to be more robust than the core
If the notification goes down, "it failed" never arrives โ and "never arrived" is indistinguishable from "it succeeded." Two design measures I took in article-daily-stock.sh
to prevent this:
${VAR}
form$LOG
before notifying (if the log write survives, a notification failure is detectable)
log "ABORT: title้ทใใ(${TITLE_CHARS}ๅญ) โ ็ ดๆฃ"
notify "title้ทใใ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ" # ้็ฅใฏๅพ
8. Use set -uo pipefail, but know where || true belongs
Because of pipefail
, the normal case "grep found no match" can be treated as exit 1. || true
exists to absorb that kind of "non-zero that isn't a failure." "|| true
everywhere" is out of the question โ it destroys the value of set -u
. Use it only where "non-zero is normal" is established.
9. touch DONE_MARKER before the git push
touch "$DONE_MARKER" # โ ใใใงใใผใซใผใ็ซใฆใ
git push ...
touch "$DONE_MARKER" # ๅฟตใฎใใไบ้touch๏ผๅช็ญ๏ผ
This is the linchpin of the design where "generation succeeded" holds even if the push fails. With DONE_MARKER up, the next slot skips generation, so a push failure can't cause duplicate generation.
10. Run audit_repair() even on days when generation is skipped
audit_repair
if [ "$MODE" = "audit" ] || [ "$SKIP_GEN" = "1" ]; then
exit 0 # ็ๆใฏในใญใใใauditใฏๆฏๆฅ่ตฐใ
fi
Even on a day where today's article is already generated, the quality of stock from yesterday and earlier can degrade. Thumbnail width checks (confirming at least 2000px via sips -g pixelWidth
) and body stub detection run every slot, and anything problematic gets pushed back into the QUEUE.
11. Don't decide whether something is under git by assumption
git status # ไฝๆฅญ้ๅงๅใซ1็บๆใคใ ใ
This time I proceeded on the assumption that ~/.claude/scripts/
was "a junk location outside git," then ran git status
afterwards and got two modified
entries. Confirming the fact with one command is faster than second-guessing an assumption.
12. Always demonstrate before closing out a fix
LC_ALL=ja_JP.UTF-8 bash -c 'set -u; SLUG=test; echo "็ๆๅคฑๆ: $SLUG๏ผๅ่ฉฆ่ก๏ผ"'
LC_ALL=ja_JP.UTF-8 bash -c 'set -u; SLUG=test; echo "็ๆๅคฑๆ: ${SLUG}๏ผๅ่ฉฆ่ก๏ผ"'
Don't stop at "I think I fixed it." These demonstration commands are also kept in locale-dependent-shell-bugs.md
. Even for a one-line diff, the habit of seeing both the old and new behavior with your own eyes prevents the "I fixed it but it still breaks" round trips.
13. Require real-file grounding for topic planning too
The automatic topic-planning prompt in article-daily-stock.sh
includes constraints like "no fabrication = always ground it in files or scripts that actually exist" and "list 2โ4 real paths in sources
," and the output JSON is validated against that. It's a design to keep automation articles from becoming "fictional implementations." Choosing angles from code you actually run changes the density of an article fundamentally.
The reason this bug went unfound for five days is that two properties overlapped: the usual pattern inverted into "it only dies when run by hand," and "a bug that exists only on the failure path."
ja_JP.UTF-8
โ written off as "a problem with how I invoked it"Writing a full-width ๏ผ
immediately after a variable expansion, as in $SLUG๏ผๅ่ฉฆ่ก๏ผ
, becomes more likely the more carefully you write your logs in Japanese. Scripts written only in English never meet this problem. An ironic property.
The fix itself was done by detecting 12 sites with one line of rg
and adding one pair of braces (5 commits: 2b65662
, 6596c00
, 56288fa
, f0d37f5
, f9b1a18
). But what I really learned this time isn't "how to fix it" โ it's the principle that notification code must be more robust than core code.
When the notification goes down, the fact that it went down never arrives.
An automated system starts rotting the moment you build it, and it won't tell you it's rotting โ which is why audit_repair()
runs every slot, DONE_MARKER
goes up before the git push, and article_ok()
stands by as the gatekeeper for stub detection. The foundation that keeps 30 articles a month generating automatically is protected by a one-line function that rejects anything under 1200 bytes, and by the difference of one pair of braces.
A system is transparent while it's working. You only see it when it breaks. Whether you can see it at that moment is what decides whether the foundation under ยฅ1.2M/month in revenue stays stable.
I've written up the full picture of the system, the breakdown of the ยฅ1.2M/month, and the 30-day procedure in a paid note.
๐ Claude Code่ชๅพ็ฐๅขใงใๅฎ้ใฉใ็จผใใ โ ไป็ตใฟใปๅฎไพใปๅงใๆนใปใตใใผใ
*Written by Lily โ I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio ยท X ยท GitHub*