Turning Conversation Logs into Obsidian Long-Term Memory: A Pipeline That Distills Them Every Night

A developer built a nightly pipeline that distills conversation logs from Claude and Codex into an Obsidian vault for long-term memory. The system runs via launchd, handles silent failures with preflight checks and notifications, and splits the distillation into two runs to avoid timeouts. Key lessons include preventing fabrication by using calendar API snapshots instead of prose descriptions.

This is part of my "Claude Code environment" series. In the post about splitting memory into four layers https://zenn.dev/bokuwalily/articles/f534402187bd07 I described "conversation logs = layer 1" and "the Obsidian Vault = layer 4." This time I'll write about the implementation of the plumbing that distills from layer 1 into layer 4, automatically, every night. Raw conversation logs are enormous and unreadable. The human-authored Obsidian wiki, on the other hand, is structured long-term memory. I run an unattended job that appends "just the last 28 hours, every night, while preserving the domain structure" from the former into the latter. What this article is really about is the way to kill silent failures and the mechanisms that prevent fabrication — things I only figured out by actually doing it. launchd fires it every morning, and it's roughly a three-step process. extract conversations.py claude -p , append only the last 28 hours of diffs to the relevant domain in the Vault git commit + push to a private repo so a mess can be reverted The distillation is split into two runs by source Claude conversations and Codex conversations . I'll explain why below, but the short version is that combining them into one run overran the time window and timed out every single night. This is where everything got stuck first. The Vault lives under ~/Documents , which is synced by iCloud and protected by macOS TCC privacy protection . Even running git from launchd fails because it can't write into the protected area. The nasty part is that this tends to become a "silent failure." So I detect it early in a preflight check and log it loudly plus send a notification. launchd配下では ~/Documents 保護領域 に触れない場合がある。 ここで早期検知し、サイレント失敗 exit 0偽装 を防ぐ。 if cd "$VAULT" && git rev-parse --git-dir /dev/null 2 &1 ; then echo "❌ FDA未付与: launchdから '$VAULT' にアクセス不可 TCC保護 " "$LOG" notify fail "FDA未付与（設定→フルディスクアクセス→/bin/bash を許可）" exit 1 fi The fix is to "grant Full Disk Access to /bin/bash under System Settings → Privacy & Security → Full Disk Access." The script itself lives outside the protected area ~/.claude/scripts/ . If you put it in ~/Documents , launchd can't exec it. I also tried escaping to ~/ via a symlink, but that doesn't work becauseiCloud treats the symlink as a conflict and breaks it.When a protected area overlaps with iCloud sync, the constraints pile up in counterintuitive ways. Keeping the script outside protection and the data inside protection — drawing that line firmly — is the stable choice. If you close the lid and it sleeps, the nightly job freezes. caffeinate -s only works on AC power, so on battery it just stops. So I made it self-recovering: "retry across multiple slots until it succeeds, and quit immediately once it does." exit 0 a no-op that doesn't even dirty the log mkdir lock stale locks are auto-reclaimed based on process liveness DONE MARKER="$HOME/.claude/logs/.vault-ingest-done-${TODAY}" -f "$DONE MARKER" && exit 0 本日成功済みなら即終了 Splitting the distillation into two runs Claude/Codex was for the same reason. On busy days, digesting 28 hours of logs plus rewriting articles didn't fit in the 40-minute window, so it timed out day after day, and that in turn froze the hot cache. By giving each sub-step its own independent marker , even if one times out the next slot retries just the remainder. This is the single most important lesson. When I threw "write today's brief from the conversation logs" at claude -p , it would sometimes plausibly invent commitment hours or durations for events that aren't in the notes. Seeing prose like "M/D–M/D," it would inflate it into "I'm tied up for this entire period." There are two countermeasures. ① Make an actual snapshot of the calendar the canonical source. Instead of prose, I dump the time-stamped events pulled from the Google Calendar API into a separate file as ground truth, and instruct: "read this as the one and only canonical schedule." ② On fetch failure, preserve the last-known-good. Even if the API fails, don't wipe the canonical file to empty. I keep the previously successful values with a stale mark. if -n "$CAL SRC" ; then cp "$CAL TMP" "$CAL SNAPSHOT"; cp "$CAL SNAPSHOT" "$CAL LASTGOOD" elif -s "$CAL LASTGOOD" ; then 取得失敗: 前回goodを温存し ⚠️stale 印で再構築（last-known-good） { echo "⚠️ 本日取得失敗。以下は前回成功時点の値 stale 。"; tail -n +4 "$CAL LASTGOOD"; } "$CAL SNAPSHOT" fi And I constrain it hard on the prompt side too: "Don't add information that isn't in the notes by guessing." "For date and time claims, explicitly separate confirmed information from speculation." If you're going to let an AI write your long-term memory, the only option is to structurally eliminate room for invention. When archiving a brief with a date stamp, I only accept files newer than this run's start time as today's output. START STAMP=$ mktemp ... ラン開始時刻スタンプ ...生成... if -s "$BRIEF SRC" && "$BRIEF SRC" -nt "$START STAMP" ; then cp "$BRIEF SRC" "$ARCH FILE"; touch "$DONE MARKER" 新しい時だけ完了扱い else notify fail "ブリーフ未完 — 次スロットで自動再試行" 古いまま＝失敗、再試行へ fi Without this, even when generation times out, it would copy "yesterday's old brief" and record it as "success." By judging freshness with mtime, I make sure the job never disguises a failure as a success. Next time I'll write about the boss of this unattended job — running Claude Code's autonomous loop 24 hours a day and stopping runaways with cost https://zenn.dev/bokuwalily/articles/claude-autopilot . Written by Lily — I ship iOS apps and automate my content stack with Claude Code. Follow along: Portfolio https://bokuwalily.com · X https://x.com/bokuwalily · GitHub https://github.com/bokuwalily