{"slug": "turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them", "title": "Turning Conversation Logs into Obsidian Long-Term Memory: A Pipeline That Distills Them Every Night", "summary": "A developer built a nightly pipeline that distills conversation logs from Claude and Codex into an Obsidian vault for long-term memory. The system runs via launchd, handles silent failures with preflight checks and notifications, and splits the distillation into two runs to avoid timeouts. Key lessons include preventing fabrication by using calendar API snapshots instead of prose descriptions.", "body_md": "This is part of my \"Claude Code environment\" series. In [the post about splitting memory into four layers](https://zenn.dev/bokuwalily/articles/f534402187bd07) I described \"conversation logs = layer 1\" and \"the Obsidian Vault = layer 4.\" This time I'll write about the implementation of **the plumbing that distills from layer 1 into layer 4, automatically, every night.**\n\nRaw conversation logs are enormous and unreadable. The human-authored Obsidian wiki, on the other hand, is structured long-term memory. I run an unattended job that appends \"just the last 28 hours, every night, while preserving the domain structure\" from the former into the latter. What this article is really about is the **way to kill silent failures** and the **mechanisms that prevent fabrication** — things I only figured out by actually doing it.\n\nlaunchd fires it every morning, and it's roughly a three-step process.\n\n`extract_conversations.py`\n\n)`claude -p`\n\n, append only the last 28 hours of diffs to the relevant domain in the Vault`git commit`\n\n+ push to a private repo (so a mess can be reverted)The distillation is split into two runs by source (Claude conversations and Codex conversations). I'll explain why below, but the short version is that **combining them into one run overran the time window and timed out every single night.**\n\nThis is where everything got stuck first. The Vault lives under `~/Documents`\n\n, which is synced by iCloud and protected by macOS TCC (privacy protection). **Even running git from launchd fails because it can't write into the protected area.**\n\nThe nasty part is that this tends to become a \"silent failure.\" So I **detect it early in a preflight check and log it loudly plus send a notification.**\n\n```\n# launchd配下では ~/Documents(保護領域) に触れない場合がある。\n# ここで早期検知し、サイレント失敗(exit 0偽装)を防ぐ。\nif ! ( cd \"$VAULT\" && git rev-parse --git-dir >/dev/null 2>&1 ); then\n  echo \"❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)\" >> \"$LOG\"\n  notify_fail \"FDA未付与（設定→フルディスクアクセス→/bin/bash を許可）\"\n  exit 1\nfi\n```\n\nThe fix is to \"grant Full Disk Access to `/bin/bash`\n\nunder System Settings → Privacy & Security → Full Disk Access.\" The script itself lives outside the protected area (`~/.claude/scripts/`\n\n). If you put it in `~/Documents`\n\n, launchd can't exec it.\n\nI also tried escaping to\n\n`~/`\n\nvia a symlink, but that doesn't work becauseiCloud treats the symlink as a conflict and breaks it.When a protected area overlaps with iCloud sync, the constraints pile up in counterintuitive ways. Keeping the script outside protection and the data inside protection — drawing that line firmly — is the stable choice.\n\nIf you close the lid and it sleeps, the nightly job freezes. `caffeinate -s`\n\nonly works on AC power, so on battery it just stops. So I made it self-recovering: **\"retry across multiple slots until it succeeds, and quit immediately once it does.\"**\n\n`exit 0`\n\n(a no-op that doesn't even dirty the log)`mkdir`\n\nlock (stale locks are auto-reclaimed based on process liveness)\n\n```\nDONE_MARKER=\"$HOME/.claude/logs/.vault-ingest-done-${TODAY}\"\n[ -f \"$DONE_MARKER\" ] && exit 0   # 本日成功済みなら即終了\n```\n\nSplitting the distillation into two runs (Claude/Codex) was for the same reason. On busy days, digesting 28 hours of logs plus rewriting articles didn't fit in the 40-minute window, so it timed out day after day, and that in turn froze the hot cache. By giving **each sub-step its own independent marker**, even if one times out the next slot retries just the remainder.\n\nThis is the single most important lesson. When I threw \"write today's brief from the conversation logs\" at `claude -p`\n\n, it would sometimes **plausibly invent commitment hours or durations for events that aren't in the notes.** Seeing prose like \"M/D–M/D,\" it would inflate it into \"I'm tied up for this entire period.\"\n\nThere are two countermeasures.\n\n**① Make an actual snapshot of the calendar the canonical source.** Instead of prose, I dump the time-stamped events pulled from the Google Calendar API into a separate file as ground truth, and instruct: \"read this as the one and only canonical schedule.\"\n\n**② On fetch failure, preserve the last-known-good.** Even if the API fails, don't wipe the canonical file to empty. I keep the previously successful values with a `stale`\n\nmark.\n\n```\nif [ -n \"$CAL_SRC\" ]; then\n  cp \"$CAL_TMP\" \"$CAL_SNAPSHOT\"; cp \"$CAL_SNAPSHOT\" \"$CAL_LASTGOOD\"\nelif [ -s \"$CAL_LASTGOOD\" ]; then\n  # 取得失敗: 前回goodを温存し ⚠️stale 印で再構築（last-known-good）\n  { echo \"⚠️ 本日取得失敗。以下は前回成功時点の値(stale)。\"; tail -n +4 \"$CAL_LASTGOOD\"; } > \"$CAL_SNAPSHOT\"\nfi\n```\n\nAnd I constrain it hard on the prompt side too: \"Don't add information that isn't in the notes by guessing.\" \"For date and time claims, explicitly separate confirmed information from speculation.\" If you're going to let an AI write your long-term memory, **the only option is to structurally eliminate room for invention.**\n\nWhen archiving a brief with a date stamp, I **only accept files newer than this run's start time as today's output.**\n\n```\nSTART_STAMP=$(mktemp ...)          # ラン開始時刻スタンプ\n# ...生成...\nif [ -s \"$BRIEF_SRC\" ] && [ \"$BRIEF_SRC\" -nt \"$START_STAMP\" ]; then\n  cp \"$BRIEF_SRC\" \"$ARCH_FILE\"; touch \"$DONE_MARKER\"   # 新しい時だけ完了扱い\nelse\n  notify_fail \"ブリーフ未完 — 次スロットで自動再試行\"   # 古いまま＝失敗、再試行へ\nfi\n```\n\nWithout this, even when generation times out, it would copy \"yesterday's old brief\" and record it as \"success.\" By judging freshness with mtime, I make sure the job **never disguises a failure as a success.**\n\nNext time I'll write about the boss of this unattended job — [running Claude Code's autonomous loop 24 hours a day and stopping runaways with cost](https://zenn.dev/bokuwalily/articles/claude-autopilot).\n\n*Written by **Lily** — I ship iOS apps and automate my content stack with Claude Code.\n\nFollow along: [Portfolio](https://bokuwalily.com) · [X](https://x.com/bokuwalily) · [GitHub](https://github.com/bokuwalily)*", "url": "https://wpnews.pro/news/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them", "canonical_source": "https://dev.to/bokuwalily/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-distills-them-every-night-18c2", "published_at": "2026-07-12 05:00:05+00:00", "updated_at": "2026-07-12 05:12:50.997959+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-agents"], "entities": ["Claude", "Codex", "Obsidian", "iCloud", "Google Calendar", "launchd", "macOS TCC"], "alternates": {"html": "https://wpnews.pro/news/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them", "markdown": "https://wpnews.pro/news/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them.md", "text": "https://wpnews.pro/news/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them.txt", "jsonld": "https://wpnews.pro/news/turning-conversation-logs-into-obsidian-long-term-memory-a-pipeline-that-them.jsonld"}}