{"slug": "show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian", "title": "Show HN: ObsidianYouTubeSync – Sync YouTube History into Obsidian", "summary": "A developer released ObsidianYouTubeSync, an open-source tool that downloads a user's full YouTube watch history via browser cookies and yt-dlp, extracts captions and transcripts, summarizes each video with Google Gemini, and writes each video as a Markdown note with YAML frontmatter and hierarchical AI-assigned tags into an Obsidian vault. The tool adds incremental sync that skips already-processed videos, automatic Shorts skipping, vault-wide retagging, ThreadPoolExecutor-based parallel processing, and a native Model Context Protocol server for AI agent integration, with output formatted for GraphRAG, LlamaIndex, LangChain and OpenClaw pipelines.", "body_md": "# 📺 ObsidianYouTubeSync — YouTube History Downloader, Transcript Extractor & AI Knowledge Graph Builder\n\n**Download, sync, and organize your entire YouTube watch history into a structured, AI-tagged knowledge graph — ready for Obsidian, OpenClaw, and GraphRAG pipelines.**\n\nThis tool automates the full pipeline: it downloads YouTube video metadata, extracts captions and transcripts, generates LLM-powered summaries using Google Gemini (GenAI), and saves each video as a richly categorized Markdown note with a consistent hierarchical taxonomy. Built for users with **large YouTube watch libraries** who want to transform passive viewing history into an active, searchable, and connectable dataset.\n\n**Use cases:** Personal knowledge management (PKM), YouTube history cataloging, knowledge graph construction, transcript dataset preparation, GraphRAG ingestion, OpenClaw/LLM agent tooling, personal AI training data, second brain building.\n\nIf you have hundreds or thousands of videos in your YouTube history, finding insights buried in that content is nearly impossible without structure. This tool solves that by:\n\n- **Downloading** your full YouTube watch history (incremental or bulk) using`yt-dlp`\n- **Extracting** full captions and transcripts for every video (with fallback to auto-generated captions)\n- **Summarizing** each video using Google Gemini LLM (GenAI) — concise, searchable summaries\n- **Categorizing** content with a shared hierarchical taxonomy (e.g.,`Technology/ArtificialIntelligence` ,`Science/Neuroscience` ) using AI classification\n- **Organizing** notes into Obsidian with clean YAML frontmatter (tags, url, channel, date, summary) — queryable via Dataview\n- **Enabling** downstream pipelines: export structured notes as a knowledge graph dataset for**GraphRAG** ,**OpenClaw** , vector databases, or LLM fine-tuning\n\n``` php\ngraph TD\n    A[YouTube Watch History] -->|Browser Cookies & yt-dlp| B(Extract Metadata & Transcripts)\n    B --> C{Google Gemini GenAI}\n    C -->|Analyze text| D[Generate Concise Summary]\n    C -->|Classify concepts| E[Apply AI Taxonomy Tags]\n    D --> F[Construct Structured Markdown]\n    E --> F\n    F --> G[(Obsidian Vault / Knowledge Graph)]\n    G --> H[Ready for GraphRAG & Agents]\n    \n    classDef default fill:#f9f9f9,stroke:#5c3d9e,stroke-width:2px,color:#333;\n    classDef ai fill:#e6dfff,stroke:#8b6cef,stroke-width:2px,color:#333;\n    class C ai;\n```\n\n| Feature | Details | \n|---|---|\n| **LLM Summarization** | Google Gemini (GenAI) generates concise, high-quality summaries for every video | \n| **Full Transcript Download** | Pulls captions/transcripts (manual + auto-generated) from YouTube | \n| **Hierarchical AI Tagging** | CamelCase taxonomy tags like `Technology/AI/LLM` ,`Person/ElonMusk` | \n| **History Sync** | Download & sync your entire YouTube watch history via browser cookies + `yt-dlp` | \n| **Incremental Updates** | Smart detection — only processes new videos, skips already-synced ones | \n| **YouTube Shorts Skip** | Automatically detects and skips Shorts to keep your knowledge base high-signal | \n| **Vault-wide Retagging** | Retag any folder (Apple Notes, Books, etc.) using the same global AI taxonomy | \n| **Parallel Processing** | `ThreadPoolExecutor` with configurable pool — syncs 100s of videos in minutes | \n| **MCP Server** | Native [Model Context Protocol](https://modelcontextprotocol.io) server for AI agent integration | \n| **GraphRAG-Ready Output** | Structured Markdown with rich metadata nodes for knowledge graph pipelines | \n\nEach YouTube video becomes a structured Markdown node with rich metadata. This format is directly ingestible by **OpenClaw**, **GraphRAG**, LlamaIndex, LangChain, or any knowledge graph pipeline:\n\n```\n---\nurl: https://www.youtube.com/watch?v=VIDEO_ID\nchannel: \"Lex Fridman\"\ndate_synced: 2026-02-27\nsummary: \"A deep conversation on the nature of intelligence and the trajectory of AGI development, exploring consciousness, Gödel's theorems, and the limits of computation.\"\ntags:\n  - Technology\n  - Technology/ArtificialIntelligence\n  - Technology/ArtificialIntelligence/LLM\n  - Science/CognitiveScience\n  - Person/LexFridman\n  - Person/JohnCarmack\n  - AGI\n  - ConsciousnessDebate\n  - ComputationalLimits\n---\n\n# The Future of AGI with John Carmack\n\n## Extracted Links\n- https://en.wikipedia.org/wiki/Gödel%27s_incompleteness_theorems\n\n## Description\nIn this episode, Lex Fridman sits down with John Carmack to discuss...\n\n## Raw Transcript\n[Full verbatim transcript for semantic search, embedding, and RAG ingestion]\n```\n\n- **Entities** (`Person/` ,`Company/` ,`Location/` ) form graph nodes\n- **Taxonomy tags** form typed edges between concepts\n- **Summaries** are ideal embedding targets for semantic similarity\n- **Transcripts** provide raw text for chunking, dense retrieval, and fine-tuning\n- **Dates + channels** enable temporal and source-based graph traversals\n\n- **Python 3.10+**\n- **uv** — Fast Python Package Manager ([Installation Guide](https://docs.astral.sh/uv/getting-started/installation/) )\n- **Google Gemini API Key** — free tier available at[Google AI Studio](https://aistudio.google.com/)\n- **Chrome, Safari, or Firefox** logged into YouTube (for`yt-dlp` history access)\n- **Obsidian** (optional but recommended for visualization and Dataview queries)\n\n```\n# Clone the repository\ngit clone https://github.com/mr8lu/ObsidianYouTubeSync.git\ncd ObsidianYouTubeSync\n\n# Set up Python environment using uv (Fast Python Package Manager)\nuv sync\n# Copy the example environment file\ncp .env.example .env\n```\n\nEdit `.env` with your credentials:\n\n```\nGEMINI_API_KEY=your_gemini_api_key_here\n\n# Optional: Webshare rotating proxy (recommended for large history syncs)\nWEBSHARE_PROXY_USER=your_proxy_user\nWEBSHARE_PROXY_PASS=your_proxy_password\n```\n\n**Important — Vault Path**: The default target directory is `~/Documents/Obsidian Vault`. **You must update `OBSIDIAN_VAULT_PATH` in `sync.py` and `retag_notes.py` if your vault lives elsewhere** before running any syncs.\n\n```\n# Full sync (downloads and updates your watch history, skipping existing videos)\n./run.sh --sync\n\n# Re-tag and re-categorize all existing notes with the latest taxonomy\n./run.sh --retag\n```\n\nApply consistent LLM-powered categorization to *any* existing notes folder:\n\n```\n# Dry-run preview (no files modified)\npython3 retag_notes.py --folder \"~/Documents/Obsidian Vault/Apple Notes\" --dry-run\n\n# Live retag\npython3 retag_notes.py --folder \"~/Documents/Obsidian Vault/Books\"\n\n# Control parallelism\npython3 retag_notes.py --folder \"~/Documents/Obsidian Vault/Books\" --workers 10\n```\n\nThis toolkit is designed to be fully manageable by local-first AI agents. There are two primary integration paths:\n\nYou can teach the [OpenClaw](https://openclaw.ai) Mac AI agent how to autonomously run your syncs and retag folders using a native `SKILL.md` file.\n\n**Installation:**\nCreate a folder at `~/.openclaw/skills/obsidian-youtube-sync/` and drop in a `SKILL.md` file that explain how to execute `./run.sh` and `uv run retag_notes.py`. (A working template is provided upon request).\nOnce installed, you can simply ask OpenClaw: *\"Sync my YouTube watch history and let me know what themes I learned about today.\"*\n\nAlternatively, you can expose the toolkit as an **MCP server** to Claude Desktop, Cursor, or other MCP clients.\n\n**Setup for Claude Desktop:**\nAdd the following to your `claude_desktop_config.json`:\n\n```\n{\n  \"mcpServers\": {\n    \"ObsidianYouTubeSync\": {\n      \"command\": \"uv\",\n      \"args\": [\n        \"--directory\",\n        \"/absolute/path/to/ObsidianYouTubeSync\",\n        \"run\",\n        \"mcp_server.py\"\n      ]\n    }\n  }\n}\n```\n\n*Note: Ensure you replace `/absolute/path/to/` with the actual path to the repository on your machine.*\n\nThis tool is purpose-built as a **data pipeline** for knowledge graph construction. Here's how to plug it into downstream systems:\n\n1. Run `./run.sh --sync` to sync your complete YouTube history\n2. Point your GraphRAG pipeline at the `~/Documents/Obsidian Vault/YouTube/` folder\n3. The YAML frontmatter is parsed as node metadata; transcripts are chunked for retrieval; tags form the typed edge schema\n\n- Each note file is a self-contained document node with entity tags (`Person/` ,`Company/` ,`Location/` ) pre-extracted\n- The taxonomy hierarchy (`Technology/ArtificialIntelligence/LLM` ) maps directly to ontological class trees\n- Channel metadata provides provenance edges in the graph\n\n- Summaries are ideal for dense embedding (short, factual, topic-rich)\n- Transcripts can be chunked with metadata-aware splitters\n- Tags can be used as filters/facets for hybrid search\n\n- Generate instruction-response pairs from `(transcript) → summary` pairs\n- Use hierarchical tags as classification labels\n- Export with a simple Python script reading the YAML frontmatter\n\n| Feature | Status | \n|---|---|\n| YouTube History Sync (incremental & full) | ✅ Done | \n| AI Summarization via Google Gemini (GenAI) | ✅ Done | \n| Full Transcript / Caption Extraction | ✅ Done | \n| Hierarchical AI Taxonomy Tagging | ✅ Done | \n| Parallel Processing (ThreadPoolExecutor) | ✅ Done | \n| Vault-wide Retagging Engine | ✅ Done | \n| Proxy / Rate-limit Support | ✅ Done | \n| MCP Server for AI Agent Integration | ✅ Done | \n| GraphRAG Export Helper Script | 🔜 Planned | \n| Obsidian Community Plugin | 🔜 Planned | \n| Notion / Logseq Export | 🔜 Planned | \n| Local LLM Support (Ollama) | 💡 Considering | \n| YouTube Playlist Sync | 💡 Considering | \n| OpenClaw Native Connector | 💡 Considering | \n\n- **Native YAML Properties** :`tags:` ,`url:` ,`summary:` ,`channel:` ,`date_synced` — all recognized by Obsidian natively\n- **Hierarchical Tags** :`Parent/Child` tags map to Obsidian's Tag Pane tree view\n- **Dataview Queries** : Query your entire watch history like a database\n- **Graph View** : Entity tags create visual connections between videos sharing people, companies, or concepts\n\n- **Local-First** : All notes live on your machine. No cloud sync.\n- **No Account Data** : Only public video metadata (title, description, transcript) is sent to Gemini for summarization. Your YouTube account credentials are never accessed or stored.\n- **Credential Security** : API keys loaded via`.env` (excluded from git). Proxy credentials stored locally only.\n- **macOS Permissions** : Ensure Terminal has \"Full Disk Access\" in System Settings → Privacy & Security.\n\n| Environment | Supported | \n|---|---|\n| macOS (Intel & Apple Silicon) | ✅ | \n| Linux | ✅ | \n| Windows (WSL) |  | \n| Python 3.10+ | ✅ | \n| Obsidian 1.0+ | ✅ | \n| Chrome / Safari / Firefox cookies | ✅ | \n| Claude Desktop (MCP) | ✅ | \n| GraphRAG / LlamaIndex | ✅ | \n| OpenClaw | ✅ | \n\nContributions are welcome! See [CONTRIBUTING.md](https://github.com/mr8lu/ObsidianYouTubeSync/blob/main/CONTRIBUTING.md) for how to get started.\n\nIf you find this useful, please ⭐ **star the repo** — it helps more researchers, PKM enthusiasts, and knowledge graph builders discover it.\n\nDistributed under the MIT License. See `LICENSE` for more information.", "url": "https://wpnews.pro/news/show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian", "canonical_source": "https://github.com/mr8lu/ObsidianYouTubeSync", "published_at": "2026-10-23 19:48:14+00:00", "updated_at": "2026-10-23 20:12:38.541332+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "agent-protocols", "large-language-models", "generative-ai"], "entities": ["ObsidianYouTubeSync", "Obsidian", "YouTube", "Google Gemini", "yt-dlp", "Model Context Protocol", "GraphRAG", "OpenClaw"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian", "markdown": "https://wpnews.pro/news/show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian.md", "text": "https://wpnews.pro/news/show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian.txt", "jsonld": "https://wpnews.pro/news/show-hn-obsidianyoutubesync-sync-youtube-history-into-obsidian.jsonld"}}