{"slug": "llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag", "title": "LLM Wiki's Two-Step Chain-of-Thought Ingest: How Incremental Cache and Source Traceability Replace Traditional RAG", "summary": "A developer built LLM Wiki, a cross-platform desktop application that replaces traditional RAG retrieval with a two-phase chain-of-thought ingestion pipeline, generating persistent wiki pages with full source traceability. The system caches intermediate LLM analysis as JSON so incremental document updates cost roughly $0.02 instead of re-embedding an entire corpus, and it propagates source deletions immediately to avoid stale data. The project has reached 19,536 GitHub stars and is trending #10 among TypeScript repositories.", "body_md": "Traditional RAG systems retrieve chunks and answer from scratch every time. LLM Wiki takes a different path: it analyzes documents once, generates persistent wiki pages with full source traceability, and caches intermediate results so incremental updates cost pennies instead of dollars. The result is a cross-platform desktop application (19,536 stars, trending #10 on GitHub TypeScript) that builds and maintains a structured knowledge base rather than ephemeral retrieval.\n\nMost RAG pipelines chunk documents, embed them, and retrieve at query time. LLM Wiki splits ingestion into two explicit phases:\n\nThis separation buys you three things:\n\nTraditional RAG treats the knowledge base as a hidden index. LLM Wiki treats it as the product.\n\nThe ingest queue is a serial processor with crash recovery. Each document enters the queue, gets analyzed, then generates one or more wiki pages. The queue persists to disk, so if the app crashes mid-ingest, it resumes from the last checkpoint.\n\n```\n┌─────────────┐\n│ Raw Source  │ (PDF, Office, EPUB, image, URL)\n└──────┬──────┘\n       │\n       v\n┌─────────────┐\n│ Parse       │ (MinerU, built-in, or cloud PDF processor)\n└──────┬──────┘\n       │\n       v\n┌─────────────┐\n│ Analyze     │ (LLM extracts entities, structure, relationships)\n└──────┬──────┘\n       │\n       v\n┌─────────────┐\n│ Cache       │ (Store intermediate representation)\n└──────┬──────┘\n       │\n       v\n┌─────────────┐\n│ Generate    │ (LLM writes wiki page with source links)\n└──────┬──────┘\n       │\n       v\n┌─────────────┐\n│ Index       │ (Update knowledge graph, vector store)\n└─────────────┘\n```\n\nThe analyze step runs a chain-of-thought prompt that identifies key concepts, relationships, and metadata. The cache stores this analysis as JSON. The generate step reads the cache and writes Markdown with frontmatter that includes source references, creation date, and entity tags.\n\nWhen a source document changes, the system compares the new analysis to the cached version. If the structure is identical, it skips generation. If entities or relationships changed, it regenerates only the affected wiki pages and updates the knowledge graph edges.\n\nThis is cheaper than re-embedding the entire corpus. A 100-page PDF might cost $0.50 to analyze once, then $0.02 per incremental update. Traditional RAG would re-embed all 100 pages every time.\n\nEvery wiki page includes a `sources` array in its frontmatter:\n\n```\n---\ntitle: \"Kubernetes Scheduler Internals\"\nsources:\n  - type: pdf\n    path: raw/sources/k8s-design-docs/scheduler.pdf\n    page: 12\n  - type: image\n    path: raw/sources/diagrams/scheduler-flow.png\n---\n```\n\nThe `raw/sources/` directory is auto-watched. When you drop a new PDF into a subfolder, the ingest queue picks it up. When you delete a file, the system marks the corresponding wiki pages as orphaned and optionally removes them.\n\nThis solves a common RAG problem: stale data. If you delete a source document, traditional RAG keeps serving chunks from it until you manually rebuild the index. LLM Wiki propagates deletions immediately.\n\nThe knowledge graph uses four signals to compute relevance between wiki pages:\n\n| Signal | Description | Weight | \n|---|---|---|\n| Direct links | Explicit `[[wikilinks]]` in page content | High | \n| Source overlap | Pages derived from the same source document | Medium | \n| Adamic-Adar | Shared neighbors weighted by neighbor rarity | Medium | \n| Type affinity | Pages with similar entity types (person, concept, tool) | Low | \n\nThe graph runs Louvain community detection to cluster pages into topics. Each cluster gets a cohesion score based on internal edge density. The UI surfaces \"surprising connections\" (high Adamic-Adar score between distant clusters) and \"knowledge gaps\" (low-cohesion clusters with few internal links).\n\nThis is useful for agents that need to explore a knowledge base without manual tagging. The graph provides a navigation layer that RAG's flat vector space cannot.\n\nLLM Wiki extracts images from PDFs and runs them through a vision LLM to generate factual captions. The captions are indexed alongside text, so image-aware search returns both text snippets and relevant images.\n\nThe lightbox preview shows the image, caption, and a \"jump to source\" button that opens the original PDF at the correct page. This is harder than it sounds: you need to track image coordinates in the PDF, map them to page numbers, and preserve that metadata through the ingest pipeline.\n\nThe vision LLM prompt is tuned for factual descriptions, not creative captions. It avoids phrases like \"a beautiful sunset\" and focuses on \"bar chart showing Q3 revenue by region.\"\n\nLLM Wiki includes optional vector search via LanceDB. You can configure any OpenAI-compatible embedding endpoint. The vector store indexes wiki page content, not raw source chunks.\n\nThis is a key difference from traditional RAG. The embeddings represent synthesized knowledge, not raw text. When you search for \"Kubernetes scheduler latency,\" you retrieve wiki pages that summarize scheduler behavior, not individual PDF paragraphs.\n\nThe system supports hybrid search: combine vector similarity with knowledge graph traversal to find pages that are semantically close *and* structurally connected.\n\nThe ingest queue is a JSON file on disk. Each entry includes:\n\nIf the app crashes, it reads the queue file on restart and resumes from the last checkpoint. You can cancel or retry individual items from the UI.\n\nThis is critical for long-running ingests. A folder with 500 PDFs might take hours to process. Without crash recovery, a single failure would force you to start over.\n\nThe folder import preserves directory structure. If you import `raw/sources/projects/alpha/`, the wiki creates a `projects/alpha/` namespace with pages grouped by folder.\n\nThe folder name becomes a classification hint for the LLM. A file in `raw/sources/legal/contracts/` gets analyzed with a prompt that mentions \"legal contract context.\" This improves entity extraction without requiring manual metadata.\n\nLLM Wiki lets you configure models per project. You can route Chat and Ingest to different endpoints:\n\nYou can add custom headers for API keys, set streaming preferences, and configure retry logic. The system supports any OpenAI-compatible endpoint, including local models via Ollama or vLLM.\n\nThis mode restricts the LLM to answering exclusively from imported sources. It disables general knowledge and forces the model to cite wiki pages or source documents.\n\nThis is useful for compliance-sensitive workflows where you need to prove every answer came from approved material. The system logs every source citation, so you can audit the retrieval path.\n\nYou can export a complete project as a `.llmwiki` archive. The archive includes:\n\nImport the archive on another device, and the wiki rebuilds its index from the exported pages. This is faster than re-ingesting sources because the analysis cache is included.\n\nCommon failure modes:\n\nObservability is minimal. The UI shows ingest progress and error logs, but there is no structured telemetry or distributed tracing. For production deployments, you would need to add OpenTelemetry instrumentation.\n\n| Dimension | LLM Wiki | Traditional RAG | \n|---|---|---|\n| Knowledge artifact | Persistent wiki pages | Ephemeral retrieval | \n| Incremental updates | Cache analysis, regenerate pages | Re-embed entire corpus | \n| Source traceability | Every page links to source | Chunk metadata only | \n| Knowledge graph | 4-signal graph with clustering | Flat vector space | \n| Multimodal | Vision LLM captions, lightbox | Text-only or basic OCR | \n| Crash recovery | Persistent queue with checkpoints | Restart from scratch | \n| Cost per update | $0.02 (incremental) | $0.50 (full re-embed) | \n\n**Use LLM Wiki when:**\n\n**Avoid LLM Wiki when:**\n\nLLM Wiki shifts the cost from query time to ingest time. If you query the same knowledge base repeatedly, this trade-off pays off. If you ingest once and query once, traditional RAG is simpler.", "url": "https://wpnews.pro/news/llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag", "canonical_source": "https://dev.to/mech_app_ai/llm-wikis-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-traceability-replace-4acc", "published_at": "2026-09-15 10:06:52+00:00", "updated_at": "2026-09-15 10:39:16.173311+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["LLM Wiki", "GitHub", "MinerU", "Louvain"], "alternates": {"html": "https://wpnews.pro/news/llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag", "markdown": "https://wpnews.pro/news/llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag.md", "text": "https://wpnews.pro/news/llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag.txt", "jsonld": "https://wpnews.pro/news/llm-wiki-s-two-step-chain-of-thought-ingest-how-incremental-cache-and-source-rag.jsonld"}}