# LoreDocs: a local knowledge vault > Source: > Published: 2026-08-20 00:00:00+00:00 When you spend hours building a model, you want the insights you capture to stay with you long after the notebook closes. A conversation with an AI assistant can surface a useful snippet, but when the session ends that context disappears. For data engineers and AI practitioners who move between experiments, code reviews, and production pipelines, that fleeting memory creates hidden rework. LoreDocs was built to give that knowledge a permanent home, so the effort you invest in prompting and discovery never gets lost. ## Session memory is a great start, but it is not a knowledge base AI chat tools excel at keeping a short-term thread alive. They can remember the last few hundred tokens, let you ask follow-up questions, and even summarize a recent discussion. That works well for a single debugging session, but it falls short when you need to reference a design decision made weeks ago, trace the evolution of a data pipeline, or share a piece of documentation with a teammate who joins the project later. The problem is twofold. The memory lives in the runtime of the chat server. When the process restarts, the context is gone. The memory is also tied to a single conversation, not to the broader set of artifacts that make up an AI workflow -- code files, experiment logs, schema definitions, and external notes. When you have to re-enter that information, you waste time and risk inconsistency. A durable store that can be queried, versioned, and linked to the files you already own solves those gaps. It lets you treat the output of every chat as a primary artifact, just like a Git commit or a data table. That is the premise behind LoreDocs: a local, portable vault that lives alongside your code and gives you the same level of control you expect from a database. LoreConvo handles session memory -- it saves and recalls the thread of a conversation automatically. LoreDocs is the complementary layer for everything that needs to survive longer than a session: schemas, experiment logs, model notes, and the design decisions that shape a project over months. They are different retrieval problems, and we built separate tools for them rather than trying to stretch one into the other. For a deeper look at how the two products work together, see [LoreConvo and LoreDocs on the Anthropic Marketplace](/blog/loreconvo-loredocs-anthropic-marketplace) and [the technical case for a dedicated vault store](/blog/loredocs-deep-dive-technical-case). ## A vault that lives where your projects live LoreDocs stores everything in a single SQLite file that you own. Because the file sits on your own filesystem, you retain full control over backups, encryption, and sharing. There is no hidden cloud service that could become a single point of failure, and you can move the file to a new machine simply by copying it. The vault model is built around named vaults. You can create separate vaults for different teams, projects, or domains, and tag them for easy discovery. The workspace-scoped auto-vault feature mirrors the way many data engineering tools bind configuration to a directory: calling `vault_open_workspace(path)` either opens an existing vault bound to that path or creates a new one. The call is idempotent, so onboarding a new repository is as simple as opening the folder -- the vault appears automatically without extra configuration steps. Free users start with three vaults, which covers most personal experiments. The Pro tier removes that limit and unlocks semantic search and auto-discovered document relationships when a project's knowledge base grows beyond what keyword search can navigate efficiently. ## Turning raw files into searchable knowledge A knowledge vault earns its keep when you can find what you need quickly. LoreDocs provides two complementary search mechanisms. The built-in FTS5 full-text search lets you locate documents by keyword across all vaults, using the same indexing technology that powers many modern databases. For data engineers who already rely on SQLite, this feels familiar and performant without any additional infrastructure. Pro users also get a hybrid semantic search built on LanceDB. The system creates embeddings for each paragraph, combines them with BM25 full-text scores, and ranks results with reciprocal rank fusion. Documents are split at paragraph boundaries, keeping each chunk under 256 tokens to preserve context while staying within model limits. You enable semantic mode with a single flag on `vault_search` , and the index can be rebuilt on demand if you add a large batch of documents. Importing existing knowledge is straightforward. If you already maintain an Obsidian vault, point `vault_import_dir` at the root folder and LoreDocs will walk the directory tree, read markdown files, and extract YAML frontmatter tags automatically. For ad-hoc files, `vault_add_doc` accepts a path to a text file, making it easy to add logs, experiment notes, or code snippets without leaving the terminal. Because every document is versioned, you can always roll back to a previous state -- useful when you need to understand why a model was tuned a certain way or trace a schema change across several iterations. ## Linking chat, code, and data without friction LoreDocs integrates directly with the tools you already use through a suite of MCP (Model Context Protocol) tools that expose vault operations as native actions. Whether you are working in Claude Code, Cowork, OpenAI Codex, or Hermes Agent, you can add a `.mcp.json` file to your project and the vault becomes available to the AI without any additional setup. When you run a chat session with LoreConvo, the assistant can pull relevant passages from the vault in real time. The `vault_prime` call injects the entire context of a chosen vault into a single request, allowing the model to answer questions that depend on multiple documents. This eliminates the need to copy-paste snippets manually and reduces the chance of misquoting a source. For environments that do not read `.mcp.json` , LoreDocs offers a Python fallback script. Any agent that can execute Python can use it to retrieve documents, making the vault accessible even in custom pipelines or batch jobs. Because the vault is stored locally, a data pipeline can also read the latest version of a schema definition directly from the vault, ensuring that transformation logic always matches the documented contract. ## Bringing it all together LoreDocs was built to fill the gap between fleeting AI chat memory and a robust, searchable knowledge repository. Local SQLite storage, multi-vault organization, full-text and semantic search, versioning, and MCP integration combine to turn every conversation, note, and code snippet into a durable artifact you own and control. Paired with LoreConvo, you get a workflow where the assistant can retrieve and reference the exact documentation that informed a decision, and you can later audit that decision through the version history. The result is less time re-entering information, fewer inconsistencies across experiments, and a clearer path from prototype to production. Explore the full LoreDocs feature set at [/tools](/tools), or reach out at [/contact](/contact) if you want to talk through how a local knowledge vault fits into your data engineering pipeline. *Get posts like this delivered weekly -- subscribe to Dispatches from the Labyrinth.*