# I kept losing context between LLMs, so I built a Markdown-based memory layer (and here is what I learned) > Source: > Published: 2026-07-27 13:24:11+00:00 I kept running into the same problem: after a few weeks away from a project, I'd forgotten why I made certain architecture decisions. Switching between Claude, ChatGPT, and Gemini meant re-explaining everything from scratch. And in long chats, the platform's compaction slowly erased the details I actually cared about. The problem wasn't the tools; it was that my knowledge was trapped inside one chat or one platform, and it didn't stay with me. So I built Nexus: a local, LLM-agnostic memory layer that lives in plain Markdown, not in any one model. I looked at the existing Obsidian MCPs first, but most are tied to a plugin, require Obsidian itself to be running, or are locked to one platform. What I needed was independent of both the model and the storage: Markdown as the single source of truth, everything above it (index, embeddings) rebuildable from that, and working with any LLM. And honestly, I wanted to build the knowledge system myself, make the design decisions myself, and end up with an engineering pipeline that works with me, not against me. Where it has helped me the most is keeping continuity on a project across long breaks. A while ago, I was building a fairly complex project. I worked out the architecture with Claude in a chat, turned it into a roadmap, and had Claude save it into Nexus. From there the loop was simple: Claude planned each milestone, Claude Code implemented it, and Nexus remembered the progress. Then I went on a three-week break. When I came back, I'd forgotten where I left off, but asking Nexus one question brought the whole thing back: what the project was for, which decisions we'd made and why, and exactly which milestone I was on. Control was back in my hands in a minute instead of an hour of re-reading old chats. One design choice that matters here: deleting a note through Nexus doesn't actually delete it; it just marks it status: archived. When something I believed turns out to be wrong, I don't erase the old version, I add the new one and keep the old one marked as superseded, cross-linked to its replacement. The path I took to a conclusion is as valuable to me as the conclusion itself. Months later, when I'm debugging or chasing an idea, the model can see "you already tried this, here's why it didn't work", saving me an afternoon of rediscovering why that idea failed. I don't really think of Nexus as just a RAG system. Retrieval is just one piece of it. It's a local stdio MCP server, meaning it works with any MCP host (Claude Desktop, Cursor, VS Code, Cline) driving any tool-capable model. The Markdown files are the only source of truth. Search is hybrid: a small local embedding model for meaning (multilingual-e5-small), plus keyword/BM25 for exact terms (like API names or error codes) that embeddings tend to blur. Embeddings run locally, so after a one-time ~3 min model download, it works fully offline. Every change is a git commit, making the vault's history a readable log. And to answer the common question: Why not just vector search? Because "these two notes are similar" isn't the same as "these are the same note." Similarity is a hint, not a decision. Nexus never auto-merges on similarity; it returns candidates, and the LLM (on my behalf) makes the call. Auto-merging on closeness is exactly how a knowledge base quietly corrupts itself. Building the core engine was one thing, but actually using it every day exposed me to the realities of the MCP ecosystem and LLM behavior. Here are the hard-won engineering lessons I'm carrying forward to every future AI project. One of the most valuable lessons came from a bug that wasn't actually in Nexus. I wanted the model to receive a vault manifest at the start of every session using MCP's standard initialize.instructions field. I implemented it, wrote tests, and everything passed perfectly. But in real usage with Claude Desktop, the model behaved as if the manifest didn't exist. The mistake wasn't in the implementation—it was in what I was testing. I had verified that the server produced the well-formed payload, not that the host actually delivered it to the model. Claude Desktop was silently discarding it. The lesson: Delivery tests must always verify what reaches the end consumer (the model), not just what your component emits. Knowing that Claude Desktop ignored initialize.instructions, I had a choice: I could hack around it by embedding the entire ~900-token vault manifest inside the tool descriptions. Technically possible? Yes. Architecturally right? No. It would permanently bloat every tool description with hundreds of tokens every session and mix vault data into what should be a static tool contract. I chose to keep the standard implementation. If hosts begin supporting it in the future, Nexus will automatically benefit. Good engineering is knowing when to reject a feasible workaround because it violates your design principles. While improving search results, I realized raw cosine similarity is incredibly easy to misread. Naively, I interpreted a number like 0.77 as "77% relevant." But models like multilingual-e5-small don't behave that way. Even completely unrelated text often scores a floor of ~0.70, while genuinely relevant matches usually begin around 0.85+. A number without context is more misleading than no number at all. A raw cosine score must ship with a calibration note tied specifically to that embedding model. The biggest UX improvements didn't come from a roadmap; they came from watching how Claude naturally interacted with Nexus. I noticed Claude asking a question involving four notes, which required five separate tool calls. Solution: Implement a Batch Read tool to cut latency. The verified field existed but had no practical meaning. Solution: Either document its semantics or remove it from the surface. Every moment where the interaction felt inefficient for the LLM became a product improvement. The project slowly shifted from "building functionality" to "improving the AI's experience of using it." It's early and built for one person, so I've tried to be honest about the rough edges: upsert rewrites and re-embeds the whole note instead of diffing. Fine for my note sizes, wasteful if they get big. Backlinks are computed by scanning the vault on every read O(n). Simple and correct, but needs a real index if the vault scales. The chunk-id anchors () are invisible in reading mode but show up in Obsidian's edit view. That's the whole idea: your knowledge shouldn't disappear because you switched models or closed a chat, and the systems managing it shouldn't be black boxes. If this sounds like an engineering philosophy you agree with, you can check out the project here: [github.com/etkaozer/nexus](https://github.com/etkaozer/nexus)