{"slug": "building-personal-memory-store", "title": "Building Personal Memory Store", "summary": "Oleh built a self-managed, decentralized personal memory store for AI agents using the open-source Graphiti framework and FalkorDB, storing context as entity-relationship graphs alongside a SQLite database for todos and reminders. The store is exposed to agents through an MCP server and to humans via a REST API, with an LLM classifier deciding whether each stored \"Episode\" goes to the vector database, SQLite, or both. The author argues that usefulness equals intelligence plus context, and that owning personal context avoids lock-in to any single model provider.", "body_md": "# Building personal memory store\n\nSep 2026 · 8 min read\n\nWhen I played with Grok Bot, I was hooked by how proactive and useful it could be. It's an impressive product, both in its capabilities and its UI. But it didn't know me, my life context, or what I care about right now. It was cool, but I'd never call it a personal butler that can help me with all my daily chores and projects.\n\nIntelligence is only one part of what makes AI agents useful. The other part is context. **Usefulness = intelligence + context.**\n\nWe are fortunate to have many intelligence providers, with models getting more powerful and more affordable every few weeks. And this is precisely why it makes little sense to keep all my personal context with any one model provider. I want to be able to switch between models without being locked into any of them, and own my personal context and have full control over it.\n\nWhen I started designing my little village of AI agents, one of the first decisions I made was to build a decentralised memory store that I could manage myself and connect to any third-party model or agent. It's been a fun journey, and in this post I'll describe how I built it.\n\nThis is Part 1 of my Personal Little AI World setup series.\n\n## Architecture\n\nI use open-source Graphiti framework and FalkorDB to orchestrate my memory store. Graphiti parses information and stores context as relationships between entities: \"Entity → Relationship → Entity\".\n\nWith more queries the graph grows and develops more complex relationships like \"Oleh friends with James who works at XYZ competitor of Acme\".\n\nThe good news is that Graphiti abstracts away most of the complexity for you - entity deduplication, query parsing, conflict management, etc. The framework has been great at hiding the complexity while still being developer-friendly enough for me to experiment with custom entity types and search strategies.\n\nGraphiti supports several graph databases out of the box. I went with FalkorDB because it's the simplest option and works well at my small, personal scale.\n\nAlongside context, I want my memory store to store my todos and reminders. Vector DBs are a bad choice for storing tabular data, as similarity search can't precisely answer questions like \"what's due before Friday?\". To store tabular data, I added a SQLite db and a thin piece of logic that parses free-text todo queries into SQL rows.\n\nI expose memory store data via an MCP server for agents and a REST API for analytics for humans.\n\nWhen an agent decides to add a new memory, it calls `add_memory` with the new context as a function parameter. I always store an input query (Graphiti uses the \"Episode\" term for raw text) in the Episode store. This means I always have a source of truth for everything further down the pipeline. Then I run an LLM classifier call to decide whether the Episode should be added to the Vector DB, stored inside SQLite, or both.\n\nFor search, I expose a `search_memories` function that performs similarity search across my context and returns a list of relevant facts, entities, and source-of-truth Episodes. For Todos, I decided to simply attach a list of active TODOs to the response. It's not the most elegant solution, but it works really well at my scale and keeps things simple.\n\n```\n<FACTS>\n{to_prompt_json(fact_json)}\n</FACTS>\n<ENTITIES>\n{to_prompt_json(entity_json)}\n</ENTITIES>\n<EPISODES>\n{to_prompt_json(episode_json)}\n</EPISODES>\n<Todos>\n{to_prompt_json(active_todos)}\n</Todos>\n```\n\n## Markdown first or graph first?\n\nThere are two schools of thought when it comes to storing personal context.\n\n**Markdown first** (e.g. gbrain): keep the source of truth as Markdown files and give agents access to a filesystem, with (optional) graphs built on top.\n\nThe biggest benefit is that you can easily edit files, read them on demand, and store text artefacts (e.g. emails) in their original form. You can always go back to the source of truth.\n\nThe downside is that you don't get the transactional guarantees of a database, you or your agent need to manage a library of files, and as it grows, you also need an indexing/search layer rather than scanning Markdown files directly.\n\n**Graph first**: store everything as a graph, linked back to the parent Episode, and rely on a combination of vector, keyword, and graph search to retrieve relevant context. You can also generate Markdown files from the graph on demand.\n\nThe downside is that a graph is harder for a human to read and edit directly. Once you turn an email or conversation into entities and relationships, you lose some of the original context. You might know that \"Oleh works at Acme\", but the context behind that relationship is stored in the parent Episode. You have to explicitly trace the relationship back to the Episode to understand where that fact came from.\n\nI don't think one approach is obviously better for a *personal* context store. My decision to build graph first system came down to two things: 1) Graphiti works really well with this model and 2) I'm very unlikely to successfully maintain a Markdown-based memory system, because... I'm lazy.\n\n## Conflict management\n\nWhat happens if two contradictory facts come into your memory store? Graphiti uses timestamps to figure out which fact is newer and treats the newer fact as replacing the old one. So if last season you supported Arsenal and now you're a big City fan, the newer fact will supersede the old one.\n\nIn my system, it's the user's (aka my) responsibility to resolve contradictions. The system will highlight when two facts can't coexist, and it's up to the user to decide which colours they want to support this season.\n\n## Deployments\n\nI deploy the memory store and my fleet of agents to [exe.dev](https://exe.dev/). The exe model is simple, yet unusual: you pay a monthly fee for a pool of CPU and memory, then split it across as many VMs as you like.\n\nThis lets me spin up agents and memory stores in dedicated, isolated VMs at no extra cost! Exe also gives me super convenient abstractions around authentication and integrations, making the platform perfect for my little AI world. Exe is just such a great example of elegant ergonomics and dev-friendly design, so this post will include several love notes to the platform. (unfortunately, this post is not sponsored by exe)\n\n## Backups\n\nDon't forget to back up your memory store! In my case, it's a simple cron job that pushes a db dump to a 3rd party storage provider. Do whatever makes most sense for you.\n\n## Connect Claude to my personal memory store\n\nFirst ode of love to exe.dev! My memory store is a VM running on the exe platform. Each VM is private by default, but conveniently exposes a URL. For my memory store, it's [memory-store.exe.xyz](https://memory-store.exe.xyz) (unless you're logged into my account, you'll see a 307 redirect).\n\nExe also allows me to generate a bearer token and securely send data to the VM. It also manages port forwarding between the public URL and the FastAPI MCP server running on the VM.\n\nAdd a new Connector to Claude and add an `Authorization` header with your token.\n\nClaude can now add and search my memories!\n\n**Pro tip**: from my experience, Claude sometimes needs an extra nudge to actually use these tools. Anthropic allows you to add instructions for every conversation. I'm still figuring out the best wording.\n\n## Connecting memory store to my fleet of Agents\n\nSecond ode of love to exe.dev! I'll talk about my agent fleet in more detail in Part 2 of this series, but as TLDR, each Agent runs in a separate VM on the exe platform. I can tag each VM and then create rules like \"allow each VM tagged `agent` to access the memory store\".\n\nThis is incredibly sleek! I don't have to worry about auth, don't store any keys inside the VM (with a running Agent inside it!) and each new VM with tag `agent` talks to the memory store out of the box! How cool is that!\n\n## Is it useful?\n\nMillion dollar question: does having a personal (sovereign!) memory layer and connecting it to Claude, Pebble smart ring and a fleet of agents make my life easier, more productive and/or more enjoyable?\n\nYes, sort of, maybe, it depends! Voice notes over pebble smart ring (more in part 2) to save todos works really well, especially if I add a deadline for it \"Remind me to do x on Sunday\".\n\nSome of the facts derived from my interactions with agents and stored in my memory store are genuinely useful and correct. But I wonder: will they ever actually be useful in the future?\n\nAnd some of the facts, especially those derived from my voice, are completely nuts.\n\nI don't know who Jeff is, and I've never had coffee with Sheetal :(\n\nMy hope and the promise of personal memory overall is that, over time, it builds up a critical mass of knowledge that becomes invaluable to an agent trying to resolve any personal matter.\n\nI can already see glimpses of it, but I currently still need to babysit extraction and remove \"Jeff had coffee with Sheetal\" nonsense facts. I reckon in a few months it will be clear if this whole setup actually worth it. And whether it works or doesn’t, that could make for a pretty good story to tell in Part 3 of this series.\n\n## Teaser for Part 2\n\n- Deploying Hermes agents with custom plugins\n- How I use my smart ring [repebble.com](https://repebble.com/index)\n- Building my own Grok Bot alternative with Rust, TypeScript and Tauri\n- More love to [exe.dev](https://exe.dev/)", "url": "https://wpnews.pro/news/building-personal-memory-store", "canonical_source": "https://www.oleh.cafe/writing/building-personal-memory-store", "published_at": "2026-09-15 09:50:48+00:00", "updated_at": "2026-09-15 10:09:47.919889+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "large-language-models", "developer-tools"], "entities": ["Graphiti", "FalkorDB", "SQLite", "Grok Bot", "MCP", "Oleh"], "alternates": {"html": "https://wpnews.pro/news/building-personal-memory-store", "markdown": "https://wpnews.pro/news/building-personal-memory-store.md", "text": "https://wpnews.pro/news/building-personal-memory-store.txt", "jsonld": "https://wpnews.pro/news/building-personal-memory-store.jsonld"}}