{"slug": "agent-memory-as-a-file-format", "title": "Agent Memory as a File Format", "summary": "Simon Willison's blog post proposes a new approach to AI agent memory, suggesting it should be treated as a portable file format called 'memoryfield' rather than a complex pipeline. The format consists of Markdown pages with optional YAML frontmatter and a SQLite vector index, arguing that agents work best with files and that prose-based memories are simpler and more effective than existing systems that tie users to specific harnesses or require complex infrastructure.", "body_md": "# Agent memory as a file format\n\nMemoryfields - a vastly simpler way to do agent memory\n\nMany model benchmarks [start from a blank context\nwindow](https://simonwillison.net/tags/pelican-riding-a-bicycle/). The tabula\nrasa of AI. To some extent, this makes sense, to keep the benchmarks fair.\n\nBut real agents should never start from a blank context window. They should\nstart with as much relevant information available to the agent as possible.\nYour AI agents should start with **memories**.\n\n## Why existing agent memory systems don't seem to work\n\nThe trouble is, a lot of agent memory systems are actually pretty rubbish. I think there are roughly three popular kinds of memory system at the moment, each of them not working in their own way.\n\nThe first are ones that deliberately tie you into a specific harness - usually written by the lab that rents you that harness. Said lab desperately wants to transition out of the (highly competitive) \"API business\" and into the (much more lucrative) \"platform business\". This form of system usually works by mining information out of your conversation history, with the result that most of their memories are all about you, even though information about the world is generally much more useful.\n\nAnother kind is ludicrously complicated. I know of one prominent system that needs pgvector, a Neo4j graph database and an LLM of its own just to decide what's worth remembering. This complexity is not only difficult to administer, but, for reasons I will explain: these Big Systems bamboozle the models as well. They also fail to scale with the model frontier as it moves forward.\n\nThe final kind is the \"High Modernist\" variety, which imagine an idealised, rationalist form of memory. Inevitably, this involves a graph, and sometimes logical propositions as well. This kind systematically strips information from its context and leaves it isolated and senseless to the agent (and you). How useful, after all, is a simple list of \"distilled facts\"?\n\nWhat they have in common is that they treat memory as a process. But memory - especially to a model - is much better represented as data.\n\n## Memory should be a data format, not a multi-stage pipeline\n\n[Brooks](https://martinfowler.com/bliki/MythicalManMonth.html) said:\n\nShow me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.\n\nSo, here is the \"memoryfield\" portable memory file format:\n\n```\nmy-memories.memoryfield.zip\n├── carbon-fibre-woks.md\n├── finnish-bureaucracy-tips.md\n├── [... many more md files...]\n├── wec-2026-season-notes.md\n└── nomic-embed-text-v1.5.sqlite3\n```\n\nA memoryfield is:\n\n- Markdown \"pages\", with\n- (optional) YAML frontmatter and\n- (optional) SQLite vector index for semantic search\n\nAgents work best with files. Allow me to explain.\n\n## Design decision 1: use prose, not chunks or \"facts\"\n\nThe main reason why RAG pipelines can be very complicated is that they are trying to make a mass of existing, human-authored documents legible to an AI agent. Often these documents are very hard for the agent to read directly, eg: because they are big PDFs.\n\nBut agent memories are not complicated legacy documents. A memory, at the time it is being formed, is occurring directly to an AI agent which is fully able to write prose. That prose does not need to be chunked, enriched, double-summarised or otherwise mechanically processed: just have the agent write the memory directly in its favourite format (which is Markdown).\n\nA memoryfield page looks like this:\n\n```\n---\ntitle: Carbon Fibre Woks\ncreated: '2026-03-01T09:00:00Z'\nupdated: '2026-08-22T14:30:00Z'\nuuid: 6aa615f0-486f-48a7-a210-ba4f5ff18c8b\nsummary: Thermal properties of carbon fibre cookware\n---\n\nCarbon fibre woks conduct heat evenly, but...\n```\n\nThe one limitation, admittedly, is that the page has to be short enough to fit into a vector embedding: so there is a soft limit of about 8kb (~2000 tokens).\n\nBut this is a highly beneficial restriction in practice: 8,000 characters is about 1,300 words, or the length of a medium-length magazine article. That is, in fact, a restriction it would make sense to impose anyway. To add more detail, add more pages - agents do not struggle to do this.\n\n## Design decision 2: semantic jump, not graph walking\n\nA key piece of prior art was [Karpathy\nwikis](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).\nKarpathy wikis are oriented around hyperlinked Markdown files: modelled on\nthose used by Roam or Obsidian. The idea was that the agent would walk the\n\"knowledge graph\" to find relevant pages.\n\nBut in practice, having an AI agent traverse a knowledge graph is slow and unreliable - as well as being confusing for the agent.\n\nTraversal is slow because the model needs to frequently stop to make serial tool calls to read successive pages.\n\nThe rough algorithm for an agent to walk a knowledge graph:\n\n- Read wiki front page [tool call]\n- find relevant links\n\n- Read linked page(s) [tool call]\n- find relevant links\n\n- Decide if enough relevant information has been found\n- If not, go to #2\n\nIf the relevant information is N steps deep in the knowledge graph, N+1 tool calls are required to retrieve it. This is slow, as your billion (trillion?) dollar LLM model has to pause for each tool call, each of which takes maybe 2-3 seconds. It also heavily penalises deeply nested knowledge graphs, which frankly cuts across the whole point of them.\n\nKnowledge graphs are also unreliable. Because the AI can only tell if the material is relevant by looking at the link text, or maybe page title, if that is externalised somehow. That puts great pressure on the agent to do 1990s-SEO-style page metadata hacking to ensure that the link text/title/caption of each page is snappy and accurate. Doing so punishes digression, the ambient noting of side details and the kind of implicit lore that is both common and highly useful in larger text corpuses.\n\nIn practice, relevant information is often missed in Karpathy wikis because it is not titled or captioned in a way which looks appealing enough to the searching agent.\n\nAnd knowledge graphs are also confusing to the agent because they often have to pore over a lot of irrelevant information as they walk around the graph. Inadvertently reading irrelevant information (the frontpage is often the main offender) puts a bunch of noise into the model's context window, which lowers the quality of their output and makes them look fixated on weird stuff.\n\nThis is all solved by using semantic search to just jump directly to **all**\nthe relevant pages (based on their actual content, not their page metadata) and\nhaving the agent read all relevant pages, at once, *in parallel* - which the vast\nmajority of them will do now. So in a memoryfield, at most 2 tool calls are\nrequired (#1 to search, #2 to read in parallel). Relevant stuff actually gets\nfound and irrelevant input tokens are minimised.\n\n## Design decision 3: More model, less mechanism\n\nOne of the issues posed by \"high mechanism\" memory systems - the kind that\ninclude a lot of specially crafted APIs or databases - is that to use them,\nagents must navigate an interface maze to achieve their goal. If the interface\nis large, then you're loading a lot of `openapi.json`\n\ninto the context. If\nthe interface is small, then it is limiting. Even if the balance is right,\noften the API is still wrong: recall the times when you had to use an API\nwritten by someone else who hadn't foreseen your needs. Did you enjoy that\nexperience?\n\nMemoryfields then, being a \"low mechanism\" system (just a file format), gives\nagents much greater latitude to invent their own access patterns. While some\n[(hopefully) helpful tooling](https://github.com/calpaterson/memoryfield-tool/)\nis provided, agents are fully free to use whatever access patterns they like.\nFor example using `perl`\n\nto do find-and-replaces across the whole corpus, or\nputting inline CSV files inside memories that they then query with SQLite\n(both real examples I have personally seen).\n\nBeing \"low mechanism\" also means that memoryfields scale with the model frontier. As models get better, agents think of more stuff to do. One of the recentish breakthroughs is that the models are accidentally very good at bash. They are good at Markdown too. And SQLite. One of the reasons that I think memoryfields work well inside real agents is that agents fundamentally can \"get\" what is going on from their training data (which is all you have until you can read your memories) in a way that as a disembodied LLM call within a \"memory pipeline\" they cannot.\n\nAs models get better, they automatically start to write memories a bit more cleverly. The memory systems of the \"bag-on-the-side\" rarely do this. There are only so many ways to more imaginatively use a fixed set of API endpoints. Memoryfields will scale with the model frontier.\n\n## Design decision 4: Open format, interchangeable, transport invariant\n\nAs your collection of memories builds, they start to become precious. Your built up treasure of learned lessons and hard-won established facts. You don't want to be locked in to a specific harness, model or agent.\n\nI've written an [RFC-style spec for the file\nformat](https://github.com/calpaterson/memoryfield-spec/blob/main/SPEC.md) -\nmainly to remove ambiguities and avoid tying it to a specific embedding\nfunction.\n\nIf you want, you can surely vibe code whatever tooling you need from the spec\nalone. But I also provide a\n[skill](https://github.com/calpaterson/memoryfield-skill) and an\n[agent-optimised command line\ntool](https://github.com/calpaterson/memoryfield-tool) to go with it.\n\nThe canonical \"archival\" format of a memoryfield is as a zipfile. That's to make data exchange as easy as possible. But I've deliberately left the spec open to being served from local files, Amazon S3, on GitHub or over HTTP. In fact, anything that has files works. I personally use a mixture of these transports: Syncthing for personal memoryfields, S3 for those I share with others.\n\n## Getting started\n\nYou could have your agent pull down\n[SPEC.md](https://github.com/calpaterson/memoryfield-spec/blob/main/SPEC.md)\nand vibe an implementation, but probably the simplest way to get started is\nto use my tooling:\n\n```\n# Requires: ollama, uv and npx (comes with npm)\n#\n# 1. Pull the embedding model:\nollama pull nomic-embed-text\n# 2. Install the CLI tool:\nuv tool install git+https://github.com/calpaterson/memoryfield-tool\n# 3. Install the skill:\nnpx skills add calpaterson/memoryfield-skill -g -y\n```\n\nYour agent should help you get up and running from here.\n\nIf you want a demo memoryfield to try out, try\n[ soapstones.memoryfield.zip](https://blobs.calpaterson.com/soapstones.memoryfield.zip).\nSoapstones was an earlier project of mine on agent memories and this curated\nexport contains a lot of high-value-to-weight memories on how agents can get\naccess to data (like how to search Reddit as an agent, how to use Jina Reader,\nhow to use the MediaWiki API to read wikis effectively).\n\n## \"Isn't this just some RAG\" - and other common objections\n\nIsn't this just some RAG?\n\n\"RAG\", as it stands, is now interpreted incredibly broadly - as soon as any agent retrieves data, 'RAG has happened'. In that sense: yes, this is some RAG.\n\nBut: almost all agents retrieve data. For example by searching the web. And most of the techniques that are usually associated with a \"RAG system\" are not present here. There is no chunking, there is no re-ranking, there is no hybrid search.\n\nThe other side of it of course is that it's the agents that write the memories. RAG systems are often about reads, but memoryfields are for writing too.\n\nIsn't\n\n`nomic-embed-text-v1.5`\n\nover 2 years old? Aren't there newer and better models?\n\nEmbedding models are neither as large as frontier models, nor as fast moving.\n[ nomic-embed-text-v1.5](https://ollama.com/library/nomic-embed-text) remains a\ngood balance between small and powerful. It is small enough (270MB)\nand fast enough to run on non-GPU hardware, and is a widely popular and frequently\nrecommended default embedding model.\n\nThe spec, though, allows for some other embedding to be used.\n\nHow can I judge what is a good memory to store? How can I avoid filling my memory with crap?\n\nThis is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search. Irrelevant memories take up space, yes, and perhaps you want to periodically have a clean out, but they don't hamper an agent in any way.\n\nFor best results: insert liberally into the memoryfield. The one tip I would give, though: memories work best when they include citations, ideally in the form of URLs. That helps future passes over memories to strengthen them and helps agents fact check outdated or otherwise suspect material.\n\nWhat about security? What about\n\n[\"Disregard that!\"?]\n\n**You must not share your context window, including via memories, with parties you don't trust.**\n\nOne of the reasons the spec includes a static zipfile format is to allow you to\nmanually review and pin (via `sha256sum`\n\n) memoryfields you get from others.\n\nThere remains [no way to have an agent distinguish \"good prompt\" from \"evil\nprompt\"](https://calpaterson.com/disregard.html).\n\n## Data first\n\nNow that the flowchart is obvious I might as well state it explicitly:\n\n- Write a memory as Markdown\n- Embed it and save the vector to SQLite\n- Search semantically to find memories again\n\nMemoryfields are unusual as a memory system in that they specify a data structure and not a process. There's no extraction pipeline, no background processing services, no pluggable - well, anything. There is a vector index, but it's a deletable cache, not the system.\n\nMemory is data! The less fixed machinery we put between the agent and that data, the better the agent can be.\n\n## Contact/etc\n\n## Notes\n\nIf you have time, please take a look at the\n[spec](https://github.com/calpaterson/memoryfield-spec/blob/main/SPEC.md). Any\n(human) review of that is highly valued.\n\nMy install procedure includes, by my count, four different package managers (Ollama, uv, NPM, Vercel Skills). It does feel like there must be a better way. Answers on a postcard to the usual address.", "url": "https://wpnews.pro/news/agent-memory-as-a-file-format", "canonical_source": "https://calpaterson.com/memoryfields.html", "published_at": "2026-08-31 11:17:25+00:00", "updated_at": "2026-08-31 11:53:25.267891+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-research"], "entities": ["Simon Willison", "memoryfield", "Markdown", "SQLite", "pgvector", "Neo4j", "nomic-embed-text-v1.5"], "alternates": {"html": "https://wpnews.pro/news/agent-memory-as-a-file-format", "markdown": "https://wpnews.pro/news/agent-memory-as-a-file-format.md", "text": "https://wpnews.pro/news/agent-memory-as-a-file-format.txt", "jsonld": "https://wpnews.pro/news/agent-memory-as-a-file-format.jsonld"}}