{"slug": "ai-agents-dont-need-more-context-they-need-memory", "title": "AI Agents Don’t Need More Context. They Need Memory.", "summary": "A developer is building BaseMyAI, a local-first memory infrastructure layer for AI agents, arguing that current systems rely on retrieval rather than true memory. The project features temporal memory to track how facts change over time and agent isolation, with a native storage engine in Rust.", "body_md": "We keep making AI models better at reasoning.\n\nWe give them larger context windows.\n\nWe connect them to tools.\n\nWe let them search files, browse repositories, call APIs, execute code, and operate increasingly complex workflows.\n\nAnd yet one problem keeps showing up:\n\n**the agent forgets.**\n\nNot necessarily because the model is bad.\n\nBecause most agent systems still treat memory as an afterthought.\n\nI’ve been thinking about this problem for a while, and it eventually led me to start building **BaseMyAI**: a local-first memory infrastructure layer for AI agents.\n\nThis is the first post where I want to document what I’m building, why I think this problem matters, and what I’m learning along the way.\n\nA common approach to agent memory looks roughly like this:\n\nThis is useful.\n\nBut I don't think it is memory.\n\nIt is retrieval.\n\nA real memory system has to answer harder questions.\n\nWhat does the agent currently believe?\n\nWhat information is outdated?\n\nWhich fact replaced another fact?\n\nWhich memories belong to this agent?\n\nWhich memories are temporary?\n\nWhich ones must survive for months?\n\nWhat happened before a certain decision?\n\nWhat information is actually relevant to the current task?\n\nAnd just as importantly:\n\n**what should be forgotten?**\n\nOnce agents start operating for days, weeks, or months, these questions become much more important than simply finding the nearest embedding.\n\nLong context windows are incredible.\n\nBut throwing everything into the prompt doesn't scale particularly well.\n\nImagine an engineering agent that has worked on the same codebase for six months.\n\nDuring that time it has seen:\n\nTechnically, you could keep feeding more information back into the model.\n\nBut eventually you're paying for a huge amount of irrelevant context.\n\nAnd worse: old information can conflict with new information.\n\nThe problem becomes less about:\n\n“How much context can the model read?”\n\nand more about:\n\n“What is the smallest amount of correct context the model needs right now?”\n\nThat is a memory problem.\n\nOne concept I find particularly important is **temporal memory**.\n\nConsider these two facts:\n\nDatabase: PostgreSQL\n\nDatabase: native embedded engine\n\nA basic retrieval system might return both.\n\nBut they're not necessarily contradictory.\n\nMaybe PostgreSQL was used three months ago and the project later migrated to a native engine.\n\nThe missing dimension is time.\n\nThe system should understand something closer to:\n\n2026-04\n\nDatabase = PostgreSQL\n\n2026-07\n\nDatabase = native embedded engine\n\nNow an agent can reason about the evolution of the project instead of treating every stored fact as equally current.\n\nThat distinction becomes extremely important in long-running software projects.\n\nAnother problem appears when multiple agents are involved.\n\nImagine:\n\ncoding-agent\n\nresearch-agent\n\nsupport-agent\n\nmarketing-agent\n\nThey may share some knowledge.\n\nBut they should not automatically share everything.\n\nAn agent's memory needs an identity and a boundary.\n\nThis raises interesting architecture questions around:\n\nFor BaseMyAI, agent isolation is one of the fundamental primitives rather than something added later.\n\nThere is another requirement I care about: memory should be able to live close to the user.\n\nAgent memory can contain some of the most sensitive information on a machine:\n\nsource code, conversations, documents, product strategy, credentials metadata, personal preferences, and months of accumulated context.\n\nSending all of that to another hosted database should not be the only architecture available.\n\nSo BaseMyAI is being designed around a **local-first and encrypted** model.\n\nThat decision makes the engineering considerably more interesting.\n\nI'm currently building a native storage engine in Rust with things like persistent indexes, bounded memory management, WAL durability, snapshots, compaction, and concurrency controls.\n\nThe goal isn't to build infrastructure for the sake of infrastructure.\n\nThe goal is to make long-term agent memory predictable enough that developers can actually trust it.\n\nNone of this means vector search is bad.\n\nVector search is extremely useful.\n\nBaseMyAI itself uses vector retrieval as one part of memory.\n\nThe distinction I'm making is architectural:\n\n```\nVector search\n      ↓\nis a component of\n      ↓\nAgent memory\n```\n\nrather than:\n\n```\nVector database = Agent memory\n```\n\nMemory also needs structure, lifecycle, chronology, identity, durability, and context selection.\n\nThat's the layer I'm interested in.\n\nMy current mental model looks something like this:\n\n```\n                    ┌─────────────────┐\n                    │    AI Agent     │\n                    └────────┬────────┘\n                             │\n                             ▼\n                    ┌─────────────────┐\n                    │ Context Compiler│\n                    └────────┬────────┘\n                             │\n             ┌───────────────┼───────────────┐\n             ▼               ▼               ▼\n        Semantic          Temporal        Structured\n         Recall            Memory          Relations\n             │               │               │\n             └───────────────┼───────────────┘\n                             ▼\n                    ┌─────────────────┐\n                    │ Durable Memory  │\n                    └─────────────────┘\n```\n\nThe important component here might actually be the **context compiler**.\n\nThe storage engine can know millions of things.\n\nThe model shouldn't receive millions of things.\n\nThe context compiler's job is to transform long-term memory into a small, relevant, current representation for a particular request.\n\nI'm increasingly convinced that this layer will be critical for serious autonomous agents.\n\nBaseMyAI is still being built.\n\nA lot of the work right now is deep infrastructure work rather than polished product work.\n\nRust.\n\nStorage engines.\n\nMemory accounting.\n\nConcurrency.\n\nIndexes.\n\nDurability.\n\nRetrieval.\n\nTemporal semantics.\n\nAnd probably many design decisions I'll discover were wrong six months from now.\n\nThat's exactly why I want to write about it here.\n\nInstead of only publishing BaseMyAI once everything looks finished, I want to document the engineering decisions, experiments, failures, benchmarks, and architectural questions as they happen.\n\nSome topics I want to explore next include agent memory models, temporal retrieval, designing a storage engine in Rust, context compilation, memory isolation between agents, and why BaseMyAI is deliberately not designed as another vector database.\n\nIf you're working on agents, retrieval systems, Rust infrastructure, knowledge graphs, or long-term AI memory, I'd genuinely like to compare approaches.\n\nThis field still feels very early.\n\nAnd I think we're only beginning to understand what **memory for software agents** should actually look like.", "url": "https://wpnews.pro/news/ai-agents-dont-need-more-context-they-need-memory", "canonical_source": "https://dev.to/mikeross27/ai-agents-dont-need-more-context-they-need-memory-470o", "published_at": "2026-08-17 23:11:37+00:00", "updated_at": "2026-08-17 23:42:57.198551+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["BaseMyAI", "PostgreSQL", "Rust"], "alternates": {"html": "https://wpnews.pro/news/ai-agents-dont-need-more-context-they-need-memory", "markdown": "https://wpnews.pro/news/ai-agents-dont-need-more-context-they-need-memory.md", "text": "https://wpnews.pro/news/ai-agents-dont-need-more-context-they-need-memory.txt", "jsonld": "https://wpnews.pro/news/ai-agents-dont-need-more-context-they-need-memory.jsonld"}}