# How Does AI Agent Memory Work Across Sessions, and Why It Keeps Failing > Source: > Published: 2026-09-09 04:38:51+00:00 *AI agent memory isn't a brain, it's three bolted-together tricks: a context window, a vector database, and a summarizer working overtime, and knowing that changes how you build.* - There's no persistent "memory" inside an LLM: every call starts from zero and the model only knows what's stuffed back into its prompt - Letta's MemGPT architecture manages three separate memory tiers, core, recall, and archival, because a single context window can't hold everything - Mem0 stores memories across three backends at once, a vector store, a key-value store, and a graph database, and added graph memory in January 2026 - OpenAI's "dreaming" update from April 2025 runs a background process that rewrites ChatGPT's memory state instead of just appending new facts - Even 1 million token context windows in Claude Opus 5 and GPT-5.5 only perform reliably across 60 to 70 percent of that space before retrieval gets shaky - The fix isn't a bigger context window, it's treating memory as a retrieval and curation problem, not a storage problem Ask a founder why their AI agent forgot a customer's name mid-conversation and you'll usually get a shrug: "the memory glitched." It didn't glitch. There's no memory to glitch. Large language models are stateless. Every single call to the model starts from nothing, and whatever the agent seems to "remember" is just text that got copied back into the prompt before the model ran. That's the honest answer to the question a lot of founders never quite ask: how does AI agent memory work across sessions? It's a copy-paste system dressed up to look like recall, and once you see that, the failures stop looking mysterious. That's not a knock on the technology. It's how it has to work, given what an LLM actually is: a function that takes text in and produces text out, with zero internal state between calls. So every product that claims to "remember you" is running an engineering workaround on top of that constraint, not a genuine memory system in the way a brain has one. There are three workarounds in practice, and they get combined, layered, and mismanaged in ways that explain almost every complaint you've ever had about an AI agent losing the thread. The context window is the text the model can see in a single call, and that's it. Claude Opus 5 and Claude Sonnet 5 both ship with a 1 million token context window, and GPT-5.5 matches that. Sounds huge. It isn't, in practice. Analysis of long-context performance shows models reliably use something like 60 to 70 percent of their advertised window before accuracy on buried details starts to slip, a pattern researchers call "lost in the middle." Put a fact at the start of a 200,000 token conversation and ask about it at the end, and the odds of the model actually using it correctly drop, even though the text is technically still sitting right there in the prompt. You can write a lot on a whiteboard, but cram it full and the reader starts skimming instead of reading, and that's roughly what happens to a model too. That's the first reason your AI agent "forgets" something you told it twenty minutes ago in a long session: it's not gone, it's just buried under everything that came after it, and the model's attention degrades toward the middle. This is a real limitation of the architecture, not a bug someone forgot to patch. It's why nobody serious is trying to solve memory by just making windows bigger forever. Bigger windows cost more, run slower, and don't fix the attention problem. [How Does AI Agent Memory Actually Work, And Why Investors Are Betting Big On It](https://startupfortune.com/how-does-ai-agent-memory-actually-work-and-why-investors-are-betting-big-on-it/) How does AI agent memory work? It runs on a pipeline of embeddings, vector databases, and summarization that lets an agent recall facts outside its context window. That distinction between short-term context and true long-term memory is now the thesis behind a wave of AI infrastructure funding, from Pinecone's vector database roots to newer... - [ai agent memory systems](https://startupfortune.com/how-does-ai-agent-memory-actually-work-and-why-investors-are-betting-big-on-it/) - [persistent memory for ai agents](https://startupfortune.com/how-does-ai-agent-memory-actually-work-and-why-investors-are-betting-big-on-it/) ## Vector databases are the second trick, and they're a guess, not a lookup Because you can't just keep stuffing everything into the context window, the standard workaround is retrieval. The agent converts past conversations, documents, and facts into embeddings, numerical fingerprints of meaning, and stores them in a vector database. When a new session starts, the system searches that database for the embeddings closest in meaning to whatever you just said, and pastes the results back into the prompt before the model responds. A lot of "AI memory" products are really just this trick with a nicer name, and it's worth being blunt about what it is: a similarity guess, not a database lookup. A vector search doesn't know your name is important. It knows a chunk of text mentioning your name scored 0.81 on a cosine similarity check against your latest message, and a different chunk scored 0.79. Most systems only pull back the top handful of matches, so if the fact you actually need scores 0.76 while three irrelevant chunks score higher, it never makes the cut and never enters the prompt at all. That's why an agent can recall your shoe size from three months ago while forgetting you said "I hate PDF exports" last Tuesday. Neither memory is prioritized. Both are just floating in a vector space waiting to be the closest match to something. Mem0, one of the more widely adopted memory layers that bolts onto existing agent frameworks rather than replacing them, doesn't rely on vector search alone anymore. It stores each memory across three separate backends at once, a vector store for semantic search, a key-value store for fast direct lookups, and a graph database for relationships between entities, and it added graph memory as a formal feature in January 2026. That's a tell about where the field has actually moved: pure vector similarity wasn't good enough on its own, so the serious players started layering structured lookups and relationship graphs on top of it rather than trusting embeddings to carry the whole job. ## Summarization is the third trick, and it's where facts quietly die The third workaround is summarization, and it's the sneakiest one because it looks like progress. When a conversation or a memory store gets too large to fit in context, the system asks the model to compress it: turn forty messages into a paragraph, turn a paragraph into a sentence. Do that enough times, across enough sessions, and you get compounding lossy compression. Details that seemed unimportant in the first summarization pass get dropped, and there's no way to recover them later because the raw text is gone. This is exactly the failure mode that shows up in production agents that ran fine for weeks and then quietly went sideways. One widely cited pattern from teams running long-lived support agents: a bot answers tickets correctly for three weeks, then starts responding to billing questions with onboarding instructions and repeats the same canned response four times in a row. Nothing broke. The summarization layer just kept compressing older context until the specific facts that would have prevented the mix-up were gone, replaced by a vaguer and vaguer gist of what the conversation used to be about. OpenAI's own memory system for ChatGPT is a useful real-world case study of these two failure modes colliding, because OpenAI has been public about redesigning it twice. Memory first shipped in April 2024 as "saved memories," a system that only wrote something down when you gave it a strong explicit cue, like telling it you're traveling to Singapore in July. That approach was simple and predictable, but it missed almost everything you didn't explicitly flag. In April 2025, OpenAI added a second layer it calls "chat history," which runs a background process nicknamed "dreaming" that periodically re-reads your past chats and resynthesizes what it thinks it should remember, rather than just appending new saved facts on top of old ones. That's a direct admission that append-only memory doesn't work at scale. If you never revisit and re-curate what's stored, the memory store fills up with outdated, contradictory, or irrelevant facts, and the retrieval step starts surfacing the wrong ones. Letta, the company that grew out of the original MemGPT research paper, took a more structural approach to the same problem. Instead of one flat memory store, Letta's runtime manages three explicit tiers: core memory, a small block that lives directly in the context window the way RAM sits next to a CPU, recall memory, searchable conversation history stored outside the active context like a disk cache, and archival memory, long-term storage the agent has to deliberately query through a tool call, more like cold storage than instant recall. The point of separating these isn't cleverness for its own sake. It's an acknowledgment that not all information deserves equal retrieval priority, and that letting an agent decide when to reach into cold storage produces more reliable behavior than shoving everything into one undifferentiated pile and hoping the retrieval step picks the right five results. [How Do AI Agent SLAs Actually Work, and Why Founders Get Burned](https://startupfortune.com/how-do-ai-agent-slas-actually-work-and-why-founders-get-burned/) How do AI agent SLAs work when the product is non-deterministic? Most enterprise contracts still borrow cloud infrastructure uptime math that never accounts for wrong answers, degraded task completion, or human fallback, and startups are signing away money over it. - [AI agent SLA requirements for enterprises](https://startupfortune.com/how-do-ai-agent-slas-actually-work-and-why-founders-get-burned/) - [why AI founders struggle with SLA](https://startupfortune.com/how-do-ai-agent-slas-actually-work-and-why-founders-get-burned/) ## Long-term memory architecture is a design decision, not a bug to patch Here's the part founders keep getting wrong. They treat memory failures as a bug ticket, something to fix with a patch or a bigger model. It's an architecture decision, made or skipped, at the point they built the product. If you didn't design your AI agent's long-term memory architecture on purpose, you got a default one by accident: whatever your framework's out-of-the-box memory module does, which is usually a sliding window of the last N messages plus a summary bolted on when that window fills up. That default works fine for a five-minute conversation and falls apart the moment a customer comes back after two weeks expecting the agent to remember them. Frankly, most teams find this out the expensive way. A returning customer has to re-explain their problem from scratch, a support handover loses context mid-ticket, a sales agent forgets a prospect's stated budget from the call two weeks earlier. User research on chatbot memory complaints puts the frustration rate at 75 percent among people who've hit an AI system that forgot something they'd already told it. That's most of your users, not a fringe complaint. The practical fix isn't complicated, but it does require treating retrieval and curation as first-class problems rather than an afterthought. Decide explicitly what belongs in fast-access core memory versus what can live in a searchable archive. Don't rely on vector similarity alone, pair it with structured lookups for facts that matter regardless of phrasing, the way Mem0's graph layer does for entity relationships. Re-curate stored memories periodically instead of only ever appending to them, the way OpenAI's dreaming process does, so contradictions and stale facts get cleaned out rather than accumulating forever. And test your agent specifically for degraded recall in the middle of long sessions, since that's where the lost-in-the-middle effect bites hardest and where nobody thinks to check until a customer complains. None of this means agent memory is a solved problem. Vector similarity is still a guess dressed up as recall, summarization still throws details away, and context windows still lose the plot past a certain length no matter how many tokens the marketing page advertises. Strip away the branding and agentic AI memory systems are just context windows, vector retrieval, and summarization, wired together and given a friendlier name than what they are: three separate, lossy workarounds for a model that forgets everything the moment the call ends. **Also read:** [How Does Prompt Caching Work for LLMs, and Why It Cuts Bills in Half](https://startupfortune.com/how-does-prompt-caching-work-for-llms-and-why-it-cuts-bills-in-half/) • [How Does Invoice Factoring Work for Startups With No Collateral](https://startupfortune.com/how-does-invoice-factoring-work-for-startups-with-no-collateral/) • [How Does a Merchant of Record Work, and Why SaaS Founders Are Switching](https://startupfortune.com/how-does-a-merchant-of-record-work-and-why-saas-founders-are-switching/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](/community/) Almost there. Sign in and your reply posts straight away.