Every call you make to a large language model is stateless. The model reads the text you send, generates a reply, and forgets everything the moment it finishes. That is fine for a one-off question and a disaster for an agent - a system that takes multi-step actions, learns from feedback, and picks up work across sessions. AI agent memory architecture is the layer you build on top of that stateless model to give it continuity: a way to remember what happened, what’s true, and how to act.
I wrote a full guide on The Ultimate Guide to LLM Memory that covered the context window itself. This piece is the sequel - moving from “how a model holds a conversation” to “how you architect memory that survives across sessions.” By the end you’ll have runnable, framework-agnostic Python for each memory type and a decision framework for which ones you actually need. No vendor lock-in, no hand-waving.
An AI agent memory architecture is the two-layer system - working memory held in the context window plus long-term memory stored externally - that lets a stateless model persist and recall information across turns and sessions. Working memory holds the active task; long-term memory holds durable facts, past events, and learned behaviors that outlive any single conversation.
A language model has no internal store that updates between calls. The Cognitive Architectures for Language Agents (CoALA) framework, a peer-reviewed taxonomy from Sumers, Yao, Narasimhan, and Griffiths, formalizes the split most systems now use: information storage divided into working memory (the immediate context) and long-term memory(persistent, external stores the agent reads from and writes to).
The practical consequence is that memory is not one thing. It is a pipeline that moves information between a small, fast, in-context layer and larger, slower, external layers - a design MemGPT explicitly borrowed from operating systems, treating the context window like RAM and external storage like disk.
The model receives a block of text, processes it, returns a response, and retains nothing. There is no hidden state carried forward. When ChatGPT “remembers” what you said three turns ago, it is because your application resent those turns inside the prompt - not because the model stored them.
Working memory lives inside the context window and is rebuilt on every call. Long-term memory lives outside the model in a database, vector store, or file, and is queried when relevant. The entire craft of agent memory is deciding what to promote from working into long-term, what to retrieve back, and what to drop.
Agent memory is usually split into four types: working memory (the active context window), episodic memory (records of past interactions), semantic memory (durable facts and preferences), and procedural memory (learned skills and workflows). Working memory is short-term; the other three are long-term stores that persist across sessions.
This four-part split isn’t a marketing invention. The episodic-versus-semantic distinction comes from psychologist Endel Tulving’s 1972 work on human memory, and CoALA adapts it directly for language agents. MachineLearningMastery’s breakdown and IBM’s overview both land on the same categories.
The context window itself - recent turns, the current task state, whatever the user just clarified. Think of it as RAM: fast, limited, and wiped when the session ends. Everything else in this article exists to feed the right information into this layer at the right time.
Records of specific past interactions, tied to a time and context: “On July 2, the user booked a trip to London and asked for city-center hotels.” Episodic memory is what lets an agent say “last week you told me…” It is typically stored as a timestamped log or as indexed conversation chunks.
Durable facts, preferences, and constraints that hold across time: “prefers Python over JavaScript,” “budget cap is $50K,” “allergic to peanuts.” The key operation here is updating rather than appending. When “budget cap $50K” becomes “$75K,” you overwrite the entry - you do not store two contradictory facts and hope retrieval picks the right one.
Learned skills, tool-use patterns, and workflows - the agent’s repertoire of effective actions. A coding assistant that has learned your project’s test-then-refactor loop is using procedural memory. In practice it often lives as prompt templates, tool policies, or code rather than as retrieved text.
Retrieval-augmented generation (RAG) is read-only retrieval from a shared, static corpus; every user draws from the same documents and nothing changes based on the interaction. Agent memory readsandwrites: it is stateful, user-scoped, and updates as the agent learns. A larger context window delays the memory problem but does not solve it.
This is the single most confused distinction in the space, so it’s worth being precise. RAG, introduced by Lewis et al. in 2020, fetches relevant chunks from a document index and injects them into the prompt. It is stateless and query-matched. Agent memory is a two-phase system: a write phase that persists information about the user or task over time, and a read phase that retrieves it later. Mem0’s own writeup on the difference frames it well - RAG reads the library, memory writes the autobiography.
And no, a bigger context window is not a substitute. Stuffing an entire history into a 1M-token window increases latency and cost, and models still struggle to use information buried deep in long contexts. A larger window filled with contradictory or outdated entries makes reasoning worse, not better. The two are complementary: the strongest architectures use RAG for shared domain knowledge and memory for user-specific state.
Agent memory works as a four-stage loop:writeextracts what matters from a turn,consolidatededuplicates and resolves contradictions,retrievepulls relevant memories back into context, andforgetdecays or evicts stale entries. Skipping the consolidate and forget stages is why naive memory systems degrade over time.
Most tutorials stop at “store embeddings, retrieve top-K.” That produces a memory that only grows and never cleans itself. A production-grade architecture treats memory as a lifecycle.
After each turn, decide what is worth persisting. Not every message contains a durable fact - “thanks!” does not need to be remembered. A common pattern is an extraction step: an LLM call that pulls candidate facts out of the exchange before anything is stored.
This is the stage naive systems skip. New information must be integrated without contradicting what’s already stored. When a new fact overlaps an existing one, you update in place rather than duplicate. Mem0’s production pipeline formalizes this as an extract-then-consolidate step, and the Generative Agents paper popularized “reflection” - periodically synthesizing raw episodic memories into higher-level semantic insights.
At query time, pull the memories relevant to the current message. Three strategies dominate: vector similarity (semantic match), recency (most recent events first), and graph traversal (following relationships between entities). Many systems combine them.
Memory that only grows becomes memory that retrieves noise. Decay and eviction - dropping low-value or stale entries - keep retrieval sharp. Redis’s engineering writeup makes the point that context pollution, where irrelevant information degrades reasoning, is a first-order problem, not an edge case.
You can implement working, semantic, and episodic memory in a few dozen lines of framework-agnostic Python before reaching for any library. The code below uses stand-inllm() andembed() functions so you can drop in whichever model and embedding provider you use.
Here’s a working skeleton. It’s deliberately dependency-light - the goal is to understand the moving parts, not to hide them behind an SDK.
class WorkingMemory: """Keeps the last N turns verbatim; older turns roll into a running summary.""" def __init__(self, max_turns=10): self.max_turns = max_turns self.buffer = [] # recent turns: {"role": ..., "content": ...} self.summary = "" # compressed history of evicted turns def add(self, role, content): self.buffer.append({"role": role, "content": content}) if len(self.buffer) > self.max_turns: # Evict the oldest turns into the running summary instead of dropping them cut = len(self.buffer) - self.max_turns overflow, self.buffer = self.buffer[:cut], self.buffer[cut:] self.summary = self._summarize(self.summary, overflow) def _summarize(self, prior, turns): text = "\n".join(f"{t['role']}: {t['content']}" for t in turns) return llm(f"Prior summary:\n{prior}\n\nNew turns:\n{text}\n\n" "Update the summary in 3-4 sentences.") def render(self): head = f"[Summary so far]\n{self.summary}\n\n" if self.summary else "" recent = "\n".join(f"{t['role']}: {t['content']}" for t in self.buffer) return head + recent
python
import numpy as npdef cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9))class SemanticMemory: """Durable facts. Near-duplicate facts overwrite the old entry (consolidation).""" def __init__(self, dedupe_threshold=0.92): self.facts = [] # list[str] self.vectors = [] # list[np.ndarray] self.threshold = dedupe_threshold def write(self, fact): vec = embed(fact) # your embedding model -> np.ndarray for i, v in enumerate(self.vectors): if cosine(vec, v) > self.threshold: # Same subject -> update in place rather than store a contradiction self.facts[i], self.vectors[i] = fact, vec return self.facts.append(fact) self.vectors.append(vec) def retrieve(self, query, k=3): q = embed(query) ranked = sorted(zip(self.facts, self.vectors), key=lambda fv: cosine(q, fv[1]), reverse=True) return [fact for fact, _ in ranked[:k]]
(In production you’d swap the in-memory lists for a real vector store - the trade-offs there, like index choice and hybrid search, are a topic on their own that I’ll cover next in this series. The logic above stays identical.)
import time, jsonclass EpisodicMemory: """Append-only log of what happened, with timestamps for recency retrieval.""" def __init__(self, path="episodes.jsonl"): self.path = path def log(self, event, meta=None): record = {"ts": time.time(), "event": event, "meta": meta or {}} with open(self.path, "a") as f: f.write(json.dumps(record) + "\n") def recent(self, n=3): with open(self.path) as f: lines = f.readlines() return [json.loads(line) for line in lines[-n:]]
python
def build_context(user_msg, working, semantic, episodic): facts = semantic.retrieve(user_msg, k=3) events = episodic.recent(n=3) return "\n".join([ "## What I know about you:", *[f"- {f}" for f in facts], "\n## Recent events:", *[f"- {e['event']}" for e in events], "\n## Conversation:", working.render(), f"\n## Current message:\n{user_msg}", ])def agent_turn(user_msg, working, semantic, episodic): working.add("user", user_msg) reply = llm(build_context(user_msg, working, semantic, episodic)) working.add("assistant", reply) # Write phase: persist durable facts + log the interaction for fact in extract_facts(user_msg, reply): # an LLM extraction step semantic.write(fact) episodic.log(f"user said: {user_msg}", {"reply": reply}) return reply
That’s the whole architecture in one screen: working memory for coherence, semantic memory for facts, episodic memory for history, and a turn loop that reads before it answers and writes after. Procedural memory slots in the same way — as retrievable workflow templates or tool policies you inject into build_context.
Not every agent needs every memory type. Match the architecture to the use case and add complexity only when operational value justifies it: episodic for interaction history, semantic for stable facts and personalization, procedural for repeatable skills. Start with working memory plus one long-term type, then expand.
The most common mistake is building all four types on day one. Redis’s guidance is blunt about it: use the simplest memory that reliably supports the task.
The four failure modes that show up at scale are context pollution (irrelevant memories degrading reasoning), temporal conflicts (new facts contradicting old ones), retrieval latency and cost, and multi-session inconsistency. Each maps to a specific stage of the memory lifecycle, and each has a standard mitigation.
Context pollution. Retrieval returns too much, or the wrong things, and the model’s reasoning degrades. Mitigation: aggressive relevance filtering, smaller top-K, and the forget stage.
Temporal conflicts. Old and new facts disagree. Mitigation: consolidate on write - overwrite rather than append, and timestamp facts so the current state is unambiguous.
Latency and cost. Every retrieval and every LLM-managed memory operation adds tokens and time. MemGPT’s own design notes that letting the model page memory in and out adds overhead on every interaction where it decides to do so. Mitigation: cache, batch, and only retrieve when the query warrants it.
Do you even need a framework? Rolling your own, as above, is the right call for learning and for simple cases. Once you need scale, temporal knowledge graphs, or multi-agent consistency, the mature options — Mem0, Letta (the production evolution of MemGPT), Zep, and LangMem — each take a different architectural stance. Mem0 is a framework-agnostic memory layer you bolt on; Letta is a full agent runtime built around the OS-style memory model. Pick based on whether you want a library or a runtime. The code above is enough to evaluate them with your eyes open instead of picking on vibes.
How does AI agent memory work? An external system captures information from each turn, stores it in working, episodic, semantic, or procedural memory, and retrieves the relevant pieces back into the context window on later turns - giving a stateless model the appearance of continuity.
What are the types of AI agent memory? Four: working memory (the active context window), episodic memory (timestamped records of past interactions), semantic memory (durable facts and preferences), and procedural memory (learned skills and workflows). Working memory is short-term; the rest are long-term.
What’s the difference between agent memory and RAG? RAG is read-only retrieval from a shared, static corpus. Agent memory reads and writes, is scoped to a specific user or session, and updates as the agent learns. They’re complementary - RAG for shared knowledge, memory for personal state.
How do AI agents remember across sessions? By persisting memories in an external store (a database, vector store, or file) outside the model, then querying that store at the start of each new session and injecting the results into the prompt.
Do I need a framework like Mem0, or can I build my own? For learning and simple agents, a few dozen lines of Python (like the code above) is enough. Reach for Mem0, Letta, Zep, or LangMem when you need scale, temporal knowledge graphs, or multi-agent consistency.
Episodic vs semantic vs procedural — what’s the difference? Episodic stores what happened (events), semantic stores what’s true (facts), and procedural stores how to act (skills). The distinction traces back to Tulving’s 1972 memory research and is adapted for agents in the CoALA framework.
If this helped, I’m writing a whole series on the internals of production LLM systems - RAG, vector databases, agents, and context engineering, all interlinked. Follow along for the rest, and if you haven’t yet, read [INTERNAL LINK: The Ultimate Guide to LLM Memory] - the post this one builds on. RAG and vector databases are up next.
How AI Agent Memory Actually Works - And How to Build It was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.