How AI Agents Remember: A Simple Guide to Memory, Vector Search, and RAG

A guide on AI agent memory explains that large language models retain nothing between requests, so the surrounding agent must manage a limited context window — roughly 4,000 to 8,000 tokens for older models, up to 272,000 tokens for GPT-5 and Claude 3.7 Sonnet, and up to 1 million tokens for Gemini 2.5. The guide recommends a rolling context window that keeps the newest messages and drops the oldest when space runs out, and includes a LangGraph/LangChain code example using Ollama's llama3.2 that summarizes once state holds more than 6 messages while keeping the last 2. It notes the approach is easy to build but discards older information even when important.

An AI chatbot may seem to remember earlier messages, but the language model LLM does not remember them on its own. It uses the information sent to it with each request. The agent — the system around the model — keeps track of useful information. It sends the model the details it needs for each task. Here’s why this matters: 1. The model has limited space. It can read only a certain amount of text at once. This limit is called the context window. Long conversations, files, and search results can fill it up. Older information may then be left out, even if it is important. 2. More text can cost more and take longer. Sending the whole conversation each time adds extra work. Sending only useful details can save time and money. 3. Too much unrelated information can hurt answers. Important details can get lost in the clutter. Clear, focused input helps the model answer better. 4. Long tasks need progress tracking. The agent needs to know what is finished, what is next, and what has been decided. Otherwise, it may repeat work or ignore earlier decisions. 5. The model may lack needed information. Private data and recent information may not be part of its training. Search tools can find these details when needed. 6. Users need control over memory. Memory management helps decide what to save, forget, or avoid storing. It can also let users view and delete saved information. Two simple ways to manage memory are a rolling context window and keyword search. A rolling window keeps recent messages. Keyword search finds saved information by matching words. What is a context window? It is the space available for the information sent to the model in one request. Think of it as short-term memory for that request. What is context length? It is the maximum amount of text the model can handle at once, measured in tokens. What is a token? A token is a small piece of text. In English, it is roughly three-quarters of a word, or about four characters. · 1,000 tokens ≈ 750 words How big are context windows? · Older models: about 4,000 tokens ~3,000 words, ~12 pages or 8,000 tokens ~6,000 words, ~24 pages · GPT-5 and Claude 3.7 Sonnet: up to 272,000 tokens · Gemini 2.5: up to 1 million tokens Why it matters The context window has limited space. Include what the model needs and leave out unrelated details. The simplest approach Send the current question and the conversation so far. When there is not enough space, keep the newest messages. Rolling context window · Send the conversation history that fits in the window with each request. · When the window fills up, remove the oldest messages to make room for new ones. Pros · Easy to build · Simple · Works for many use cases Cons · Older information is removed even when it is important. · Long questions and answers can fill the window quickly. python import argparseimport osimport refrom langchain core.messages import AIMessage, AnyMessage, HumanMessage, RemoveMessage, SystemMessage, from langchain core.runnables import RunnableLambdafrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphMAX MESSAGES = 6 summarize once state holds more than this many messagesKEEP LAST = 2 how many recent messages survive a summarization passOLLAMA MODEL = "llama3.2" also try gpt-oss:20b, qwen:4b, deepseek-r1:1.5bOLLAMA URL = os.environ.get "OLLAMA HOST", "http://localhost:11434" class State MessagesState : summary: strdef strip thinking msg: AIMessage - AIMessage: """Reasoning models deepseek-r1, ... may inline <think blocks; keep them out of state.""" if isinstance msg.content, str : msg.content = re.sub r"<think . ?</think ", "", msg.content, flags=re.DOTALL .strip return msgdef make llm model: str | None : from langchain ollama import ChatOllama model = model or OLLAMA MODEL print f"Using Ollama model '{model}' at {OLLAMA URL}\n" return ChatOllama model=model, base url=OLLAMA URL, temperature=0 | RunnableLambda strip thinking def build graph llm : def chat state: State : prompt: list AnyMessage = SystemMessage "You are a helpful, concise assistant." if state.get "summary" : prompt.append SystemMessage "Earlier messages in this conversation were condensed into this summary. " f"Treat it as things you remember about the user:\n{state 'summary' }" prompt += state "messages" print f" - sending {len prompt } messages to the model" return {"messages": llm.invoke prompt } def should summarize state: State : return "summarize" if len state "messages" MAX MESSAGES else END def summarize state: State : old, summary = state "messages" :-KEEP LAST , state.get "summary", "" rules = "Write it as short bullet points. Always keep every personal fact about the user " " name, location, preferences and the topics discussed. Output only the bullets." if summary: instruction = f"This is the summary so far:\n{summary}\n\n" f"Rewrite the summary so it also covers the new messages above. {rules}" else: instruction = f"Summarize the conversation above. {rules}" new summary = llm.invoke old + HumanMessage instruction .content print f" rolled {len old } old messages into the summary" RemoveMessage tells the add messages reducer to drop these ids from state. return {"summary": new summary, "messages": RemoveMessage id=m.id for m in old } builder = StateGraph State builder.add node "chat", chat builder.add node "summarize", summarize builder.add edge START, "chat" builder.add conditional edges "chat", should summarize, "summarize", END builder.add edge "summarize", END The checkpointer persists state per thread id across invocations. return builder.compile checkpointer=InMemorySaver DEMO TURNS = "Hi My name is Deepak and I live in Noida.", "I'm learning LangGraph. What is a StateGraph in one sentence?", "What is a checkpointer for?", "My favourite language is Python.", "Give me one tip for writing good prompts.", "What's a reducer in LangGraph?", "Do you still remember my name and city?", def show state graph, config : state = graph.get state config .values print f" state: {len state 'messages' } messages kept, summary={'yes' if state.get 'summary' else 'no'}" def main - None: parser = argparse.ArgumentParser description="LangGraph rolling-context demo" parser.add argument "-i", "--interactive", action="store true", help="chat yourself instead of the scripted demo" parser.add argument "-m", "--model", help=f"Ollama model name default: {OLLAMA MODEL} " args = parser.parse args graph = build graph make llm args.model config = {"configurable": {"thread id": "demo"}} def turn text: str : print f"USER: {text}" result = graph.invoke {"messages": HumanMessage text }, config reply = next m for m in reversed result "messages" if isinstance m, AIMessage print f"AI: {reply.content}" show state graph, config print if args.interactive: print "Type 'quit' to exit, 'summary' to print the rolling summary.\n" while text := input " " .strip .lower not in {"quit", "exit"}: if text == "summary": print graph.get state config .values.get "summary" or " no summary yet ", "\n" elif text: turn text else: for text in DEMO TURNS: turn text print "Final rolling summary:\n" + graph.get state config .values.get "summary" or " none " What it is: Full-text search finds information by matching words. An agent can use it to find useful past messages or documents and add them to the model’s input. How it works 1. Build an index First, the system prepares the text for searching: · Split the text into words. This is called tokenization. · Make words easier to match. For example, change them to lowercase and shorten “running” and “runs” to “run.” These steps are called normalization and stemming. · Remove common words such as “the,” “is,” and “and.” This is called stop-word removal. The system then lists which messages or documents contain each word. This list is called an inverted index . It works like the index at the back of a book. Why it is fast: The system looks up a word in the index instead of reading every document again. 2. Put the best matches first A scoring method called BM25 helps rank matching passages. It considers: · How often a search word appears in a passage. · How rare the word is across the saved documents. Less common words carry more weight. · The length of the passage. The score is adjusted so length does not unfairly affect the result. 3. Search and use the results When a user asks something: 1. Prepare the search question in the same way as the saved text. 2. Use BM25 to score the matching passages. 3. Choose the best few results. This is called selecting the “top K.” 4. Add those results to the model’s input, shortening or summarizing them if needed. This gives the model useful background without sending the full history. In practice You can build a simple version in Python. Larger systems usually keep the index in a database. Useful memory is about more than finding old messages. An agent can also remember a user’s preferences and past requests to give more helpful answers. Example: Imagine a food-ordering assistant. · Without memory: When the user says “Order my usual lunch,” the assistant must ask what food they want, whether they have dietary preferences, and where to deliver it. · With memory: The assistant has saved that the user is vegetarian, usually orders paneer tikka, and wants weekday deliveries to the office. It can reply, “Paneer tikka from your usual place, delivered to your office. Shall I place the order?” To do this, the agent saves conversation history in a database. For each new request, it finds relevant details and adds them to the model’s input. Semantic search and vector stores can help with this. """Full-text search BM25 memory with LangGraph.Only the last WINDOW messages stay in the prompt. Older messages are moved into aninverted index; on every turn the user's question is searched with BM25 and thetop K past messages are added to the prompt. START - retrieve - chat - archive - END1. Index: tokenize - lowercase - drop stop words - stem - inverted index2. Rank: BM25 term frequency, rarity of the term, document length 3. Answer: analyze the question the same way, score, take the top K, add to prompt"""import argparseimport heapqimport mathimport refrom collections import Counter, defaultdictimport snowballstemmerfrom langchain core.messages import AIMessage, AnyMessage, HumanMessage, RemoveMessage, SystemMessagefrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphfrom rolling context import OLLAMA MODEL, make llmWINDOW = 4 messages kept verbatim in the prompt 2 user/assistant turns TOP K = 3 past messages pulled back in by searchSNIPPET CHARS = 300 long hits are shortened before going into the promptSTOP WORDS = set "a an and are as at be but by can do does for from had has have how i i'm in is it its " "me my of on or so that the their them then there these they this to was we what when " "where which who why will with you your about could would should did any all just " "he him his she her our us get got hi hello s m t d ll re ve".split stemmer = snowballstemmer.stemmer "english" def analyze text: str - list str : """Tokenize, lowercase, remove stop words and stem: 'running', 'runs' - 'run'.""" tokens = re.findall r" a-z0-9 +", text.lower return stemmer.stemWords t for t in tokens if t not in STOP WORDS class BM25Index: """An in-memory inverted index ranked with BM25. Real systems keep this in a database.""" def init self, k1: float = 1.5, b: float = 0.75 : self.k1, self.b = k1, b self.docs: list str = self.doc len: list int = self.postings: dict str, dict int, int = defaultdict dict term - {doc id: term freq} def add self, text: str - int: doc id = len self.docs terms = analyze text self.docs.append text self.doc len.append len terms for term, tf in Counter terms .items : self.postings term doc id = tf return doc id def search self, query: str, k: int = TOP K - list tuple float, str : if not self.docs: return n docs, avg len = len self.docs , sum self.doc len / len self.docs scores: dict int, float = defaultdict float for term in set analyze query : posting = self.postings.get term if not posting: continue the index lookup skips every document without this term Rare terms few documents contain them get a higher weight. idf = math.log 1 + n docs - len posting + 0.5 / len posting + 0.5 for doc id, tf in posting.items : More occurrences help, with diminishing returns; long documents are penalized. length norm = 1 - self.b + self.b self.doc len doc id / avg len scores doc id += idf tf self.k1 + 1 / tf + self.k1 length norm top = heapq.nlargest k, scores.items , key=lambda kv: kv 1 return score, self.docs doc id for doc id, score in top class State MessagesState : retrieved: list str def build graph llm, index: BM25Index, top k: int = TOP K : def retrieve state: State : query = state "messages" -1 .content hits = index.search query, top k print f" search terms: {analyze query }" for score, text in hits: print f" hit {score:5.2f} {text :90 r}" if not hits: print " no hits " return {"retrieved": text :SNIPPET CHARS for , text in hits } def chat state: State : prompt: list AnyMessage = SystemMessage "You are a helpful, concise assistant." if state "retrieved" : memories = "\n".join f"- {m}" for m in state "retrieved" prompt.append SystemMessage "These earlier messages from this conversation were found by search. " f"Use them if they help answer the user:\n{memories}" prompt += state "messages" print f" - sending {len prompt } messages to the model" return {"messages": llm.invoke prompt } def archive state: State : old = state "messages" :-WINDOW for m in old: index.add f"{'user' if isinstance m, HumanMessage else 'assistant'}: {m.content}" if old: print f" archived {len old } messages into the index {len index.docs } docs, {len index.postings } terms " return {"messages": RemoveMessage id=m.id for m in old } builder = StateGraph State builder.add node "retrieve", retrieve builder.add node "chat", chat builder.add node "archive", archive builder.add edge START, "retrieve" builder.add edge "retrieve", "chat" builder.add edge "chat", "archive" builder.add edge "archive", END return builder.compile checkpointer=InMemorySaver DEMO TURNS = "Hi, I'm Deepak. I'm planning a trip to Japan in April.", "My laptop is a ThinkPad X1 running Fedora Linux.", "Invoice 4512 from Acme Corp is overdue by 30 days, I need to chase them.", "My son's birthday is on 14 March and she loves painting.", "Explain BM25 in one sentence.", "Suggest a name for a cat.", Everything above has left the prompt window by now; only search can bring it back. "When is my son's birthday, and what gift could I get her?", "Which invoice was overdue, and from which company?", "What operating system runs on my laptop?", def main - None: parser = argparse.ArgumentParser description="LangGraph full-text search BM25 memory demo" parser.add argument "-i", "--interactive", action="store true", help="chat yourself instead of the scripted demo" parser.add argument "-m", "--model", help=f"Ollama model name default: {OLLAMA MODEL} " parser.add argument "-k", "--top-k", type=int, default=TOP K, help="past messages to retrieve per turn" args = parser.parse args index = BM25Index graph = build graph make llm args.model , index, args.top k config = {"configurable": {"thread id": "demo"}} def turn text: str : print f"USER: {text}" result = graph.invoke {"messages": HumanMessage text }, config reply = next m for m in reversed result "messages" if isinstance m, AIMessage print f"AI: {reply.content}\n" if args.interactive: print "Type 'quit' to exit, 'index' to list the indexed messages.\n" while text := input " " .strip .lower not in {"quit", "exit"}: if text == "index": print "\n".join f" {i} {d}" for i, d in enumerate index.docs or " empty ", "\n" elif text: turn text else: for text in DEMO TURNS: turn text What is semantic memory? It is long-term memory for facts, concepts, and knowledge. An agent retrieves this information when it needs it. “Semantic” means “related to meaning.” Semantic search looks for similar meanings, not just matching words. Example: A user has saved this message: “I’m vegetarian, please don’t suggest chicken or fish.” Later, the user asks: “What food should I avoid recommending?” · Keyword search: It looks for words such as “food,” “avoid,” and “recommending.” These words are not in the saved message, so the search can miss it. · Semantic search: It looks for a similar meaning. It can connect “food to avoid” with “don’t suggest chicken or fish” and find the message. How semantic search works 1. Turn each message into a list of numbers called an embedding . Think of it as a point on a map of meanings. 2. Messages with similar meanings are close on this map. For example, “car” and “automobile” are close, while “car” and “banana” are far apart. 3. Turn the question into an embedding too. Then find the saved messages with the closest embeddings. How are results saved and retrieved? There are two stages: saving information first, then finding it when a question comes in. Phase 1: Saving before any question is asked 1. Create an embedding for each message. For example, the vegetarian message becomes a list of numbers like 0.9, 0.1, … . These numbers are only an illustration. 2. Save the embedding and the original text in a vector database. This database searches lists of numbers. 3. Here are some example entries. The numbers are for illustration only: Text Embedding 0.9, 0.1, … “My office address is on MG Road.” 0.1, 0.9, … “I usually order lunch around 1 pm.” 0.4, 0.7, … Phase 2: Retrieving when the user asks something 1. Create an embedding for the question. Use the same embedding model as before. For example, “What food should I avoid recommending?” becomes a list like 0.85, 0.15, … . 2. Compare the embeddings. A measure called cosine similarity compares the directions of the number lists. Higher scores mean a closer match. Here are example scores, not results from a real search: 3. Choose the best few matches. Sort the results by score and keep the top K, such as the top two. A minimum score can help leave out weak matches. 4. Add useful results to the model’s input. For example: Past messages: Question: What food should I avoid recommending? 5. The model answers using the saved details: “Avoid recommending meat or fish, since you’re vegetarian.” """Semantic memory with a vector store, using LangGraph and Ollama.Each thing the user tells the assistant is embedded turned into a vector of numbersthat captures its meaning and saved in LangGraph's long-term Store. On every turn thequestion is embedded too, and the memories whose vectors are closest in meaning cosinesimilarity are added to the prompt.Unlike the BM25 example, no shared keywords are needed: "a gift for my kid's special day"finds "my daughter's birthday is on 14 March". And unlike a checkpointer, the Store isshared across threads, so a brand-new conversation can still recall facts from an old one. START - recall - chat - remember - END"""import argparseimport uuidfrom langchain core.messages import AIMessage, AnyMessage, HumanMessage, SystemMessagefrom langchain core.runnables import RunnableConfigfrom langchain ollama import OllamaEmbeddingsfrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphfrom langgraph.store.base import BaseStorefrom langgraph.store.memory import InMemoryStorefrom rolling context import OLLAMA MODEL, OLLAMA URL, make llmEMBED MODEL = "nomic-embed-text" install with: ollama pull nomic-embed-textEMBED DIMS = 768 vector size produced by nomic-embed-textWINDOW = 4 recent messages sent verbatimTOP K = 3 memories retrieved per turnMIN SCORE = 0.45 cosine similarity below this is treated as "not related"class State MessagesState : memories: list str def make store embed model: str = EMBED MODEL - InMemoryStore: """A key-value store with a vector index on the 'text' field. Swap for PostgresStore in production.""" embeddings = OllamaEmbeddings model=embed model, base url=OLLAMA URL return InMemoryStore index={"embed": embeddings, "dims": EMBED DIMS, "fields": "text" } def build graph llm, store: BaseStore, top k: int = TOP K : def namespace config: RunnableConfig : Memories belong to a user, not to a thread, so every thread of that user can see them. return "memories", config "configurable" "user id" def recall state: State, config: RunnableConfig, , store: BaseStore : query = state "messages" -1 .content hits = store.search namespace config , query=query, limit=top k for h in hits: mark = "use " if h.score = MIN SCORE else "skip" print f" {mark} {h.score:.3f} {h.value 'text' r}" if not hits: print " memory is empty " return {"memories": h.value "text" for h in hits if h.score = MIN SCORE } def chat state: State : prompt: list AnyMessage = SystemMessage "You are a helpful, concise assistant." if state "memories" : facts = "\n".join f"- {m}" for m in state "memories" prompt.append SystemMessage "Things the user told you in earlier conversations retrieved by meaning . " f"Use them when relevant:\n{facts}" prompt += state "messages" -WINDOW: print f" - sending {len prompt } messages to the model" return {"messages": llm.invoke prompt } def remember state: State, config: RunnableConfig, , store: BaseStore : text = state "messages" -2 .content the user message of this turn Simple heuristic: statements are worth remembering, questions usually aren't. if not text.rstrip .endswith "?" : store.put namespace config , str uuid.uuid4 , {"text": text} embedded on put print " saved to semantic memory" return {} builder = StateGraph State builder.add node "recall", recall builder.add node "chat", chat builder.add node "remember", remember builder.add edge START, "recall" builder.add edge "recall", "chat" builder.add edge "chat", "remember" builder.add edge "remember", END Checkpointer = short-term memory per thread; store = long-term memory per user. return builder.compile checkpointer=InMemorySaver , store=store Two separate conversations threads . The second starts with an empty history and its questions share almost no words with the facts, so only semantic search can help.DEMO SESSIONS = { "monday": "Hi, I'm Deepak and I live in Noida.", "I'm allergic to peanuts.", "My daughter turns eight on 14 March and she adores painting.", "I drive a red Tata Nexon electric car.", "At work I'm building an AI agent with LangGraph.", , "friday": "What should I get my kid for her special day?", "Suggest a quick snack recipe that is safe for me to eat.", "Any tips for a long road trip in my vehicle?", "What framework am I using for my project at the office?", ,}def main - None: parser = argparse.ArgumentParser description="LangGraph semantic memory vector store demo" parser.add argument "-i", "--interactive", action="store true", help="chat yourself instead of the scripted demo" parser.add argument "-m", "--model", help=f"Ollama chat model default: {OLLAMA MODEL} " parser.add argument "-e", "--embed-model", default=EMBED MODEL, help=f"Ollama embedding model default: {EMBED MODEL} " parser.add argument "-k", "--top-k", type=int, default=TOP K, help="memories to retrieve per turn" args = parser.parse args store = make store args.embed model graph = build graph make llm args.model , store, args.top k user id = "deepak" def turn text: str, thread id: str : print f"USER: {text}" config = {"configurable": {"thread id": thread id, "user id": user id}} result = graph.invoke {"messages": HumanMessage text }, config reply = next m for m in reversed result "messages" if isinstance m, AIMessage print f"AI: {reply.content}\n" if args.interactive: thread = 1 print "Type 'quit' to exit, 'new' to start a new thread memory is kept , 'memories' to list them.\n" while text := input f" thread {thread} " .strip .lower not in {"quit", "exit"}: if text == "new": thread += 1 print f"--- new thread {thread}: chat history is empty, semantic memory is not ---\n" elif text == "memories": items = store.search "memories", user id , limit=100 print "\n".join f"- {i.value 'text' }" for i in items or " empty ", "\n" elif text: turn text, f"thread-{thread}" else: for thread id, turns in DEMO SESSIONS.items : print f"=============== thread '{thread id}' new conversation ===============\n" for text in turns: turn text, thread id Most organizations have large internal knowledge bases: past tickets, policies, product docs, troubleshooting guides. This is a gold mine of knowledge, but it’s hard to use in the moment. A customer service representative handling a live request often can’t find the relevant article or similar past case quickly enough to act on it. The idea What if an LLM could draw on that same knowledge base? The solution This is exactly what RAG makes possible. The knowledge base is indexed ahead of time chunked, embedded, and stored in a vector database . At query time, the most relevant pieces are retrieved and handed to the LLM as context, so its answers are accurate, current, and specific to your organization. Think of the diagram as a library getting ready before anyone walks in to ask a question. Reading left to right: 1. Source document This is your raw knowledge: a help article, a policy, an old support ticket. It’s usually too long to use all at once. 2. Chunks The dashed lines are scissors. The document gets cut into small, bite-sized pieces, each about a paragraph or two. Small pieces make it easier to find the exact part that answers a question, instead of handing over a whole 50-page manual. 3. Encoder Each chunk goes through an encoder, a model that reads the text and captures what it means. 4. Dense vectors The encoder turns each chunk into a list of numbers. Each little square in the strip stands for one number, and its shade shows how big the number is. You can think of this list as the chunk’s meaning fingerprint: chunks that say similar things get similar fingerprints, even if they use different words. “How do I reset my password” and “I forgot my login” end up close together. 5. Vector database These fingerprints are stored as points in a big map. The gray dots are chunks that were already stored. The teal dots are the new ones just added. Chunks with similar meanings sit near each other on this map. Why it matters All of this happens once, ahead of time. Later, when a customer service rep asks something like “customer can’t log in after an update,” that question gets its own fingerprint. The system looks for the nearest points on the map and grabs those chunks. The LLM then uses them to write an answer based on your company’s actual knowledge. Retrieval and generation So far win RAG we have seen how a document is divided into chunks and saved in to vector database. Now lets looks at the retrieval and response generation part. The figure below shows once customer service use case. Retrieval steps 1–4 : finding the right information The rep asks: “Customer updated the app and can’t log in. What should I do?” Generation steps 5–6 : writing the answer How AI Agents Remember: A Simple Guide to Memory, Vector Search, and RAG https://pub.towardsai.net/how-ai-agents-remember-a-simple-guide-to-memory-vector-search-and-rag-8a7aebb1c129 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.