cd /news/ai-agents/how-ai-agents-remember-a-simple-guid… · home › topics › ai-agents › article
[ARTICLE · art-165918] src=pub.towardsai.net ↗ pub= topic=ai-agents verified=true sentiment=· neutral

How AI Agents Remember: A Simple Guide to Memory, Vector Search, and RAG

A guide on AI agent memory explains that large language models retain nothing between requests, so the surrounding agent must manage a limited context window — roughly 4,000 to 8,000 tokens for older models, up to 272,000 tokens for GPT-5 and Claude 3.7 Sonnet, and up to 1 million tokens for Gemini 2.5. The guide recommends a rolling context window that keeps the newest messages and drops the oldest when space runs out, and includes a LangGraph/LangChain code example using Ollama's llama3.2 that summarizes once state holds more than 6 messages while keeping the last 2. It notes the approach is easy to build but discards older information even when important.

by read20 min views1 publishedNov 8, 2026

An AI chatbot may seem to remember earlier messages, but the language model (LLM) does not remember them on its own. It uses the information sent to it with each request.

The agent — the system around the model — keeps track of useful information. It sends the model the details it needs for each task.

Here’s why this matters:

  1. The model has limited space. It can read only a certain amount of text at once. This limit is called the context window. Long conversations, files, and search results can fill it up. Older information may then be left out, even if it is important.

  2. More text can cost more and take longer. Sending the whole conversation each time adds extra work. Sending only useful details can save time and money.

  3. Too much unrelated information can hurt answers. Important details can get lost in the clutter. Clear, focused input helps the model answer better.

  4. Long tasks need progress tracking. The agent needs to know what is finished, what is next, and what has been decided. Otherwise, it may repeat work or ignore earlier decisions.

  5. The model may lack needed information. Private data and recent information may not be part of its training. Search tools can find these details when needed.

  6. Users need control over memory. Memory management helps decide what to save, forget, or avoid storing. It can also let users view and delete saved information.

Two simple ways to manage memory are a rolling context window and keyword search. A rolling window keeps recent messages. Keyword search finds saved information by matching words.

What is a context window?

It is the space available for the information sent to the model in one request. Think of it as short-term memory for that request.

What is context length?

It is the maximum amount of text the model can handle at once, measured in tokens.

What is a token?

A token is a small piece of text. In English, it is roughly three-quarters of a word, or about four characters.

· 1,000 tokens ≈ 750 words

How big are context windows?

· Older models: about 4,000 tokens (~3,000 words, ~12 pages) or 8,000 tokens (~6,000 words, ~24 pages)

· GPT-5 and Claude 3.7 Sonnet: up to 272,000 tokens

· Gemini 2.5: up to 1 million tokens

Why it matters

The context window has limited space. Include what the model needs and leave out unrelated details.

The simplest approach

Send the current question and the conversation so far. When there is not enough space, keep the newest messages.

Rolling context window

· Send the conversation history that fits in the window with each request.

· When the window fills up, remove the oldest messages to make room for new ones.

Pros

· Easy to build

· Simple

· Works for many use cases

Cons

· Older information is removed even when it is important.

· Long questions and answers can fill the window quickly.

import argparseimport osimport refrom langchain_core.messages import (    AIMessage,    AnyMessage,    HumanMessage,    RemoveMessage,    SystemMessage,)from langchain_core.runnables import RunnableLambdafrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphMAX_MESSAGES = 6  # summarize once state holds more than this many messagesKEEP_LAST = 2  # how many recent messages survive a summarization passOLLAMA_MODEL = "llama3.2"  # also try gpt-oss:20b, qwen:4b, deepseek-r1:1.5bOLLAMA_URL = os.environ.get("OLLAMA_HOST", "http://localhost:11434")class State(MessagesState):    summary: strdef _strip_thinking(msg: AIMessage) -> AIMessage:    """Reasoning models (deepseek-r1, ...) may inline <think> blocks; keep them out of state."""    if isinstance(msg.content, str):        msg.content = re.sub(r"<think>.*?</think>", "", msg.content, flags=re.DOTALL).strip()    return msgdef make_llm(model: str | None):    from langchain_ollama import ChatOllama    model = model or OLLAMA_MODEL    print(f"Using Ollama model '{model}' at {OLLAMA_URL}\n")    return ChatOllama(model=model, base_url=OLLAMA_URL, temperature=0) | RunnableLambda(_strip_thinking)def build_graph(llm):    def chat(state: State):        prompt: list[AnyMessage] = [SystemMessage("You are a helpful, concise assistant.")]        if state.get("summary"):            prompt.append(                SystemMessage(                    "Earlier messages in this conversation were condensed into this summary. "                    f"Treat it as things you remember about the user:\n{state['summary']}"                )            )        prompt += state["messages"]        print(f"   -> sending {len(prompt)} messages to the model")        return {"messages": [llm.invoke(prompt)]}    def should_summarize(state: State):        return "summarize" if len(state["messages"]) > MAX_MESSAGES else END    def summarize(state: State):        old, summary = state["messages"][:-KEEP_LAST], state.get("summary", "")        rules = (            "Write it as short bullet points. Always keep every personal fact about the user "            "(name, location, preferences) and the topics discussed. Output only the bullets."        )        if summary:            instruction = (                f"This is the summary so far:\n{summary}\n\n"                f"Rewrite the summary so it also covers the new messages above. {rules}"            )        else:            instruction = f"Summarize the conversation above. {rules}"        new_summary = llm.invoke(old + [HumanMessage(instruction)]).content        print(f"   ** rolled {len(old)} old messages into the summary")        # RemoveMessage tells the add_messages reducer to drop these ids from state.        return {"summary": new_summary, "messages": [RemoveMessage(id=m.id) for m in old]}    builder = StateGraph(State)    builder.add_node("chat", chat)    builder.add_node("summarize", summarize)    builder.add_edge(START, "chat")    builder.add_conditional_edges("chat", should_summarize, ["summarize", END])    builder.add_edge("summarize", END)    # The checkpointer persists state per thread_id across invocations.    return builder.compile(checkpointer=InMemorySaver())DEMO_TURNS = [    "Hi! My name is Deepak and I live in Noida.",    "I'm learning LangGraph. What is a StateGraph in one sentence?",    "What is a checkpointer for?",    "My favourite language is Python.",    "Give me one tip for writing good prompts.",    "What's a reducer in LangGraph?",    "Do you still remember my name and city?",]def show_state(graph, config):    state = graph.get_state(config).values    print(f"   state: {len(state['messages'])} messages kept, summary={'yes' if state.get('summary') else 'no'}")def main() -> None:    parser = argparse.ArgumentParser(description="LangGraph rolling-context demo")    parser.add_argument("-i", "--interactive", action="store_true", help="chat yourself instead of the scripted demo")    parser.add_argument("-m", "--model", help=f"Ollama model name (default: {OLLAMA_MODEL})")    args = parser.parse_args()    graph = build_graph(make_llm(args.model))    config = {"configurable": {"thread_id": "demo"}}    def turn(text: str):        print(f"USER: {text}")        result = graph.invoke({"messages": [HumanMessage(text)]}, config)        reply = next(m for m in reversed(result["messages"]) if isinstance(m, AIMessage))        print(f"AI:   {reply.content}")        show_state(graph, config)        print()    if args.interactive:        print("Type 'quit' to exit, 'summary' to print the rolling summary.\n")        while (text := input("> ").strip()).lower() not in {"quit", "exit"}:            if text == "summary":                print(graph.get_state(config).values.get("summary") or "(no summary yet)", "\n")            elif text:                turn(text)    else:        for text in DEMO_TURNS:            turn(text)        print("Final rolling summary:\n" + (graph.get_state(config).values.get("summary") or "(none)"))

What it is: Full-text search finds information by matching words. An agent can use it to find useful past messages or documents and add them to the model’s input.

How it works

1. Build an index

First, the system prepares the text for searching:

· Split the text into words. This is called tokenization.

· Make words easier to match. For example, change them to lowercase and shorten “running” and “runs” to “run.” These steps are called normalization and stemming.

· Remove common words such as “the,” “is,” and “and.” This is called stop-word removal.

The system then lists which messages or documents contain each word. This list is called an inverted index. It works like the index at the back of a book.

Why it is fast: The system looks up a word in the index instead of reading every document again.

2. Put the best matches first

A scoring method called BM25 helps rank matching passages. It considers:

· How often a search word appears in a passage.

· How rare the word is across the saved documents. Less common words carry more weight.

· The length of the passage. The score is adjusted so length does not unfairly affect the result.

3. Search and use the results

When a user asks something:

  1. Prepare the search question in the same way as the saved text.

  2. Use BM25 to score the matching passages.

  3. Choose the best few results. This is called selecting the “top K.”

  4. Add those results to the model’s input, shortening or summarizing them if needed.

This gives the model useful background without sending the full history.

In practice

You can build a simple version in Python. Larger systems usually keep the index in a database.

Useful memory is about more than finding old messages. An agent can also remember a user’s preferences and past requests to give more helpful answers.

Example: Imagine a food-ordering assistant.

· Without memory: When the user says “Order my usual lunch,” the assistant must ask what food they want, whether they have dietary preferences, and where to deliver it.

· With memory: The assistant has saved that the user is vegetarian, usually orders paneer tikka, and wants weekday deliveries to the office. It can reply, “Paneer tikka from your usual place, delivered to your office. Shall I place the order?”

To do this, the agent saves conversation history in a database. For each new request, it finds relevant details and adds them to the model’s input. Semantic search and vector stores can help with this.

"""Full-text search (BM25) memory with LangGraph.Only the last WINDOW messages stay in the prompt. Older messages are moved into aninverted index; on every turn the user's question is searched with BM25 and thetop K past messages are added to the prompt.    START -> retrieve -> chat -> archive -> END1. Index:  tokenize -> lowercase -> drop stop words -> stem -> inverted index2. Rank:   BM25 (term frequency, rarity of the term, document length)3. Answer: analyze the question the same way, score, take the top K, add to prompt"""import argparseimport heapqimport mathimport refrom collections import Counter, defaultdictimport snowballstemmerfrom langchain_core.messages import AIMessage, AnyMessage, HumanMessage, RemoveMessage, SystemMessagefrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphfrom rolling_context import OLLAMA_MODEL, make_llmWINDOW = 4  # messages kept verbatim in the prompt (2 user/assistant turns)TOP_K = 3  # past messages pulled back in by searchSNIPPET_CHARS = 300  # long hits are shortened before going into the promptSTOP_WORDS = set(    "a an and are as at be but by can do does for from had has have how i i'm in is it its "    "me my of on or so that the their them then there these they this to was we what when "    "where which who why will with you your about could would should did any all just "    "he him his she her our us get got hi hello s m t d ll re ve".split())_stemmer = snowballstemmer.stemmer("english")def analyze(text: str) -> list[str]:    """Tokenize, lowercase, remove stop words and stem: 'running', 'runs' -> 'run'."""    tokens = re.findall(r"[a-z0-9]+", text.lower())    return _stemmer.stemWords([t for t in tokens if t not in STOP_WORDS])class BM25Index:    """An in-memory inverted index ranked with BM25. Real systems keep this in a database."""    def __init__(self, k1: float = 1.5, b: float = 0.75):        self.k1, self.b = k1, b        self.docs: list[str] = []        self.doc_len: list[int] = []        self.postings: dict[str, dict[int, int]] = defaultdict(dict)  # term -> {doc_id: term freq}    def add(self, text: str) -> int:        doc_id = len(self.docs)        terms = analyze(text)        self.docs.append(text)        self.doc_len.append(len(terms))        for term, tf in Counter(terms).items():            self.postings[term][doc_id] = tf        return doc_id    def search(self, query: str, k: int = TOP_K) -> list[tuple[float, str]]:        if not self.docs:            return []        n_docs, avg_len = len(self.docs), sum(self.doc_len) / len(self.docs)        scores: dict[int, float] = defaultdict(float)        for term in set(analyze(query)):            posting = self.postings.get(term)            if not posting:                continue  # the index lookup skips every document without this term            # Rare terms (few documents contain them) get a higher weight.            idf = math.log(1 + (n_docs - len(posting) + 0.5) / (len(posting) + 0.5))            for doc_id, tf in posting.items():                # More occurrences help, with diminishing returns; long documents are penalized.                length_norm = 1 - self.b + self.b * self.doc_len[doc_id] / avg_len                scores[doc_id] += idf * tf * (self.k1 + 1) / (tf + self.k1 * length_norm)        top = heapq.nlargest(k, scores.items(), key=lambda kv: kv[1])        return [(score, self.docs[doc_id]) for doc_id, score in top]class State(MessagesState):    retrieved: list[str]def build_graph(llm, index: BM25Index, top_k: int = TOP_K):    def retrieve(state: State):        query = state["messages"][-1].content        hits = index.search(query, top_k)        print(f"   search terms: {analyze(query)}")        for score, text in hits:            print(f"   hit {score:5.2f}  {text[:90]!r}")        if not hits:            print("   (no hits)")        return {"retrieved": [text[:SNIPPET_CHARS] for _, text in hits]}    def chat(state: State):        prompt: list[AnyMessage] = [SystemMessage("You are a helpful, concise assistant.")]        if state["retrieved"]:            memories = "\n".join(f"- {m}" for m in state["retrieved"])            prompt.append(                SystemMessage(                    "These earlier messages from this conversation were found by search. "                    f"Use them if they help answer the user:\n{memories}"                )            )        prompt += state["messages"]        print(f"   -> sending {len(prompt)} messages to the model")        return {"messages": [llm.invoke(prompt)]}    def archive(state: State):        old = state["messages"][:-WINDOW]        for m in old:            index.add(f"{'user' if isinstance(m, HumanMessage) else 'assistant'}: {m.content}")        if old:            print(f"   ** archived {len(old)} messages into the index ({len(index.docs)} docs, {len(index.postings)} terms)")        return {"messages": [RemoveMessage(id=m.id) for m in old]}    builder = StateGraph(State)    builder.add_node("retrieve", retrieve)    builder.add_node("chat", chat)    builder.add_node("archive", archive)    builder.add_edge(START, "retrieve")    builder.add_edge("retrieve", "chat")    builder.add_edge("chat", "archive")    builder.add_edge("archive", END)    return builder.compile(checkpointer=InMemorySaver())DEMO_TURNS = [    "Hi, I'm Deepak. I'm planning a trip to Japan in April.",    "My laptop is a ThinkPad X1 running Fedora Linux.",    "Invoice #4512 from Acme Corp is overdue by 30 days, I need to chase them.",    "My son's birthday is on 14 March and she loves painting.",    "Explain BM25 in one sentence.",    "Suggest a name for a cat.",    # Everything above has left the prompt window by now; only search can bring it back.    "When is my son's birthday, and what gift could I get her?",    "Which invoice was overdue, and from which company?",    "What operating system runs on my laptop?",]def main() -> None:    parser = argparse.ArgumentParser(description="LangGraph full-text search (BM25) memory demo")    parser.add_argument("-i", "--interactive", action="store_true", help="chat yourself instead of the scripted demo")    parser.add_argument("-m", "--model", help=f"Ollama model name (default: {OLLAMA_MODEL})")    parser.add_argument("-k", "--top-k", type=int, default=TOP_K, help="past messages to retrieve per turn")    args = parser.parse_args()    index = BM25Index()    graph = build_graph(make_llm(args.model), index, args.top_k)    config = {"configurable": {"thread_id": "demo"}}    def turn(text: str):        print(f"USER: {text}")        result = graph.invoke({"messages": [HumanMessage(text)]}, config)        reply = next(m for m in reversed(result["messages"]) if isinstance(m, AIMessage))        print(f"AI:   {reply.content}\n")    if args.interactive:        print("Type 'quit' to exit, 'index' to list the indexed messages.\n")        while (text := input("> ").strip()).lower() not in {"quit", "exit"}:            if text == "index":                print("\n".join(f"[{i}] {d}" for i, d in enumerate(index.docs)) or "(empty)", "\n")            elif text:                turn(text)    else:        for text in DEMO_TURNS:            turn(text)

What is semantic memory? It is long-term memory for facts, concepts, and knowledge. An agent retrieves this information when it needs it.

“Semantic” means “related to meaning.” Semantic search looks for similar meanings, not just matching words.

Example: A user has saved this message:

“I’m vegetarian, please don’t suggest chicken or fish.”

Later, the user asks:

“What food should I avoid recommending?”

· Keyword search: It looks for words such as “food,” “avoid,” and “recommending.” These words are not in the saved message, so the search can miss it.

· Semantic search: It looks for a similar meaning. It can connect “food to avoid” with “don’t suggest chicken or fish” and find the message.

How semantic search works

  1. Turn each message into a list of numbers called an embedding. Think of it as a point on a map of meanings.

  2. Messages with similar meanings are close on this map. For example, “car” and “automobile” are close, while “car” and “banana” are far apart.

  3. Turn the question into an embedding too. Then find the saved messages with the closest embeddings.

How are results saved and retrieved?

There are two stages: saving information first, then finding it when a question comes in.

Phase 1: Saving (before any question is asked)

  1. Create an embedding for each message. For example, the vegetarian message becomes a list of numbers like [0.9, 0.1, …]. These numbers are only an illustration.

  2. Save the embedding and the original text in a vector database. This database searches lists of numbers.

Here are some example entries. The numbers are for illustration only:

Text

Embedding

[0.9, 0.1, …]

“My office address is on MG Road.”

[0.1, 0.9, …]

“I usually order lunch around 1 pm.”

[0.4, 0.7, …]

Phase 2: Retrieving (when the user asks something)

  1. Create an embedding for the question. Use the same embedding model as before. For example, “What food should I avoid recommending?” becomes a list like [0.85, 0.15, …].

  2. Compare the embeddings. A measure called cosine similarity compares the directions of the number lists. Higher scores mean a closer match. Here are example scores, not results from a real search:

  3. Choose the best few matches. Sort the results by score and keep the top K, such as the top two. A minimum score can help leave out weak matches.

  4. Add useful results to the model’s input. For example:

Past messages:

Question: What food should I avoid recommending?

  1. The model answers using the saved details: “Avoid recommending meat or fish, since you’re vegetarian.”
"""Semantic memory with a vector store, using LangGraph and Ollama.Each thing the user tells the assistant is embedded (turned into a vector of numbersthat captures its meaning) and saved in LangGraph's long-term Store. On every turn thequestion is embedded too, and the memories whose vectors are closest in meaning (cosinesimilarity) are added to the prompt.Unlike the BM25 example, no shared keywords are needed: "a gift for my kid's special day"finds "my daughter's birthday is on 14 March". And unlike a checkpointer, the Store isshared across threads, so a brand-new conversation can still recall facts from an old one.    START -> recall -> chat -> remember -> END"""import argparseimport uuidfrom langchain_core.messages import AIMessage, AnyMessage, HumanMessage, SystemMessagefrom langchain_core.runnables import RunnableConfigfrom langchain_ollama import OllamaEmbeddingsfrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, MessagesState, StateGraphfrom langgraph.store.base import BaseStorefrom langgraph.store.memory import InMemoryStorefrom rolling_context import OLLAMA_MODEL, OLLAMA_URL, make_llmEMBED_MODEL = "nomic-embed-text"  # install with: ollama pull nomic-embed-textEMBED_DIMS = 768  # vector size produced by nomic-embed-textWINDOW = 4  # recent messages sent verbatimTOP_K = 3  # memories retrieved per turnMIN_SCORE = 0.45  # cosine similarity below this is treated as "not related"class State(MessagesState):    memories: list[str]def make_store(embed_model: str = EMBED_MODEL) -> InMemoryStore:    """A key-value store with a vector index on the 'text' field. Swap for PostgresStore in production."""    embeddings = OllamaEmbeddings(model=embed_model, base_url=OLLAMA_URL)    return InMemoryStore(index={"embed": embeddings, "dims": EMBED_DIMS, "fields": ["text"]})def build_graph(llm, store: BaseStore, top_k: int = TOP_K):    def namespace(config: RunnableConfig):        # Memories belong to a user, not to a thread, so every thread of that user can see them.        return ("memories", config["configurable"]["user_id"])    def recall(state: State, config: RunnableConfig, *, store: BaseStore):        query = state["messages"][-1].content        hits = store.search(namespace(config), query=query, limit=top_k)        for h in hits:            mark = "use " if h.score >= MIN_SCORE else "skip"            print(f"   {mark} {h.score:.3f}  {h.value['text']!r}")        if not hits:            print("   (memory is empty)")        return {"memories": [h.value["text"] for h in hits if h.score >= MIN_SCORE]}    def chat(state: State):        prompt: list[AnyMessage] = [SystemMessage("You are a helpful, concise assistant.")]        if state["memories"]:            facts = "\n".join(f"- {m}" for m in state["memories"])            prompt.append(                SystemMessage(                    "Things the user told you in earlier conversations (retrieved by meaning). "                    f"Use them when relevant:\n{facts}"                )            )        prompt += state["messages"][-WINDOW:]        print(f"   -> sending {len(prompt)} messages to the model")        return {"messages": [llm.invoke(prompt)]}    def remember(state: State, config: RunnableConfig, *, store: BaseStore):        text = state["messages"][-2].content  # the user message of this turn        # Simple heuristic: statements are worth remembering, questions usually aren't.        if not text.rstrip().endswith("?"):            store.put(namespace(config), str(uuid.uuid4()), {"text": text})  # embedded on put            print("   ** saved to semantic memory")        return {}    builder = StateGraph(State)    builder.add_node("recall", recall)    builder.add_node("chat", chat)    builder.add_node("remember", remember)    builder.add_edge(START, "recall")    builder.add_edge("recall", "chat")    builder.add_edge("chat", "remember")    builder.add_edge("remember", END)    # Checkpointer = short-term memory per thread; store = long-term memory per user.    return builder.compile(checkpointer=InMemorySaver(), store=store)# Two separate conversations (threads). The second starts with an empty history and# its questions share almost no words with the facts, so only semantic search can help.DEMO_SESSIONS = {    "monday": [        "Hi, I'm Deepak and I live in Noida.",        "I'm allergic to peanuts.",        "My daughter turns eight on 14 March and she adores painting.",        "I drive a red Tata Nexon electric car.",        "At work I'm building an AI agent with LangGraph.",    ],    "friday": [        "What should I get my kid for her special day?",        "Suggest a quick snack recipe that is safe for me to eat.",        "Any tips for a long road trip in my vehicle?",        "What framework am I using for my project at the office?",    ],}def main() -> None:    parser = argparse.ArgumentParser(description="LangGraph semantic memory (vector store) demo")    parser.add_argument("-i", "--interactive", action="store_true", help="chat yourself instead of the scripted demo")    parser.add_argument("-m", "--model", help=f"Ollama chat model (default: {OLLAMA_MODEL})")    parser.add_argument("-e", "--embed-model", default=EMBED_MODEL, help=f"Ollama embedding model (default: {EMBED_MODEL})")    parser.add_argument("-k", "--top-k", type=int, default=TOP_K, help="memories to retrieve per turn")    args = parser.parse_args()    store = make_store(args.embed_model)    graph = build_graph(make_llm(args.model), store, args.top_k)    user_id = "deepak"    def turn(text: str, thread_id: str):        print(f"USER: {text}")        config = {"configurable": {"thread_id": thread_id, "user_id": user_id}}        result = graph.invoke({"messages": [HumanMessage(text)]}, config)        reply = next(m for m in reversed(result["messages"]) if isinstance(m, AIMessage))        print(f"AI:   {reply.content}\n")    if args.interactive:        thread = 1        print("Type 'quit' to exit, 'new' to start a new thread (memory is kept), 'memories' to list them.\n")        while (text := input(f"[thread {thread}]> ").strip()).lower() not in {"quit", "exit"}:            if text == "new":                thread += 1                print(f"--- new thread {thread}: chat history is empty, semantic memory is not ---\n")            elif text == "memories":                items = store.search(("memories", user_id), limit=100)                print("\n".join(f"- {i.value['text']}" for i in items) or "(empty)", "\n")            elif text:                turn(text, f"thread-{thread}")    else:        for thread_id, turns in DEMO_SESSIONS.items():            print(f"=============== thread '{thread_id}' (new conversation) ===============\n")            for text in turns:                turn(text, thread_id)

Most organizations have large internal knowledge bases: past tickets, policies, product docs, troubleshooting guides. This is a gold mine of knowledge, but it’s hard to use in the moment. A customer service representative handling a live request often can’t find the relevant article or similar past case quickly enough to act on it.

The idea

What if an LLM could draw on that same knowledge base?

The solution

This is exactly what RAG makes possible. The knowledge base is indexed ahead of time (chunked, embedded, and stored in a vector database). At query time, the most relevant pieces are retrieved and handed to the LLM as context, so its answers are accurate, current, and specific to your organization.

Think of the diagram as a library getting ready before anyone walks in to ask a question. Reading left to right:

1. Source document

This is your raw knowledge: a help article, a policy, an old support ticket. It’s usually too long to use all at once.

2. Chunks

The dashed lines are scissors. The document gets cut into small, bite-sized pieces, each about a paragraph or two. Small pieces make it easier to find the exact part that answers a question, instead of handing over a whole 50-page manual.

3. Encoder

Each chunk goes through an encoder, a model that reads the text and captures what it means.

4. Dense vectors

The encoder turns each chunk into a list of numbers. Each little square in the strip stands for one number, and its shade shows how big the number is. You can think of this list as the chunk’s meaning fingerprint: chunks that say similar things get similar fingerprints, even if they use different words. “How do I reset my password” and “I forgot my login” end up close together.

5. Vector database

These fingerprints are stored as points in a big map. The gray dots are chunks that were already stored. The teal dots are the new ones just added. Chunks with similar meanings sit near each other on this map.

Why it matters

All of this happens once, ahead of time. Later, when a customer service rep asks something like “customer can’t log in after an update,” that question gets its own fingerprint. The system looks for the nearest points on the map and grabs those chunks. The LLM then uses them to write an answer based on your company’s actual knowledge.

Retrieval and generation

So far win RAG we have seen how a document is divided into chunks and saved in to vector database. Now lets looks at the retrieval and response generation part. The figure below shows once customer service use case.

Retrieval (steps 1–4): finding the right information

The rep asks: “Customer updated the app and can’t log in. What should I do?”

Generation (steps 5–6): writing the answer

How AI Agents Remember: A Simple Guide to Memory, Vector Search, and RAG was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #ai-agents 4 stories · sorted by recency
── more on @gpt-5 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-ai-agents-rememb…] indexed:0 read:20min 2026-11-08 · —