Architecting AI Agent Memory: Short-Term vs. Long-Term A developer named Tanmay outlined a layered memory architecture for production AI agents that separates a bounded short-term context window from long-term persistent storage. The approach truncates message history to the N most recent turns while keeping a fixed system prompt, and stores extracted user facts in a ChromaDB vector store, retrieving only the top matching facts per query. The author argues that stuffing full transcripts into the prompt is an anti-pattern that inflates latency and token costs and degrades model adherence to system instructions. If your AI agent loses all state the moment a conversation ends, you haven't built an assistant—you've built a stateless chatbot. When developers first start building LLM applications, the standard approach to memory is appending every chat message into a growing array and passing it into the model on every turn. While this works for basic multi-turn conversations, it quickly breaks down in production. Context windows fill up fast. Latency increases with token count, API charges scale exponentially, and models suffer from context degradation, frequently ignoring system rules when buried under thousands of tokens of raw transcript. Trying to solve multi-session memory by stuffing more tokens into the prompt is an anti-pattern. To build production-grade agents that remember user preferences over days or months, you need a layered memory architecture: short-term context window management paired with long-term persistent storage. An effective agent memory system divides responsibility into two primary layers: Instead of sending the complete message payload on every turn, truncate the message array dynamically while preserving core system instructions. Here is a backend pattern to bound short-term context window size: python def build short term context system prompt: str, message history: list dict , max recent turns: int = 5 - list dict : """ Maintains a fixed system prompt while keeping only the N most recent user/assistant message turns. """ payload = {"role": "system", "content": system prompt} Calculate the number of raw messages to retain 2 messages per turn cutoff = max recent turns 2 recent history = message history -cutoff: if cutoff 0 else payload.extend recent history return payload As interactions occur, do not rely on raw transcripts for long-term recall. Parse incoming messages for key facts—such as technical preferences, location details, or recurring user tasks—and store them explicitly. Vector databases are ideal for unstructured memory retrieval. Below is a implementation showing how to write and retrieve context using Python and ChromaDB: python import chromadb Initialize persistent memory storage client = chromadb.PersistentClient path="./agent memory db" memory collection = client.get or create collection name="user knowledge" def store user fact user id: str, fact: str, fact id: str : """Saves an extracted user preference into the vector store.""" memory collection.upsert documents= fact , metadatas= {"user id": user id} , ids= f"{user id} {fact id}" def fetch relevant memories user id: str, current query: str, limit: int = 3 - str: """Queries long-term memory for facts matching the current query.""" results = memory collection.query query texts= current query , n results=limit, where={"user id": user id} documents = results.get "documents", 0 if not documents: return "" formatted facts = "\n".join f"- {doc}" for doc in documents return f"Retrieved User Context:\n{formatted facts}" Before making an inference call, query your persistent vector storage using the user's latest input. Inject only the top relevant facts into the system prompt alongside the truncated short-term message buffer. This gives your model access to critical preferences established weeks ago without cluttering the prompt window with unnecessary context. Relying on context window length as a replacement for real database architecture creates slow, expensive agents. By separating your runtime context from persistent storage, your agents gain long-term intelligence while keeping response latency and token costs minimal. Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect on LinkedIn https://www.linkedin.com/ .