# Architecting AI Agent Memory: Short-Term vs. Long-Term

> Source: <https://dev.to/tanmay2627/architecting-ai-agent-memory-short-term-vs-long-term-1c3a>
> Published: 2026-11-22 22:33:26+00:00

**Recommendation:** Truncate message history to the most recent turns while pinning the system prompt, and inject only the top matching facts from a persistent vector store before each inference call.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

If your AI agent loses all state the moment a conversation ends, you haven't built an assistant—you've built a stateless chatbot.

When developers first start building LLM applications, the standard approach to memory is appending every chat message into a growing array and passing it into the model on every turn. While this works for basic multi-turn conversations, it quickly breaks down in production.

Context windows fill up fast. Latency increases with token count, API charges scale exponentially, and models suffer from context degradation, frequently ignoring system rules when buried under thousands of tokens of raw transcript. Trying to solve multi-session memory by stuffing more tokens into the prompt is an anti-pattern.

To build production-grade agents that remember user preferences over days or months, you need a layered memory architecture: short-term context window management paired with long-term persistent storage.

An effective agent memory system divides responsibility into two primary layers:

Instead of sending the complete message payload on every turn, truncate the message array dynamically while preserving core system instructions.

Here is a backend pattern to bound short-term context window size:

``` python
def build_short_term_context(
    system_prompt: str,
    message_history: list[dict],
    max_recent_turns: int = 5
) -> list[dict]:
    """
    Maintains a fixed system prompt while keeping only the N 
    most recent user/assistant message turns.
    """
    payload = [{"role": "system", "content": system_prompt}]

    # Calculate the number of raw messages to retain (2 messages per turn)
    cutoff = max_recent_turns * 2
    recent_history = message_history[-cutoff:] if cutoff > 0 else []

    payload.extend(recent_history)
    return payload
```

As interactions occur, do not rely on raw transcripts for long-term recall. Parse incoming messages for key facts—such as technical preferences, location details, or recurring user tasks—and store them explicitly.

Vector databases are ideal for unstructured memory retrieval. Below is a implementation showing how to write and retrieve context using Python and ChromaDB:

``` python
import chromadb

# Initialize persistent memory storage
client = chromadb.PersistentClient(path="./agent_memory_db")
memory_collection = client.get_or_create_collection(name="user_knowledge")

def store_user_fact(user_id: str, fact: str, fact_id: str):
    """Saves an extracted user preference into the vector store."""
    memory_collection.upsert(
        documents=[fact],
        metadatas=[{"user_id": user_id}],
        ids=[f"{user_id}_{fact_id}"]
    )

def fetch_relevant_memories(user_id: str, current_query: str, limit: int = 3) -> str:
    """Queries long-term memory for facts matching the current query."""
    results = memory_collection.query(
        query_texts=[current_query],
        n_results=limit,
        where={"user_id": user_id}
    )

    documents = results.get("documents", [[]])[0]
    if not documents:
        return ""

    formatted_facts = "\n".join([f"- {doc}" for doc in documents])
    return f"Retrieved User Context:\n{formatted_facts}"
```

Before making an inference call, query your persistent vector storage using the user's latest input. Inject only the top relevant facts into the system prompt alongside the truncated short-term message buffer.

This gives your model access to critical preferences established weeks ago without cluttering the prompt window with unnecessary context.

Relying on context window length as a replacement for real database architecture creates slow, expensive agents. By separating your runtime context from persistent storage, your agents gain long-term intelligence while keeping response latency and token costs minimal.

Tanmay builds AI agents and automation for businesses and is open to freelance projects. Feel free to connect on [LinkedIn](https://www.linkedin.com/).
