Stop Dumping Chat History into Context

wpnews.pro

Every software engineer building autonomous agents eventually runs straight into the Memory Paradox.

In the first two turns of a conversation, everything feels magical. You ask the agent to analyze an API, you tell it your coding preferences, and it follows along flawlessly.

Then turn fifteen arrives.

Suddenly, you notice your latency creeping from 800 milliseconds to four seconds. Your cloud inference bill triples. And when you look at the raw payload sent to your model provider, you realize what happened: you are dumping sixty messages of raw chat history into every single API call.

To fix it, engineers usually try one of two naive shortcuts:

The Truncation Hack: You slice the history down to the last ten messages. The bill drops, but your agent instantly develops digital amnesia. It forgets the project constraints you established five minutes ago and asks you for the same credentials twice.
The Naive RAG Dump: You set up a vector database, query it on every turn, and paste the retrieved memories straight into the system prompt.

The second shortcut seems clever until the invoice lands at the end of the month.

Because modern LLM providers cache prompts from the front, changing the text inside your system prompt on every turn completely destroys your prompt prefix cache. In a twenty-five-turn session, you pay 100% full, un-cached token prices on every single message.

Last week, the team at Weaviate published an exceptional teardown of their internal memory engine, Engram. Beneath the product details lies a masterclass in production systems architecture.

Here is the universal architecture of production agent memory, the four configuration decisions that govern it, and how to structure prompts to keep your cache hit rate above 95%.

1. How Memory Actually Works: The Asynchronous Loop #

The first mistake teams make is trying to extract memories synchronously during the user request.

If your user asks a question, your application should not wait for an LLM to read the message, extract facts, update a database, and only then generate an answer. That adds two to three seconds of unnecessary blocking latency.

Production memory operates as an asynchronous background loop:

[ User Sends Message ]
         │
         ├──► [ Real-Time Path: Instant LLM Response ]
         │
         └──► [ Async Memory Queue ]
                     │
                     ▼
              1. Extract (Topic-Guided Classification)
                     │
                     ▼
              2. Reconcile (In-Place Update & Deduplication)
                     │
                     ▼
              3. Commit (Partitioned Vector / Document Store)

When a user speaks, the real-time request immediately proceeds using current context. In the background, the conversation payload enters an asynchronous extraction pipeline that runs three distinct phases:

Extract: An extraction model evaluates the message against predefined semantic topics, isolating permanent facts from conversational noise.
Reconcile (Transform): The pipeline compares newly extracted facts against existing memory state. If a user previously stated:“I use Python 3.11” and now says:“We migrated to Python 3.12” , the system rewrites the memory in place rather than creating contradictory duplicates.
Commit: The updated state is committed to a durable, partitioned vector store.

2. The Four Rules of Memory Configuration #

Turning raw messages into reliable memory requires making four concrete architectural decisions:

Decision 1: Negative Extraction Constraints

The fastest way to ruin an agent’s memory store is giving the extraction model broad, open-ended instructions like “Record all useful facts about the user.”

If a user says:

“Two hours lost to a Docker rebuild, my headphones died mid-call, and it started raining right as I stepped out. Anyway, let’s deploy the new Qwen embedding model.”

A naive extraction prompt will store four facts: the user hates Docker, their headphones broke, it rained on Thursday, and they deployed Qwen. Two weeks later, the agent is cluttering its context window with the weather from last Thursday.

The fix is adding negative extraction constraints. Defining what to ignore does significantly more work than listing what to include:

Topic: UserDecisions
Description: Record technical decisions, architectural preferences, and enduring constraints.
Constraint: DO NOT record transient events, equipment failures, weather, or temporary mood states.

In production testing, a single negative constraint eliminates over 80% of memory pollution across multi-session agents.

Decision 2: Bounded vs. Unbounded Topics (Cardinality Control)

Not all memories behave the same way:

Unbounded Topics: Designed to accumulate multiple discrete facts over time (e.g. project requirements, bug discoveries, API endpoints).
Bounded Topics: Guaranteed to hold at mostone single memory per scope.

For example, a UserProfile or a ConversationSummary must always be a bounded topic. When a new summary is generated, it does not append another row to your database; it supersedes the previous summary on the exact same primary key.

If your application needs to retrieve a single standing truth, you must enforce a bounded constraint at the database layer rather than trusting the LLM to count.

Decision 3: Scope Partitioning

Memories must never sit in one giant global bucket. In enterprise production, cross-tenant leaks are unacceptable.

You need two distinct scoping boundaries:

User Scope (Hard Wall): Every read and write requires a verifieduser_id ortenant_id . No query can ever cross this boundary.
Property Scope (Flexible Partition): Optional sub-keys likeconversation_id ,repo_name , orproject_id . A write requires the property key, while a read can optionally query across all conversations belonging to that user.

3. The Dual-Tier Architecture & The Cache Boundary #

The most critical insight in agent memory design is where memories sit inside your prompt.

Most LLM providers (including OpenAI, Anthropic, and Google) implement prefix prompt caching. If request B begins with the exact same prefix tokens as request A, the shared prefix is read from cache at up to a 90% discount with sub-second latency.

The moment you alter a single character in that prefix, the cache breaks, and you are billed in full for everything that follows.

To achieve 95%+ cache hit rates, production systems decouple memory into two distinct tiers:

Tier 1: Always-On Bounded Memory (Front of Prompt)

Certain facts are invariant across the entire session: user identity, language preferences, and global coding rules.

Instead of searching for these on every turn, fetch them once when the session initializes. Place them directly after your system prompt:

[SYSTEM PROMPT: Core persona & safety guardrails]
[STATIC USER PROFILE: Fetched once at session startup]
<--- PROMPT CACHE BREAKPOINT --->

Because this block never changes during the conversation, it remains 100% cached from the second turn onward.

Tier 2: Dynamic Per-Turn Memory (Back of Prompt)

Dynamic memories (context retrieved specifically for the current question) change on every turn.

If you paste these search results at the beginning of your prompt, you destroy your cache. Instead, place retrieved dynamic memories at the very end of the prompt, immediately following the user’s latest message:

[SYSTEM PROMPT] (Cached)
[STATIC USER PROFILE] (Cached)
[CHAT HISTORY: Turn 1 to N-1] (Cached)
[LATEST USER MESSAGE]
<--- PROMPT CACHE BREAKPOINT --->
[DYNAMIC RETRIEVED MEMORIES: Specific to latest message] (Un-cached tail)

In a 3,500-token prompt across a 25-turn conversation, this layout ensures that roughly 3,400 tokens are served directly from cache. You pay full un-cached prices for only the ~100 tokens representing the new query and its specific memory block.

4. The 3 Production Scars: Where Memory Fails #

When deploying memory systems at enterprise scale, watch out for these three failure modes:

Scar #1: Memory Hallucination Feedback Loops

If an agent hallucinates a false premise during a complex task (e.g. “We decided to deprecate MySQL”), and that statement gets extracted into memory, the hallucination becomes permanent. On future turns, the agent retrieves its own past hallucination as ground truth.

The Guardrail: Never extract memories from unverified assistant replies. Extract memories primarily from explicit user inputs, or require an explicit user confirmation before committing architectural decisions to long-term memory.

Scar #2: Query-Relevance Drift

Using standard cosine similarity on user questions often retrieves memories that share keywords but have zero relevance to the active task.

The Guardrail: Implementhybrid search (BM25 keyword matching combined with dense vector embeddings) and apply a recency decay factor so that decisions made yesterday rank higher than decisions made six months ago.

Scar #3: PII and Compliance Retention

GDPR and enterprise security policies require the right to be forgotten. If an agent records personal customer data into unstructured vector embeddings, finding and expunging that data becomes an operational nightmare.

The Guardrail: Every memory record must store a strictcreated_at timestamp, asource_message_id , and a cryptographic hash of the user identity. When a user requests data deletion, a single cascade delete removes all associated memory vectors.

5. How to Prototype This Weekend (Python Blueprint) #

You can implement this dual-tier layout in standard Python using any vector database or memory client:

from typing import List, Dict

class DualTierMemoryAgent:
    def __init__(self, user_id: str, memory_client, llm_client):
        self.user_id = user_id
        self.memories = memory_client
        self.llm = llm_client
        
        self.static_profile = self.memories.fetch_bounded(
            topic="UserProfile", user_id=self.user_id
        )
        self.history: List[Dict[str, str]] = []

    def run_turn(self, user_query: str) -> str:
        dynamic_memories = self.memories.search(
            query=user_query, user_id=self.user_id, limit=3
        )
        
        messages = [
            {"role": "system", "content": "You are a senior engineering assistant."},
            {"role": "system", "content": f"Static User Context:\n{self.static_profile}"},
            *self.history,
            {"role": "user", "content": user_query},
            {"role": "system", "content": f"Dynamic Context:\n{dynamic_memories}"}
        ]
        
        response = self.llm.chat(messages=messages)
        
        self.history.append({"role": "user", "content": user_query})
        self.history.append({"role": "assistant", "content": response})
        
        self.memories.async_extract_and_commit(user_query, self.user_id)
        
        return response

6. The Strategic Bottom Line #

For technical leaders and engineering directors, long-term memory is not about making chatbots feel conversational.

It is about unit economics and task completion rates.

Brute-forcing context windows with hundred-message histories is an operational dead end: it introduces unacceptable p99 latency spikes, invites distraction from irrelevant tokens, and inflates cloud costs.

By treating memory as a dual-tier systems architecture (asynchronous extraction, bounded cardinality, and cache-conscious prompt placement), you give your agents permanent recall while cutting token expenditure by up to 90%.

The most effective agents are not the ones that read the most tokens. They are the ones that know exactly what to forget.