Originally published on tamiz.pro.
You've spent weeks tuning your system prompt. You've tried chain-of-thought, ReAct, and tree-of-thought prompting. You've benchmarked GPT-4o against Claude 3.5 Sonnet and nothing clicks. Your agent still loses context, contradicts itself across turns, and feels like it's starting fresh every time a user comes back.
Here's the uncomfortable truth: your agent doesn't have a reasoning problem. It has a memory problem.
The models available today are remarkably capable at reasoning within the context window they're given. The gap between a "smart" agent and a "breaks after three turns" agent almost never traces back to the model's logical capabilities. It traces back to what the agent remembers, how it remembers it, and whether that memory survives the transition from demo to production.
This is a deep-dive into production-grade agent state — the architecture, patterns, and trade-offs that separate prototypes from systems that handle real users across real sessions.
Before we fix the memory problem, let's kill the reasoning narrative once and for all.
When we evaluate LLM reasoning, we're measuring something narrow: given a prompt and a constrained set of context tokens, how well does the model solve a defined problem? Chain-of-thought papers, MATH benchmarks, GPQA, LiveCodeBench — these are all stateless evaluations. The model sees the question, reasons through it, and produces an answer. Nothing persists. Nothing accumulates.
Real agent work is fundamentally different. An agent operates across multiple turns, multiple tools, and distributions of information that arrive incrementally. The reasoning challenge isn't "can the model think?" — it's "can the model think about what it already knows while also figuring out what to do next?"
Consider this conversation:
User (turn 1):"I need to book a flight from SFO to NYC next Tuesday for under $400."
Agent:Calls search tool → finds flights → returns results
User (turn 2):"Which one has the shortest layover?"
Agent:Calls another tool or reasons from prior results
User (turn 3):"Actually, change the destination to Newark."
Agent:...what does it know about the original request?
At turn 3, the agent needs to reconcile a modified goal against previous tool results, intermediate conclusions, and user preferences expressed across turns. That's not a reasoning deficiency — that's a state management deficiency. No amount of prompt engineering on the model's reasoning ability fixes this. The information simply isn't there to reason about.
You could throw every prior turn into the context window and hope the model tracks it. This fails in production for three reasons:
Context window is expensive. Every token you send costs money. A 10-turn conversation with tool results can easily consume 15,000–50,000 tokens per call. At scale, this is bankrupting.
Context window is noisy. Retrieving 40K tokens of conversation history doesn't mean the model attends to the right 40K tokens. Attention mechanisms dilute across long contexts. Critical details from turn 1 get buried.
Context window doesn't persist across sessions. When the user returns tomorrow, last week's conversation is gone unless you explicitly saved it somewhere and retrieved it again.
The solution isn't bigger windows. It's better memory architecture.
Agent state is not a single thing. It's a composite of several distinct but interacting layers. Confusing these layers is the root cause of most production failures.
The short-lived, session-bound state that drives the current interaction. This includes:
This is typically held in the context window and refreshed every turn. It's fast, flexible, and ephemeral.
Longer-lived knowledge about the world that the agent needs to reference repeatedly:
This lives outside the context window, usually in a database or vector store, and is retrieved on demand.
The agent's repertoire of actions and their outcomes:
This is typically encoded in function/tool definitions but should also include learned heuristics that improve over time.
Where each piece of information came from — critical for trust and debugging:
Without source memory, agents confidently hallucinate details they "remembered" but can't verify. This is the difference between an agent that says "Based on your profile..." and one that says "You told me on March 3rd that you prefer..."
A production-grade agent memory system has four layers, each serving a different latency and persistence profile.
The active working memory that gets injected into every LLM call. This is not raw conversation history — it's a curated buffer that the agent maintains and updates.
type ContextBuffer = {
// Current session state
sessionId: string;
turnCount: number;
// Active goal stack
currentGoal: GoalState;
subgoals: Subgoal[];
completedSubgoals: Subgoal[];
// Tool call ledger (compact, not raw logs)
toolLedger: ToolEvent[];
// Extracted facts for this turn
extractedFacts: Fact[];
// Conversation summary (rolling, for when we trim)
rollingSummary: string;
};
The key insight: the agent writes to this buffer as it works, not just receives it. After each tool call, the agent should update the buffer with structured results, not just dump raw output into the context.
Semantic and procedural memory lives here. When the agent needs to recall something beyond its current context, it queries this layer.
interface MemoryRetriever {
// Semantic recall — find relevant facts
recallSemantics(query: string, userId: string): Promise<Fact[]>;
// Episodic recall — find similar past interactions
recallEpisodes(similarTo: string, limit: number): Promise<Episode[]>;
// Procedural recall — find relevant tools/patterns
recallProcedures(taskType: string): Promise<ToolDefinition[]>;
}
Implementation options:
Long-term storage with lifecycle management. This is where memories go to survive, age, and eventually get pruned.
interface MemoryStore {
// Write with TTL and priority
write(entry: MemoryEntry): Promise<void>;
// Compaction — merge redundant memories
compact(userId: string): Promise<number>;
// Expiry — remove stale memories
expire(maxAgeDays: number): Promise<number>;
// Audit trail for provenance
getProvenance(memoryId: string): Promise<ProvenanceRecord>;
}
interface MemoryEntry {
id: string;
type: 'fact' | 'preference' | 'event' | 'procedure';
content: string;
metadata: Record<string, unknown>;
source: SourceReference;
createdAt: Date;
expiresAt?: Date;
confidence: number; // 0–1, for uncertain memories
accessCount: number; // for recency-based eviction
}
This is the layer everyone skips. Memory without forgetting is a liability. Stale preferences, outdated goals, and redundant facts crowd out signal. A production system needs explicit forgetting:
How does all of this work in practice? Here's the production-grade memory loop that runs on every agent turn:
┌─────────────────────────────────────────────────┐
│ TURN START │
│ │
│ 1. USER INPUT arrives │
│ ↓ │
│ 2. QUERY RETRIEVAL │
│ ┌─→ Semantic recall (facts, preferences) │
│ ├─→ Episodic recall (similar past turns) │
│ └─→ Procedural recall (relevant tools) │
│ ↓ │
│ 3. CONTEXT COMPOSING │
│ ┌─→ Current working memory (buffer) │
│ ├─→ Retrieved memories │
│ ├─→ Rolling summary (compressed history) │
│ └─→ System prompt + tool definitions │
│ ↓ │
│ 4. LLM CALL │
│ (agent reasons over composed context) │
│ ↓ │
│ 5. ACTION EXECUTION │
│ Tool calls → results → validate │
│ ↓ │
│ 6. MEMORY UPDATE │
│ ┌─→ Write new facts to persistence │
│ ├─→ Update working buffer │
│ ├─→ Compact/expire stale memories │
│ └─→ Update source provenance │
│ ↓ │
│ 7. RESPONSE │
│ Format and return to user │
└─────────────────────────────────────────────────┘
Step 2 — Query Retrieval: Before the LLM sees anything, the system fetches relevant memories. The query for retrieval isn't the user's raw input — it's a meta-query generated from the user input plus current goals. This two-step process (generate query → retrieve memories) prevents irrelevant memories from polluting the retrieval.
async function buildRetrievalQuery(
userInput: string,
workingMemory: ContextBuffer
): Promise<string> {
// Use a lightweight model to generate a focused retrieval query
const queryGen = await llm.call({
model: 'fast-model', // cheaper, faster model for query generation
messages: [
{ role: 'system', content: RETRIEVAL_QUERY_SYSTEM_PROMPT },
{ role: 'user', content: `Goal: ${workingMemory.currentGoal.description}\nInput: ${userInput}` }
]
});
return queryGen.content;
}
Step 3 — Context Composing: This is where most agents fail. The context isn't just "concatenate everything." It's a structured assembly with priorities:
The rolling summary is critical. Instead of keeping raw conversation history, periodically (every 5–10 turns) compress the history into a concise summary using the LLM itself:
async function compressHistory(
rawHistory: Message[],
workingMemory: ContextBuffer
): Promise<string> {
const summary = await llm.call({
model: 'summary-model',
messages: [
{ role: 'system', content: COMPRESS_SYSTEM_PROMPT },
{ role: 'user', content: formatMessages(rawHistory) }
],
maxTokens: 500
});
return summary.content;
}
Step 6 — Memory Update: After the agent acts, the system extracts and persists new knowledge:
async function updateMemories(
turnResult: AgentTurnResult,
workingMemory: ContextBuffer,
memoryStore: MemoryStore
): Promise<void> {
// Extract new facts from the turn
const newFacts = await extractFacts(turnResult);
for (const fact of newFacts) {
await memoryStore.write({
id: crypto.randomUUID(),
type: classifyFact(fact),
content: fact.text,
metadata: fact.metadata,
source: { type: 'extraction', confidence: fact.confidence },
createdAt: new Date(),
confidence: fact.confidence
});
}
// Update working buffer
workingMemory.extractedFacts.push(...newFacts);
workingMemory.turnCount++;
// Periodic compaction
if (workingMemory.turnCount % 10 === 0) {
await memoryStore.compact(workingMemory.sessionId);
}
}
The hardest part of agent memory isn't storage — it's knowing what to store. Raw conversation is noisy. You can't persist everything. You need to extract signal from the noise.
| Signal Type | Example | Priority |
|---|---|---|
| User preferences | "I prefer morning flights" | High |
| Explicit facts | "My order number is ORD-12345" | High |
| Goal states | "Looking for a refund, not an exchange" | Medium |
| Tool outcomes | "Flight search returned 3 results under $400" | Medium |
| Contextual hints | Conversation tone, urgency signals | Low |
| Chatter | "Thanks!" "Got it" "No wait" | None |
Raw Turn Output
↓
┌──────────────┐
│ Classifier │ Is this worth remembering?
└──────┬───────┘
↓ yes
┌──────────────┐
│ Extractor │ Pull out structured facts
└──────┬───────┘
↓
┌──────────────┐
│ Verifier │ Check against existing memories
│ (dedup) │ Avoid storing "SFO to NYC" when
└──────┬───────┘ "SFO to New York" already exists
↓
┌──────────────┐
│ Prioritizer │ Assign confidence, TTL, type
└──────┬───────┘
↓
┌──────────────┐
│ Persister │ Write to memory store
└──────────────┘
js
const EXTRACTION_PROMPT = `
You are extracting facts from a conversation between a user and an AI agent.
Extract ONLY the following types of information:
1. User preferences (travel, dietary, scheduling, etc.)
2. Explicitly stated facts (order numbers, dates, names)
3. Active goals and their status
4. Tool results that constrain future decisions
DO NOT extract:
- Casual conversation fillers
- Requests the agent already fulfilled in this turn
- Information already stored in existing memories (you'll be provided them)
Existing memories for this user:
{{existingMemories}}
Conversation:
{{conversation}}
Return a JSON array of facts. Each fact: {type, content, confidence (0-1), expiresAt (null if permanent)}.
`;
An agent that can't cite its sources is an agent that will confidently hallucinate. Production systems need provenance tracking — every memory must be traceable to its origin.
Self-correction: When an agent says "Based on your previous request...", it should be able to point to which previous request. If the user corrects it, the system knows what to update.
Debugging: When an agent makes a wrong decision, you need to know whether it reasoned poorly or remembered poorly. These are different bugs requiring different fixes.
User trust: Users can detect when an agent is making things up. A system that says "You mentioned this on March 3rd" vs "I think you might have said..." builds different levels of trust.
interface ProvenanceRecord {
memoryId: string;
sourceType: 'user-stated' | 'tool-result' | 'inferred' | 'system-injected';
sourceId: string; // references the original message, tool call, or system event
timestamp: Date;
confidence: number;
overwrittenBy?: string; // if this memory was superseded
}
When the agent retrieves a memory, the provenance record should be part of the retrieved context, not hidden metadata. The agent needs to know how it knows something, so it can express appropriate certainty.
Even with good extraction and retrieval, you'll hit context limits. Here's how production systems handle it.
Instead of truncating conversation history at the beginning (losing the oldest, potentially critical context), use summary anchors:
[Summary of turns 1–12: "User booked flight SFO→NYC, order ORD-999. Prefers aisle seats."]
[Summary of turns 13–24: "User requested change to Newark. Refund initiated."]
[Turn 25: ...]
[Turn 26: ...]
[Turn 27: ...]
The summaries preserve the meaning of earlier turns without the token cost. You can implement this with periodic summarization triggered by turn count or context budget.
interface ContextBudget {
totalTokens: number; // e.g., 128_000 for GPT-4o
reservedForSystem: number; // ~2,000 — prompt, tool defs
reservedForTools: number; // ~8,000 — relevant tool results
reservedForSummary: number; // ~1,000 — rolling summary
budgetForRetrieval: number; // remaining — dynamic based on retrieval score
}
Every turn, the context composer checks the budget and decides: how many retrieved memories can I include? If the budget is tight, only include memories above a relevance threshold.
The hardest case: the user returns days later. The working memory from last time is gone. How does the agent reconnect?
interface SessionLink {
currentSessionId: string;
previousSessionId: string;
linkReason: 'same_user' | 'similar_goal' | 'referenced_context';
bridgeSummary: string; // "Last time, we were booking a flight to NYC..."
linkedAt: Date;
}
When a new session starts, the system checks for potential links:
The bridge summary is the single most important artifact for cross-session continuity. It's a 2–3 sentence condensation of what happened in the previous session, generated at session close:
"Last session: You were booking a flight from SFO to NYC for next Tuesday.
We found three options under $400. You were deciding between the United
red-eye and the Delta morning flight. The conversation ended before a
selection was made."
This bridge gets injected into the first turn of the next session, giving the agent immediate continuity without reconstructing the entire conversation.
When the agent's own outputs become part of its memory, it creates feedback loops. It "remembers" things it generated rather than things the user stated. Mitigation: source tagging. Every memory entry must carry a sourceType
— and the agent's own reasoning outputs should never be written to long-term memory without user confirmation.
Agents that remember everything remember nothing useful. Without compaction and forgetting, the retrieval index becomes noisy and relevant memories get drowned out. Mitigation: TTL-based eviction and confidence-weighted pruning.
When the agent assumes continuity where none exists — mixing up conversations, attributing statements to the wrong session. Mitigation: session-scoped memory by default, with explicit cross-session linking only when verified.
When the extractor misses important information because it's embedded in an indirect statement. "I'd prefer not to fly United" is a preference, but a naive extractor might miss it among the conversational noise. Mitigation: multi-pass extraction — first pass for explicit facts, second pass for implicit preferences, with different prompts for each.
Wrap your agent framework with a memory middleware layer that intercepts every turn:
class AgentWithMemory {
constructor(
private agent: BaseAgent,
private memoryStore: MemoryStore,
private retriever: MemoryRetriever,
private contextBuffer: ContextBuffer
) {}
async execute(userInput: string, sessionId: string): Promise<AgentResponse> {
// 1. Retrieve relevant memories
const retrievalQuery = await this.buildRetrievalQuery(userInput);
const memories = await this.retriever.recall(retrievalQuery, sessionId);
// 2. Compose enriched context
const enrichedContext = this.composeContext(
this.contextBuffer, memories, userInput
);
// 3. Execute agent
const response = await this.agent.execute(enrichedContext);
// 4. Update memories
await this.updateMemories(response, sessionId);
return response;
}
}
| Framework | Memory Approach | Notes |
|---|---|---|
| LangGraph | ||
| StateGraph with persistent state | Best for complex multi-step agents; use Checkpointers for persistence |
|
| LangChain | ||
| ConversationBufferMemory, VectorStoreRetrieverMemory | Good primitives but requires manual composition | |
| LlamaIndex | ||
| QueryEngine + ChatEngine with memory | Strong retrieval integration, weaker on state management | |
| CrewAI | ||
| Agent memory via shared context | Simpler but less control over memory lifecycle | |
| Custom (recommended for prod) | ||
| Hand-rolled middleware | Full control over every layer described above |
For production systems, the trend is toward custom middleware rather than out-of-the-box memory solutions. The frameworks provide building blocks, but the architecture described here — with explicit layers, provenance, compaction, and cross-session linking — requires orchestration that no framework provides today.
You can't improve what you don't measure. Track these metrics:
Q: Can I just use a vector database and call it memory?
No. Vector databases give you retrieval, not memory. Memory requires extraction, provenance tracking, compaction, TTL management, and cross-session linking — none of which a vector store provides by default. A vector database is one component of a memory system, not the system itself.
Q: How much memory should I pre-fetch vs. on-demand?
Pre-fetch semantic memories (user preferences, domain facts) for every turn — the latency is acceptable and the relevance is high. Fetch episodic memories (past conversations) on-demand based on the retrieval query. Don't pre-fetch procedural memories (tool definitions) — load only the tools relevant to the current goal to save context tokens.
Q: What's the minimum viable memory system for a production agent?
Three things: (1) a structured context buffer that the agent writes to each turn, (2) a retrieval layer for semantic memories (user preferences + domain facts), and (3) a rolling summary mechanism to handle context window limits. Get these right before adding cross-session linking, provenance tracking, or compaction. Those are optimizations, not foundations.
The models have the reasoning. The missing piece is always the memory. Build the memory system, and the reasoning problems mostly solve themselves.