I ran into a memory problem while building a reflective AI product.
The easy version of AI memory is tempting:
Save useful facts about the user.
Put them back into the next prompt.
Call it memory.
That can work for low-risk personalization.
It is probably fine if an assistant remembers that a repo uses pnpm, or that a user prefers short answers, or that a team calls its staging branch preview
.
But the problem changes when the user is bringing personal material into the product.
In my case, the product lets people explore dreams, moods, relationship patterns, recurring symbols, and private reflections. I did not want meaningful sessions to disappear into raw chat history. But I also did not want every sentence the user typed to quietly become permanent memory.
That changed the architecture for me.
The question stopped being:
What can the model remember?
It became:
What does the user own?
What did they approve?
When is it stored?
When is it used?
When can it be d, exported, or deleted?
That distinction sounds like product design, but it quickly becomes system design.
Many AI products describe memory as if it were one thing:
Memory: on/off
That is simple, but it hides too much.
In practice, "memory" usually mixes several different jobs:
If all of that becomes one invisible bucket, the developer gets a simpler implementation and the user gets a murkier product.
You also lose the ability to answer basic questions:
Those are not only UX questions. They are architecture questions.
The biggest architectural shift for me was separating stored memory from prompt-time memory.
A user may own saved memory assets, but that does not mean the model should always be allowed to use them.
I ended up thinking about memory in two different states:
This belongs to the user.
This is currently allowed to enter the prompt.
Those are not the same promise.
A user should be able to view, export, or delete saved memory without the assistant automatically using it in every future session.
That separation led to a small but important access layer:
type MemoryAccessState = {
userMemoryEnabled: boolean;
planKey: string;
entitlementState: string;
subscriptionActive: boolean;
canUsePromptMemory: boolean;
pendingMemoryActivation: boolean;
memoryExpiresAt: string | null;
};
function getCanUsePromptMemory(state: MemoryAccessState) {
return state.userMemoryEnabled && state.subscriptionActive;
}
The important field is not userMemoryEnabled
.
It is canUsePromptMemory
.
That field answers the real runtime question: should memory be included in the prompt for this turn?
This avoids a confusing product promise.
For example:
Without this separation, memory becomes a hidden privilege state. The data exists, the user sees some of it, the model may or may not use it, and nobody can explain the boundary clearly.
I stopped treating memory as one feature and started treating it as a set of product states.
The simplified shape looks like this:
conversation
-> session note
-> user-approved memory item
-> user-authored room context
-> long-term inner map
-> retrieval evidence
-> prompt context
Each layer has a different lifecycle.
conversation
is the raw exchange. It is useful for same-session continuity, but too noisy to become long-term memory by itself.
session note
is a structured artifact created after a session. It can summarize themes, symbols, conflicts, and useful continuity points.
memory item
is smaller and more explicit. It is the kind of thing the user can inspect and approve.
room context
is user-authored background. I treat this as higher-trust than inferred memory because the user wrote it directly.
inner map
is a versioned long-term snapshot of recurring themes. This layer needs caution because it can easily sound more authoritative than it is.
retrieval evidence
is what the system selected for the current turn.
prompt context
is the final compiled memory that the model sees.
This adds work, but it makes the system easier to reason about.
The user can own assets without every asset being active in the prompt. The assistant can use relevant memory without pretending all saved history is equally important. The product can , retain, export, or delete different layers without inventing a new exception every time.
Once memory is layered, retrieval should also become more explicit.
A memory result should not just be text pasted into a prompt. It should carry enough metadata to explain why it was selected.
A simplified version:
type MemoryEvidence = {
source: "session-note" | "memory-item";
id: string;
title: string;
excerpt: string;
score: number;
reason: string;
createdAt?: string;
};
The reason
field is not fancy, but it is useful.
It can say things like:
semantic long-memory match
strong theme/symbol overlap
partial approved-memory overlap
recent continuity evidence
This helps debugging, but it also keeps the product honest.
If the assistant brings a memory into the current turn, the system should be able to explain why that memory was selected.
For a first version, retrieval does not need to be magical. I prefer a conservative hybrid approach:
user-authored context
+ latest long-term snapshot
+ relevant session notes
+ recent notes as fallback
+ approved memory items
-> compact prompt memory
Vector search is useful, but it should not be the only rule.
For personal or reflective products, recency, explicit approval, user-authored context, and clear fallback behavior matter just as much as similarity score.
Here is the kind of behavior I wanted to avoid.
Suppose a user once wrote:
I keep dreaming about a locked garden gate.
Two weeks later they write:
I felt ashamed today when I wanted to ask for help.
Bad memory behavior:
This connects to your locked garden gate dream.
Maybe it does. Maybe it does not.
The callback may be clever, but if the user did not invite that connection, and the system cannot explain why it surfaced the memory, it can feel creepy.
Better behavior:
There may be a threshold theme here, but I would not force it.
If it feels connected, we can compare today's shame with the earlier gate image.
If not, we can stay with what happened today.
The wording is different, but the real difference is architectural.
The system needs to know:
If those answers only exist inside the model response, the product is too soft in the wrong place.
The memory boundary needs to exist outside the prompt.
Another mistake is treating retention as a legal/privacy page problem only.
For memory-heavy products, retention is part of the product behavior.
I prefer making the rules boring and explicit:
Active subscription:
memory can accumulate
memory can enter prompts if enabled
d or interrupted subscription:
memory assets are retained for a defined period
memory does not enter prompts
Memory disabled:
existing assets remain manageable
new long-term memory writes stop
Deletion:
individual memory items can be deleted
account-owned data can be exported or removed
The important part is not the exact number of days. The important part is that the product does not blur ownership, access, and usage.
A user should not have to guess whether saved memory is currently active.
They should not have to delete everything just to stop the assistant from using it.
For sensitive or reflective AI products, I would start with these guardrails:
None of this requires a complicated agent framework.
It requires treating memory as product state instead of prompt decoration.
This pattern is too heavy for some products.
I would not build all of this for:
It becomes worth it when:
That last point matters.
If memory is part of what people pay for, it cannot just be a prompt trick.
I built this pattern while working on Jung Room, a non-clinical AI self-exploration room for dreams, moods, symbols, and recurring patterns.
The product-specific version has:
session notes
+ saved memory items
+ user-authored room context
+ inner-map snapshots
+ account memory controls
+ subscription-gated prompt-time memory
+ retention and deletion paths
The general lesson is broader than this product:
Memory should be inspectable before it becomes powerful.
If you are adding memory to an AI product, these are the questions I would ask early:
If those answers are unclear, the memory feature may work technically while still feeling untrustworthy.
How are you handling memory in your own AI apps?
Are you treating it as private prompt context, user-owned product state, retrieval evidence, or something else entirely?