I work at Wontopos, so I'm biased. I build a memory API. But this post isn't about it. It's about a problem I keep reading about, and I want to hear how you deal with it.
An AI that remembers things will sooner or later remember something that is no longer true.
Here is a made-up example, just to show the shape of it:
| When | What the user says | What gets stored |
|---|---|---|
| March | "I live in Seoul." | lives in Seoul |
| July | "I moved to Busan last month." | lives in Busan |
| October | "Find me a good dentist nearby." | ??? |
Both memories are true statements. One of them stopped being true. If the memory layer only looks for text that is similar to the question, both come back, and the model has to guess which one wins. Sometimes it guesses wrong, and it sounds confident when it does.
That's the part that bothers me. A missing memory is easy to notice. An old memory that sounds right is not.
From what I've read, people seem to do one of these:
In our API, an outdated memory is replaced on purpose with a supersede call, so the old value is marked as replaced and not just buried. We also keep revisions, so you can see what changed. I don't want to oversell this. Deciding when to call it is still your job, and I'm still thinking about how to do that part better.
We also put a STALE axis in Glasshouse, an open memory benchmark I'm working on (still a draft), because I couldn't find an easy way to measure this. Details are in the repo if you want to look.
I'd like to hear from people who build or run their own memory:
You don't need to sign up for anything. The details are at wontopos, and the site has a short survey on the main page. A comment here is just as good, and the survey is a bonus.