A few weeks ago I was testing a support-agent setup — nothing fancy, just
an LLM with a memory layer bolted on so it could remember basic facts
about a user across sessions. Subscription tier, shipping address, that
kind of thing.
I ran a simple scenario: the user is already on the enterprise plan. I
confirmed the memory retrieval was working — the fact ** subscription_tier:** came back correctly when I queried "what tier is the user's
Then I asked the agent, in a support-chat style prompt, what plan the user
was on.
The response:
"Sure, upgrading to our enterprise plan would unlock that feature for
you."
The user is already on enterprise. The agent had the correct fact sitting
right there in its context. It just... used it wrong. Not "forgot it" —
that's a different, more talked-about failure mode. This one is worse in a
specific way: retrieval succeeded, the fact was injected, and the response
was still confidently incorrect. Nothing failed loudly. Nothing threw an
error. If I hadn't been staring at the raw context myself, I'd have had no
way to know this happened except a confused (or annoyed) user telling me
about it after the fact.
I went looking for how the popular memory frameworks handle this — Mem0,
Zep, Letta, the usual suspects. They're all solving real problems: storage,
retrieval, contradiction handling as facts change over time. Zep in
particular does well on temporal accuracy benchmarks.
But as far as I can tell, none of them check the thing that actually broke
in my test: did the LLM's response actually reflect the memory that got retrieved for it? Every framework I looked at seems to assume that once a
So now I'm curious what other people are seeing. If you're running agents
with any kind of persistent memory in production —
Genuinely asking — I've been digging into this for a bit and I'm not sure
if I'm looking at something under-discussed or just late to a well-known problem.