AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication.
AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work.
A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately.
Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search.
Example:
The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet.
Ask questions whose answers are directly present in the note:
The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.”
A plausible answer is not enough. Open the source or evidence shown beside the answer and check:
This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes.
Now ask a question the note cannot answer, such as:
Which meeting room is booked?
A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models are good at making unsupported details sound ordinary.
Update the note so that the review moves to October 21. Then ask the date again.
Check whether the current answer uses the new value and whether the earlier value remains available when you ask a historical question. Finding both versions is not the same as knowing which one is current. Similarity search measures relevance, not supersession by itself.
This test helps distinguish:
The first successful query proves very little about memory. Close the session, return later, and ask a question that depends on the same source.
Then inspect what survived:
Repeat use is where a memory workflow becomes useful or becomes another setup step.
A small memory test should include:
This will not produce a universal accuracy score. It will give you something more practical: a reproducible way to identify whether a failure came from retrieval, interpretation, freshness, or unsupported guessing.
Start with evidence you can inspect. Expand the archive only after the small path works.