# How would you evaluate a personal AI for invented memories and revoked consent?

> Source: <https://discuss.huggingface.co/t/how-would-you-evaluate-a-personal-ai-for-invented-memories-and-revoked-consent/190672#post_5>
> Published: 2026-10-17 07:29:12+00:00

Two baselines that cover parts of this:

LongMemEval ( [\[2410.10813\] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://arxiv.org/abs/2410.10813) ) tests “information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention”, and splits memory into “indexing, retrieval, and reading”. That split answers whether to score retrieval and the final answer separately: I think you should, since an invented memory can come from either stage.

For revoked consent, TOFU ( [\[2401.06121\] TOFU: A Task of Fictitious Unlearning for LLMs](https://arxiv.org/abs/2401.06121) ) is the nearest setup: “200 diverse synthetic author profiles” with a “forget set that serves as the target for unlearning”. You can treat a revoked fact as forget-set data and check that it never comes back in answers; not only that, but it is gone from the store.

One suggestion from my side: score abstention on questions the system should not know, together with accuracy on ones it should. Otherwise, a system that refuses everything looks perfect on invented memories.