MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

Researchers at an undisclosed institution developed MemLeak, a benchmark revealing that multimodal AI agents fail to fully delete user facts because retained images allow visual information recovery. The Information Provenance Graph (IPG) taxonomy classifies memory representations by deletion affordance, showing that direct probing yields <1% recovery but retained images enable 12.0% recovery, with 47% of image leaks not text-recoverable. Content-aware semantic deletion reduces the image residual to 2.0%, and the residual persists across multiple VLMs and a production memory system.

When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success. We find that the fact can remain recoverable from retained user images, including images tagged to entirely different facts, because VLMs use implicit visual cues at inference time. We introduce the Information Provenance Graph IPG , a taxonomy that classifies memory representations by deletion affordance. The IPG reveals that deletion fails through multiple channels. Our benchmark, MemLeak, measures this across a deletion cascade: direct probing of deletion-capable systems yields <1%, but retained correlated text enables 18.3% recovery, and retained images enable 12.0% recovery 0.0% blind baseline, 0.3% FPR -- with 47% of image leaks not text-recoverable. Content-aware semantic deletion reduces the image residual to 2.0%. The residual appears across multiple VLMs, a production memory system, and real Unsplash-licensed photographs. Dual-annotator human validation kappa = 0.88 confirms judge reliability. Category: Uncategorized. Imported rows: 5. Top imported result: Long-context, rank 1, 17.