# New Benchmark Tests Whether AI Assistants Can Remember a Year of Phone Use > Source: > Published: 2026-08-17 04:31:48+00:00 ## What happened A technical report submitted to arXiv on 11 August 2026 introduces MobileMem, a benchmark and framework for studying long-term memory in on-device AI assistants. The authors say it is grounded in a year-scale collection of mobile experiences and uses a synthesis pipeline to turn user-app sessions into long, temporally consistent trajectories. The abstract describes the design; it reports no evaluation results. A technical report titled "MobileMem: Learning from a Year of Mobile Experiences" was submitted to arXiv on 11 August 2026 by 17 authors, with Ningyu Zhang listed as the submitting author. The listing places it under artificial intelligence, with cross-listings to computation and language, machine learning, multiagent systems, and multimedia. The paper is presented as a technical report, meaning it has not been through peer review at the time of posting. What follows is drawn entirely from the arXiv listing and its abstract; the full PDF is the only place the underlying detail would appear, and we have not evaluated it. The authors describe MobileMem as both a benchmark and a framework for studying on-device long-term memory, which they say is grounded in a year-scale collection of mobile experiences. Their stated motivation is that AI agents are shifting from answering isolated questions toward persistent personal assistants that accumulate user-specific experience over time, and that existing benchmarks do not reflect realistic mobile settings, where the material is heterogeneous, multimodal, evolving, and personal. That characterization of existing benchmarks is the authors' claim; the abstract does not name the benchmarks it considers inadequate or explain the comparison. On method, the abstract says MobileMem uses a knowledge-grounded synthesis pipeline to construct coherent and temporally consistent long-horizon trajectories from user-app sessions. It provides complementary text-only and multimodal settings, and covers four categories of task: multi-hop reasoning, temporal reasoning, knowledge updating, and implicit preference inference. The framing the authors offer is that the benchmark should let agents "remember the past, understand the present, and adapt to the future," and that modeling experiences rather than isolated facts moves memory beyond information retrieval toward what they call experiential intelligence. Those last phrases are the authors' positioning, not measured findings. Several things that would matter to a reader are not in the abstract. It gives no evaluation numbers, no baseline systems, and no comparison against existing memory approaches, so there is no evidence in the material we can see about how well or badly current assistants perform on the tasks. It does not say how many people contributed the year of mobile activity, how that activity was collected, or whether participants consented to its use in a public benchmark. It does not state how much of the released benchmark consists of recorded sessions versus content generated by the synthesis pipeline, nor which models were used to synthesize trajectories. The listing references a project page, but the page URL is not rendered in the listing text we reviewed, and we have not confirmed that data or code are publicly available or under what license. [Read the primary source: arxiv.org ↗](https://arxiv.org/abs/2608.13606) ## Why it matters Personal assistants are being pitched as systems that remember and learn from a user's life rather than answering isolated questions, and benchmarks shape what developers optimize for. A test built around a year of phone activity targets a gap in current evaluation. It also raises unresolved questions about where that personal data came from and how it is handled. The commercial direction the paper points at is real and already underway: assistant products are increasingly marketed on their ability to retain context across sessions rather than treat each conversation as fresh. Memory is the feature that turns a chatbot into something that behaves like a personal system. But the industry has relatively few public, shared ways to measure whether that memory works over long spans, on messy real-world material, rather than over a handful of turns in a single chat. A benchmark aimed squarely at that gap is useful regardless of whether this particular one becomes the standard. Benchmarks exert pull on what gets built. When a public test becomes the reference point, developers tune systems to score well on it, and the test's blind spots become the field's blind spots. The four task types MobileMem names are a reasonable decomposition of what long-horizon personal memory has to do — connect facts across time, reason about when things happened, overwrite information that has changed, and infer preferences a user never stated. Whether those categories are well constructed and hard enough to separate good systems from bad ones cannot be judged from an abstract. The data provenance question is not a side issue here. A year of one or more people's phone activity is among the most sensitive material a research dataset can contain: locations, messages, purchases, health and relationship signals. The presence of a synthesis pipeline suggests one plausible reason for it — generating trajectories rather than distributing raw logs is a common way to reduce exposure — but the abstract does not say that, and we are not going to assume it. Readers evaluating this work should look for explicit statements about consent, anonymization, and what is actually shipped in the release. Synthesis also carries a measurement cost that cuts against the paper's central selling point. The authors argue that existing benchmarks are inadequate because real mobile experience is messy and evolving; if the trajectories are largely constructed by a model to be coherent and temporally consistent, they may be cleaner and more internally logical than the phone activity they stand in for. That would risk rewarding memory systems that handle tidy narrative structure while leaving open how they cope with contradiction, gaps, duplicate records, and abandoned sessions. Nothing in the abstract resolves how much recorded material anchors the synthetic output. Finally, the absence of results limits what can be concluded today. A benchmark with no reported baselines establishes that a group of researchers thinks a capability is under-measured. It does not yet establish that current assistants fail at it, by how much, or which design choices help. That evidence is what would make this consequential for anyone building or buying assistant products. ## What to watch next The substantive checks are still ahead: whether the full report, the data, and the code are released, under what license; whether baseline numbers show that today's memory systems actually struggle; how much of the benchmark is recorded activity versus synthesized; and what the authors document about consent and privacy for the underlying mobile logs. The first checkpoint is the release itself. Watch whether the project page goes live with downloadable data and evaluation code, whether the license permits commercial use and redistribution, and whether the release includes a datasheet or data statement covering collection, consent, and any anonymization applied to the original mobile sessions. A benchmark that cannot be run by outsiders, or whose data terms are restrictive, will not become a shared reference no matter how well designed it is. The second is numbers. Look for baseline evaluations in the full report or a follow-up: which memory architectures and which models were tested, how they score across the text-only and multimodal settings, and whether the four task categories actually differentiate systems or all move together. Implicit preference inference is the category most likely to be contentious, because grading it requires deciding what a user's unstated preference was — worth checking how the ground truth is defined and how consistently human or model annotators agree on it. The third is independent use. The meaningful signal is other groups running MobileMem on systems the original authors did not build, and reporting results that hold up. Related to that is contamination: once a benchmark with synthesized text is public, its trajectories can be scraped into future training data, which inflates later scores. Watch for a held-out split, canary strings, or a refresh policy that keeps the test meaningful over time. The fourth is whether the on-device framing survives contact with practice. The paper positions this as on-device memory, but most current assistant memory features run at least partly in the cloud. Whether the benchmark's assumptions — storage size, latency, what a phone can index locally — match how products are actually built will determine how much the results transfer. Peer review or acceptance at a venue would also add scrutiny that a self-posted technical report has not yet received.