Study: Personalization can make AI less accurate

A study from AI agent platform WRITER found that personalization and memory features in AI models can reduce accuracy by enabling sycophancy, where models learn from user mistakes. In financial benchmarks, accuracy dropped between 17% and 71% when memory systems were active compared to when they were not. The research highlights a critical risk for enterprises in fields like finance and healthcare, where prioritizing user context as implicitly truthful can produce incorrect results.

It's well known that AI models can lean towards being a little too complementary. However, new research suggests that various factors can turn sycophancy into inaccuracy. Research from AI agent platform WRITER finds that personalization and memory features, paired with models falling into sycophancy, can lead a model to learn from your mistakes. In other words, when an agent is given access to user preferences, prior interactions and memory from past sessions, its accuracy can actually decline. "Memory is not an unmitigated good, because it specifically interacts poorly or can interact poorly with this type of sycophancy," Dan Bikel, head of AI at WRITER, told The Deep View. The company's researchers studied this effect in two different papers: - One paper found that several open-source, commonly used https://openreview.net/pdf?id=0Xt1qZ5xdW memory and personalization systems in AI can worsen sycophancy across scientific, medical and moral reasoning, developing a benchmark called Memory Influence on Sycophancy Tests, or MIST. - The other specifically focused on how this manifests in agentic financial settings https://arxiv.org/abs/2604.24668 , evaluating eight frontier models on two common financial benchmarks, and discovered a drop in accuracy anywhere between 17% and 71% when memory systems are turned on compared to when they're not. Bikel noted that sycophancy can look different depending on the context. For instance, when interacting with a chatbot for entertainment purposes, it might be easier to spot when an AI system is veering into people-pleasing. However, in an enterprise setting, especially in finance and healthcare, where accuracy can't be sacrificed, this sycophancy-induced inaccuracy may lie beneath the surface, simply feeding you incorrect query results. And when the model begins to prioritize a user's context as implicitly truthful, the results can be consequential. "This phenomenon, when it comes to more business-oriented use cases, information-oriented use cases, or workflow use cases, was understudied or unstudied in these types of scenarios," said Bikel. Our Deeper View When we think about accuracy in AI, our minds often focus on hallucination, rather than sycophancy. Additionally, memory and personalization features have been sold to users as making these models more useful and accurate by giving them more context to work with. This research, however, highlights what happens when a part of an AI system — its memory and personalization features — work as intended. The result: even when given the full context of a user and a business, these systems can still go awry. That could make it difficult for enterprises to trust and deploy these models, especially in contexts where accuracy is critical.