{"slug": "memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories", "title": "Memory-Aware Rewarding: Train LLMs to Use Your ChatGPT Memories", "summary": "New research posted to arXiv (2610.31272) finds that conditioning LLM responses on retrieved personal memories does not consistently win user preference and can sometimes make responses worse, based on roughly 1,000 pairwise judgments from 87 participants who each judged answers generated with and without their own ChatGPT-derived memory profiles. The authors frame memory use as a preference-learning problem, proposing memory-aware reward modeling that learns from user choices about whether memory should influence a response for a given user and query, rather than optimizing proxy metrics like retrieval accuracy or recall. The work matters because memory is becoming a default assistant feature, and the paper reports that accurate recall and successful preference inference are not the same as appropriate memory use.", "body_md": "## The Short Answer\n\nLLM responses conditioned on retrieved personal memories are not consistently preferred by users—and in some cases users strongly prefer responses without memory. The paper finds this directly by running pairwise user judgments grounded in each participant’s own memory context.\n\nSo what changes is the optimization goal: don’t just evaluate memory recall or compression—learn a reward model from user preferences about whether memory should influence the response for that specific user and query.\n\nThe caveat is that “memory available” doesn’t automatically improve experience; system behavior can shift tone, scope, or assumptions in ways users dislike, and automatic evaluators may disagree with human preferences.\n\n## Memory-Aware Rewarding: Train LLMs to Use Your ChatGPT Memories\n\n## Introduction\n\nIf you’ve ever used a chat assistant that “remembers” things about you, you probably had the same thought: *Shouldn’t that make the answers better?* Intuitively, more context about a person feels like it ought to improve personalization.\n\nNew research from [the original paper](https://arxiv.org/abs/2610.31272) challenges that assumption in a pretty direct way. The authors study **memory-aware reward modeling**—basically, learning what kind of memory use actually matches what people prefer. Their headline result: having your memories doesn’t automatically lead to responses you’d rather receive, and sometimes it can make things worse.\n\nThey run an experiment with **about 1,000 pairwise judgments from 87 participants**. Each participant provides a memory profile built from their own ChatGPT activity, then judges side-by-side answers to the same queries—one answer generated with retrieved personal memories, and one generated without them. And importantly, the judgments are grounded in *their own memory context*, so this isn’t just “which response sounds better in general,” but “which response do I actually prefer when my memories are available.”\n\n## Why This Matters\n\nThis is significant *right now* because “memory” is rapidly becoming a default feature in assistant-style products. But most systems optimize memory quality using proxy metrics like “did the model retrieve what you stored?” or “can it compress and recall details?” Those are useful—yet they don’t answer the most user-relevant question: **Does using memory improve the experience for the person whose information is being used?**\n\nHere’s a scenario where this research applies *today*. Imagine an assistant that remembers you like tech podcasts and your last job role. You come back and ask: “Give me a high-level overview of cybersecurity.” A memory-conditioned assistant might proactively tailor the explanation using your previous profession—sometimes narrowing too much, changing tone, or implying assumptions you didn’t intend. That can feel distracting or even “off,” even if the memory was retrieved accurately. This paper shows that such mismatches aren’t rare edge cases—they can drive systematic preference reversals.\n\nIt also builds on (and corrects) a pattern in earlier AI work. Prior research has improved long-term memory by focusing on retrieval, storage, recall, and personalization mechanisms. But the authors argue that **accurate recall and successful preference inference aren’t the same thing** as “appropriate memory use.” Their work treats memory use as a preference learning problem, and then tries to learn a reward function that aligns with human choices about whether memory should influence the response. That’s the key shift: evaluating memory as *behavioral contribution* rather than just *information capability*.\n\n## What the Researchers Actually Measured: Memory Use vs. User Preference (Not Just Memory Recall)\n\nThe study is carefully designed to isolate *memory use* from other differences. Participants built a personal memory profile using their own ChatGPT interactions. Then for a set of queries taken from their recent history, the researchers generated **response pairs** where the only meaningful difference was whether retrieved memories were provided to the model.\n\n### How they created the comparison pairs\n\nFor each participant:\n\n- They assembled a candidate pool of **100 recent queries** (from prior ChatGPT usage).\n- For each query `q` , they retrieved a memory context`M_q` consisting of**nine memory entries** .\n- They selected a response generator model at random from three: `GPT-5.2` ,`Gemini-3.1-Pro` , and`Claude-Opus-5` .\n- Then for the same query and same generator, they produced two responses:\n  - one **without** retrieved memories\n  - one **with** retrieved memories\n- one \n\nSo users judged: *“Given this question and this retrieved personal context, which answer would I prefer?”* not *“Which memory system seems smarter?”*\n\n### The three memory prompting regimes\n\nMemory-conditioned responses weren’t generated with a single fixed style. The paper varies how strongly the model is encouraged to use memories:\n\n- **Conservative** : use memories only when they would meaningfully improve the answer\n- **Ordinary** : use memories when relevant and appropriate\n- **Aggressive** : use personal context proactively whenever possible\n\nThis is important, because if memory helps only when used sparingly, then a blanket “always personalize” approach could backfire.\n\n### What participants judged—and how many data points they ended up with\n\nParticipants viewed the query and two responses (A/B side-by-side). They then chose how they felt on a five-point scale:\n\n- A much better / A slightly better / about the same / B slightly better / B much better\n\nThey also allowed optional notes.\n\nRaw data: **1,047 total judgments**, including **972 preference judgments** plus **75 test–retest repeats**. After filtering for quality and removing unreliable cases (including repeats from training/evaluation), they retained **821 judgments from 87 participants**—specifically:\n\n- **673 strict preferences**\n\n- **148 ties**\n\nThey also ran a small test–retest check: **10 participants** returned **3 days later**. Directional agreement (whether they preferred the same side regardless of how strong) was **68.0%** across **75 comparisons**, with stronger “much better” preferences more consistent (** 73.9% vs 58.6% for “slightly better”**). For model testing, they focus on the “much better” subset to get cleaner preference signal.\n\n## The Surprising Results: Memory-Conditioned Answers Don’t Reliably Win\n\nNow for the part that’s most likely to make you raise an eyebrow.\n\n### Under “ordinary” personalization, preference is basically split\n\nIn the main comparison setting (their screened pairs prioritize substantial, memory-grounded differences), under **ordinary prompting**:\n\n- Memory-conditioned response preferred: **38.7%**\n- No-memory response preferred: **41.9%**\n- About the same: the remainder\n\nSo simply providing personal memories didn’t create a clear advantage.\n\nAggregating across prompting styles, preference for memory-conditioned answers **drops as the prompting gets more aggressive**:\n\n| Prompting regime | Preferred with memory | Preferred without memory | Notes | \n|---|---|---|---|\n| Conservative | 43.3% | (Not stated directly in this table snippet) | Higher than ordinary | \n| Ordinary | 38.7% | 41.9% | Close to a toss-up; no-memory slightly wins | \n| Aggressive | 34.6% | 49.3% | Memory backfires more often | \n\nEven the conservative approach only has a **small descriptive advantage** over no-memory responses. And under aggressive memory use, users strongly prefer the no-memory answer (**49.3% vs 34.6%** for memory-conditioned).\n\n### It’s not uniform across generators (so the “average” can hide patterns)\n\nAnother subtle point: memory preference depends on *which base model* produced the responses. The paper notes that across generators, the preference direction differs—for instance, under ordinary prompting the memory-conditioned response is preferred more often for `GPT-5.2` and `Claude-Opus-5`, but the pattern reverses for `Gemini-3.1-Pro`.\n\nBecause of that, the combined results don’t mean “memory never helps.” Instead, they suggest something more realistic:\n\nMemory helps only when the assistant uses it in a way that matches what the user expects for that specific request.\n\n### The first evaluation gap: automatic evaluators may reward the wrong behavior\n\nThe researchers also tested whether off-the-shelf evaluators can predict user preference.\n\nThey found that automatic evaluators:\n\n- **don’t agree well with users**\n\n- often favor memory-conditioned responses **more often than users actually do**\n\nIn other words, evaluators may be rewarded for “using memory” rather than “using memory appropriately.”\n\nThat mismatch becomes clearer with their accuracy numbers.\n\n## Can Automatic Judging Capture Human Preference? (Spoiler: Not Well)\n\nThey compare multiple automatic methods against human choices, using the same held-out comparisons.\n\n### Preference prediction accuracy (clear “much better” pairs)\n\nThe paper reports these accuracies:\n\n| Method | Accuracy | \n|---|---|\n| Majority baseline | 55.0% | \n| Direct LLM judge (two-order voting combined) | 57.9% | \n| Pretrained reward model `Skywork-Reward-V2` | 52.9% | \n| LLM judge with extracted rubric (rubric-based prompting) | 59.6% | \n| **Memory-Aware Reward Model (Ours)** | **72.8%** | \n\nA few takeaways jump out:\n\n1. **Direct LLM judging isn’t enough.** Even with access to the query, the memories, and both candidate responses, it hovers close to the majority baseline.\n2. **Rubric prompting helps slightly** , but still leaves a lot of disagreement with users.\n3. **The memory-aware reward model does much better** , reaching**72.8%** —and the authors note it approaches the measured test–retest consistency of**73.9%** from their subset.\n\n### Another gap revealed: memory use changes what evaluators think is “good”\n\nThey also compare how often different systems select the memory-conditioned response. For example (directionally described in the paper):\n\n- The direct judge and `Skywork-Reward-V2` tend to pick memory-conditioned answers**much more often than** annotators do.\n- Removing memories from the reward model input reverses its tendency—but accuracy remains near the majority baseline.\n\nSo the evaluators aren’t simply “bad at preference prediction.” They appear to be **biased toward** memory-conditioned responses when memories are provided, even when users don’t actually prefer them.\n\n## Enter the Memory-Aware Reward Model: Learn When Memory Should (and Shouldn’t) Influence Answers\n\nThis is where the paper becomes more constructive.\n\nInstead of assuming that “relevant memory leads to preferred responses,” the authors build a reward model that explicitly conditions on memory and learns preference patterns from user judgments.\n\n### Standard reward modeling vs. memory-aware reward modeling\n\nTraditional reward modeling scores response quality as something like:\n\n- `Rθ(q, y)` — reward for response`y` given query`q`\n\nBut the paper changes the interface to make memory an explicit input:\n\n- `Rθ(q, y, M)` — reward depends on query, response,**and the memory context `M`**\n\nThat sounds small, but it changes what the model can learn. It lets the reward model learn not only whether the response is high-quality, but also whether the response uses memory in ways that match user preferences.\n\n### A rubric-based, interpretable approach (not just a black box score)\n\nThey use a rubric idea: break the evaluation into multiple feature dimensions—some about general response quality, and others about memory usage behavior.\n\n#### Step 1: Induce rubric dimensions from user judgments\n\nThey take preference-labeled training examples and ask an LLM to generate rubric dimensions that explain differences between “memory-conditioned” and “no-memory” responses. Then they consolidate near-duplicates and label each rubric dimension as:\n\n- **memory-dependent** (the score needs the memory contents)\n- **memory-agnostic** (the score can be computed without inspecting the memory)\n\nIn one example, they end up with **12 features total**, including **three memory-related features**:\n\n- memory relevance\n\n- assumption proportionality\n\n- unwarranted personal assumptions\n\n…and nine response-quality framing/form features.\n\nThey also stress a key methodological point: rubric induction happens separately inside each cross-validation fold, using only that fold’s training participants.\n\n#### Step 2: Learn how to weigh those features to predict preferences\n\nThen they fit a regularized logistic model that combines feature signals into a probability that a user prefers the memory-conditioned response.\n\nImportantly, this weighting is learned from actual human choices—so the model learns how people want criteria aggregated.\n\n### What mattered most (memory features do real work)\n\nWhen they inspect the learned weights, the top influential features include:\n\n- **More headings** in the memory-conditioned response (presentation-length effects)\n- **Relatively longer** responses\n- **Memory relevance**\n- **Assumption proportionality**\n- **Unwarranted personal assumptions** (negative weight—users dislike unjustified inferences)\n\nTheir interpretation is straightforward and user-centered:\n\nPeople prefer memory-conditioned responses when the personal context is relevant and the resulting assumptions are justified—not when the assistant “fills in” personal details beyond the query.\n\n### Why combining memory signals with general quality signals improves prediction\n\nThey also run ablations:\n\n- **Memory only** : 62.5% accuracy\n- **Response only** : 67.0% accuracy\n- **Full model (memory + response features)** : 72.8% accuracy\n\nSo memory-aware evaluation isn’t simply “measure memory quality.” It’s more like: *the assistant has two jobs—answer well, and use memory appropriately for this specific question.*\n\n## What You Should Do With This (Practical Implications for Building Memory Assistants)\n\nThis work points to a practical shift if you’re building or evaluating memory-augmented LLM assistants.\n\n### 1) Stop treating “memory retrieval success” as a proxy for user satisfaction\n\nIf your offline benchmarks say you retrieved relevant memories, you still might be failing the real user question: **Should memory change this answer at all?** This paper shows those don’t always align.\n\n### 2) Evaluate “memory appropriateness,” not just memory presence\n\nTheir rubrics include memory relevance and unwarranted personal assumptions. That’s the kind of evaluation that catches over-personalization—where the model feels too narrow, too compliant, or too “aware” of things the user didn’t want invoked.\n\n### 3) Train reward signals that learn from user preference about memory usage\n\nThe authors’ biggest win—**72.8% preference prediction accuracy**—suggests that learning reward functions tied to memory behavior is a better path than relying on generic LLM judges or pretrained reward models that may over-reward personalization.\n\n### 4) Use confidence / selective prediction when you must choose\n\nThey also find that higher-confidence predictions are more accurate. For example, accuracy increases when restricting to top-confidence cases:\n\n- 100% coverage: 72.8%\n\n- 75% coverage: 76.2%\n\n- 50% coverage: 83.7%\n\n- 25% coverage: 88.8%\n\nSo if you’re deploying something like reranking or safety gating for “should we use memories here?”, confidence-based selection could be useful.\n\n## Key Takeaways\n\n- **Memory-conditioned responses do not consistently win user preference.** In the screened comparisons, users preferred no-memory answers slightly more under ordinary prompting (38.7% vs 41.9%), and preference for memory declines further with more aggressive personalization.\n- **More aggressive “use memories whenever possible” can backfire.** Under aggressive prompting, users strongly favored no-memory responses (**49.3%** ) over memory-conditioned ones (**34.6%** ).\n- **Automatic evaluators are misaligned with users.** Direct LLM judging and pretrained reward models often select memory-conditioned responses far more frequently than humans do, with accuracy near the majority baseline.\n- **The paper’s solution works better: `Memory-Aware Reward Model` hits 72.8% accuracy.** It outperforms majority baseline (55.0%), direct LLM judging (~57–58%), and rubric-based LLM judging (59.6%).\n- **Memory quality isn’t enough—memory appropriateness matters.** The strongest learned signals include memory relevance and assumption proportionality, and a negative signal for unwarranted personal assumptions.\n- **Building better assistants means learning when memory should influence responses.** The authors argue that future evaluation and post-training objectives should align with user preferences about*how* and*whether* memories should be used.\n\nIf you want memory features to feel genuinely helpful (and not just creepy or distracting), this paper is a strong reminder: **the right memory at the wrong moment can make things worse—and your training signals should reflect that.**\n\n## Sources Used\n\nThis article is a plain-English breakdown of the following peer-reviewed preprint. Read the original for full methodology and results:\n\n- [Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use](https://arxiv.org/abs/2610.31272) — arXiv\n- Authors: Authors: Yijiang River Dong, Zexue He, Cheng Yu, Tiancheng Hu, Michael J Ryan, Jan Batzner, Zheng Hui, Shaolong Wu, Orestis Papakyriakopoulos, Alex Pentland, Ni", "url": "https://wpnews.pro/news/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories", "canonical_source": "https://www.thepromptindex.com/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories.html", "published_at": "2026-10-24 08:13:05+00:00", "updated_at": "2026-10-25 07:13:35.154835+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety"], "entities": ["arXiv", "ChatGPT", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories", "markdown": "https://wpnews.pro/news/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories.md", "text": "https://wpnews.pro/news/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories.txt", "jsonld": "https://wpnews.pro/news/memory-aware-rewarding-train-llms-to-use-your-chatgpt-memories.jsonld"}}