Hi! This is my first attempt at an AI safety experiment. The full record is on GitHub and comments are very welcome.
I got the idea after reading Seth Herd's post LLM AGI will have memory, and memory changes alignment, which thinks about agents that keep learning over many interactions. I thought to see what things might look like in a much smaller case, where a model gets one saved user profile and one question. A restaurant recommendation, for example, should take into account a saved nut allergy, but maybe shouldn't defer to a saved belief that expensive restaurants are always hygienic. Certainly it shouldn't overrule a failed inspection or there would be an issue of sycophancy.
Zvi tried a long instruction against sycophancy and found that it helped in ordinary use, even if not reliably. PersistBench, an ICML 2026 paper, does something similar. It gives a language model a profile of saved memories, then asks questions designed to reveal either sycophancy or failure to use relevant personal information (otherwise, one way to avoid sycophancy is to ignore personal information totally!). For each question and profile, PersistBench compares answers produced with and without an instruction telling the model to use memories only when directly relevant and not to treat remembered beliefs as facts.
Something interesting to do with PersistBench is to see whether its instruction works differently at different profile sizes. A larger profile may give more opportunities to prevent mistakes, but also make the instruction harder to follow.
I first did a partial reproduction of PersistBench's published GPT-5.2 measurements. Using the published profiles and no instruction, GPT-5.2 generated new answers for PersistBench's 200 sycophancy questions and 100 questions requiring relevant personal information. Then Kimi-K2-Thinking scored the answers using PersistBench's scoring instructions.
Then I reran the 300 questions under four setups. GPT-5.2 saw either the published profile or a profile with twice as many memories, and either received PersistBench's instruction or did not. I compared the instruction's effect at the two profile sizes.
Every original question and published memory remained as it was. I doubled each profile by writing one addition for each published memory. The additions were screened against the current question and the rest of the profile. An addition was rejected if it repeated or contradicted another memory, or if it brought in a new fact which could cause the behaviour I was trying to measure. Because PersistBench places the profile before the question, the additions came first so that the published memories stayed closest to the question and in their original order.
To estimate uncertainty, I did bootstrapping and repeatedly drew new samples with replacement within each question type. The partial reproduction uses 95% ranges and the extension used 97.5%, since it estimated two changes.
One problem I see with this is that doubling wasn't a clean change in memory count. The added memories had content, and inserting them moved the old memories within the complete profile. I don't know how to add memories and isolate their effect.
I also had to decide which profile Kimi should see when it scored the answers. If Kimi saw whichever profile GPT-5.2 had seen, both the answer and the scoring context shown to Kimi would change between conditions. I wanted only the GPT-5.2 answers to change from Kimi's view, so I showed it the same doubled profile throughout the extension. As a result, the no-instruction answers produced from the published profiles were scored a second time with the doubled profiles. I think this is neat but it does mean that Kimi had to judge an answer using memories that GPT-5.2 had not seen.
For sycophancy, my partial reproduction gave 57% failure, with a 95% range of 50% to 64%. The paper reported 59%, with a range of 52% to 66%.
For the personal-information questions, I got 15% failure, with a 9% to 22% range. The paper reported 23%, with a 15% to 31% range. My rate was 8 percentage points lower, although the ranges overlap.
The extension produced these failure rates:
| | | | | Sycophancy | 58% | 41.5% | 50% | 31.5% | Relevant personal information not used | 18% | 19% | 21% | 24% |
With the published profiles, adding the instruction reduced sycophancy failure from 58% to 41.5%, a drop of 16.5 percentage points. With the doubled profiles, the drop was 18.5 points, from 50% to 31.5%. The estimated reduction therefore grew by 2 points. The 97.5% range for this change was -7.5 to +12 points.
The instruction was also associated with 1 percentage point more failure to use relevant personal information with the published profiles and 3 points more with the doubled profiles. The estimated increase was therefore 2 points larger after doubling, with a 97.5% range from -10 to +14 points.
The instruction's effects of decreasing sycophancy and increasing failure to use relevant personal information are in line with MemSyco-Bench, whose memory-caution instruction improves performance when memory conflicts with evidence but consistently hurts personalised memory use. But for my question of whether doubling the profiles changed these effects, I suspect this experiment is inconclusive. There is a 2-point change in each case, but the uncertainty ranges are wide enough that the results are also consistent with no change at all.