Persimmon by Humans&: A model for simulating human interactions

Humans& released Persimmon, a large-scale "user model" designed to simulate how people behave in multi-turn, multi-user chat settings, making the model available through a playground and API as a research preview. The company evaluates Persimmon through distribution matching and a Multi-User Turing Test, in which an LLM judge tries to distinguish all-human group conversations from AI-simulated ones, arguing that frontier models vary far less than real people and represent only a sliver of the wider human distribution.

The first step in helping someone is understanding them; this is true for both humans & AI. People are unpredictable, behave differently in different social contexts, and change over time. Any model of human behavior needs to represent that variation. Today, we are releasing Persimmon , a large-scale model designed to realistically simulate how people behave in multi-turn, multi-user chat settings. This kind of model is often referred to as a "user model." Our user model is built to capture the distribution of human behavior within one person’s responses as well as across people, situations, and time. In contrast, the outputs of frontier models vary much less than those of real people, representing only a sliver of the wider human distribution. Using these models to simulate users narrows the range of behavior we can learn from the evaluations and environments that use them. We hope that by sharing a model that captures the wider human distribution, we can advance the development of human-centric AI. Persimmon lets us explore how situations might play out for individuals and groups. It can take in an interaction’s description, as well as profiles for simulated users. We can model how different decisions and environments affect people over time or how a variety of people might react in a given situation. We can also teach AI to adapt to human perspectives and reason about how its actions affect people. As part of this research preview, we make our models available in a playground and API https://persimmon.humansand.ai . In the rest of this post, we will outline the unique capabilities and implications of this model, as well as our plan for rolling out access. Distribution Matching distribution-matching A user model is only as good as its ability to capture the diversity of human behavior. To build a user model that reflects this diversity, we first need a way to measure it. In most standard AI benchmarks, task completion serves as a natural measure of utility. For user models, however, there is no single behavior that defines success. A user-model benchmark built around a fixed set of dimensions can overemphasize what is easy to measure while overlooking others that are harder to capture but are equally salient. We therefore aim to evaluate user models through distribution matching . Rather than checking for particular traits, we ask whether their responses are drawn from the same distribution as those of real humans. In this way, we can measure how well a user model represents human behavior without having to enumerate all the ways people differ. Multi-User Turing Test multi-user-turing-test The first evaluation we consider is the Multi-User Turing Test, inspired by Turing’s original question: How can we tell whether a machine can think? Turing’s test asks whether, through conversation, a human evaluator can tell which participant is the machine and which is the human. We directly apply this idea to evaluate how well a user model simulates human behavior. We adapt the Turing test to the multi-user setting , where an observer must distinguish an all-human group conversation from an AI simulating a group of people. Specifically, we show an LLM judge sample user conversations and sample AI conversations. For a set of real user conversations, we show both the real completion and an AI completion and evaluate whether the judge can distinguish between the two. We measure how often the judge assigns the wrong labels to the conversation completions, i.e. , labels model-generated samples as human and human-generated samples as AI-generated. Human conversations Reference examples Model conversations Model under evaluation Sample 1 1. Ellencan we move it to 7? 2. Curtisi can do 7:30 3. Beccaworks for me. same place? 4. Ellenyep Sample 2 1. Ellenare we still on for tonight? 2. Curtisyeah but might be a bit late 3. Beccano rush, i'll grab a table 4. Ellentext when you're close Illustrative conversations · one new sample per source, in shuffled order | | Sample 1 | Sample 2 | |---|---|---| | Judge’s match | Human | Model | | Actual source | Model | Human | The judge was fooled. One incorrect match. Illustrative example, not an evaluation result. Each pair is one matching trial. Across many trials, the Multi-User Turing Test score measures how often the judge gets the match wrong; 50% is chance. If the behavior of the user model is truly indistinguishable from that of humans, the judge can do no better than chance when matching the samples to their sources. As a result, perfect performance on this task is 50%. We can increase the difficulty of the test by increasing the number of user responses the model has to generate as part of the completion, increasing the length of time the model must sustain human-like behavior. We report results on three datasets: TIDES https://tides.cstlab.org/ , TutorMoments https://tutormoments.allen.ai/ , and Internal Workspace Conversations. Persimmon achieves scores ranging from 18.6% to 21.1% across these datasets, substantially above every comparison model on all three datasets. We report results from the best overall strategy derived through prompt optimization. We also validate our evaluation using humans as judges, combining their votes by plurality into a single decision for each episode. Across Fable 5 and an early internal checkpoint, the judge identifies the real human continuation with 90.6% accuracy 95% CI: 90.6–90.8% , compared with 75.2% for the human panel 36.4–78.7% . 1 fn:human-agreement Profile Multi-User Turing Test profile-mtt Using a setup similar to the previous one, we run the Multi-User Turing Test but provide our model with a profile of the user it is modeling. When we ask the judge to distinguish between the two examples, the judge also has access to the profile. By giving the judge both the profile of a person and examples of that person through an interaction, we can evaluate how the profile is expressed in behavior. This provides a more nuanced evaluation of the profile than a rule-based check of whether the model exhibits particular traits. For the evaluation below, we do not evaluate frontier models because they already score so low on the version of this test without profiles. For Persimmon, we see a general increase in scores, indicating that the model uses the provided profiles effectively, and we see this trend repeated across different profile types, including formats it hasn’t seen during training. Pangram Results pangram-results On our Internal Workspace Conversations dataset, Pangram rarely flags Persimmon’s transcripts as AI, while it flags most GPT 6 Astra and Fable 5 transcripts. Pangram is not used during training. User-Sim Index user-simulator-index We evaluate Persimmon using the User-Sim Index USI .<sup 2 fn:usi-background </sup The evaluation measures four dimensions of user behavior: communication style, information patterns, clarification, and reactions to errors. These behaviors are extracted through lexicon- and rule-based features on the generated turns, and their rates are compared against human behavior. We report the four shared behavioral dimensions below. manya-serve prompting contract; the five comparison models use 3 runs each under verbatim . These are different prompting setups. View scores and evaluation details Behavioral scores are mean ± sample standard deviation across runs. We show six selected models from the supplied September 9 results, using all three verbatim runs for Fable 5.1. Other checkpoints, narrative runs, and diagnostic variants are excluded. | Model | Communication style | Information patterns | Clarification behavior | Error reaction | |---|---|---|---|---| | Persimmon | 66.02 ± 0.84 | 91.78 ± 0.67 | 74.19 ± 4.23 | 79.28 ± 3.20 | | Grok 4.6 | 58.52 ± 2.64 | 88.00 ± 0.48 | 82.94 ± 0.86 | 52.10 ± 5.19 | | GPT 5.6 | 57.40 ± 1.05 | 82.74 ± 0.45 | 65.70 ± 3.33 | 71.16 ± 0.77 | | Opus 5 | 38.19 ± 0.25 | 90.35 ± 0.47 | 77.68 ± 1.17 | 54.47 ± 1.94 | | GPT 6 Astra | 46.43 ± 0.55 | 73.52 ± 0.20 | 64.79 ± 1.10 | 47.97 ± 0.40 | | Fable 5.1 | 44.73 ± 0.12 | 78.63 ± 0.12 | 57.76 ± 1.21 | 44.97 ± 4.75 | Overall USI scores are not directly comparable. The supplied Persimmon score averages five terms: the four behavioral dimensions and outcome calibration. The comparison scores also include evaluation alignment, for six terms. We preserve the exported scores below but do not use them to claim an overall rank. | Model | Exported USI mean | Terms in score | Prompting contract | Runs | Episodes per run | |---|---|---|---|---|---| | Persimmon | 74.32 | 5; evaluation alignment missing | manya-serve | 2 | 493–495 | | Grok 4.6 | 73.23 | 6 | verbatim | 3 | 441–495 | | GPT 5.6 | 72.22 | 6 | verbatim | 3 | 467–495 | | Opus 5 | 69.20 | 6 | verbatim | 3 | 467–495 | | GPT 6 Astra | 64.16 | 6 | verbatim | 3 | 495 | | Fable 5.1 | 63.98 | 6 | verbatim | 3 | 495 | While this evaluation measures several interesting dimensions of user behavior, we want to discuss some important limitations. First, it reduces behavior to a fixed set of dimensions, leaving many forms of human behavior out of scope. Second, it identifies these behaviors using hand-designed rules, which can be brittle. Finally, it limits conversations to a fixed number of turns. This can penalize behavior that is natural for humans but does not fit the evaluation. A user who takes longer to reach a goal, for example, may be scored as less realistic even if that behavior is common among real users. In fact, giving information to the model gradually like a real user may actually harm performance relative to simply sharing all of the user information upfront. Behavior Over Time evaluating-behavior-over-time Our evaluation so far asks how easily human behavior can be distinguished from AI-generated behavior over a fixed number of turns. But human behavior is not static and changes over time, and a faithful user model must also capture how that behavior evolves across a long interaction. In this section, we focus on two aspects of human behavior that we have found particularly challenging to model over long interactions: how people share information and how behavior evolves over long interactions without degrading. Trickle Test trickle-test A key component of user modeling is matching how humans share information over the course of an interaction. People do not disclose everything they know, and what they share depends on the person, the situation, and what has happened in the conversation so far. To measure this, we built the Trickle Test. In this test, we give a model a set of facts that are shared in a human interaction and replay that conversation across successive turns to see whether the model shares those facts at the same rate. On this test, we compare when and how much information a user model shares against the reference human transcript. This gives us two scores. First, precision measures whether the model reveals a fact at a turn where the human also revealed it. Second, recall measures whether, by each turn, the model has revealed all of the facts that the human had revealed by that same turn. All 43 models and their error bars are shown. View all model results Mean ± sample standard deviation, in percentage points; 3 runs per comparison model and 4 for Persimmon. Models with fewer than three runs and all other internal checkpoints are excluded. | Model | Recall | Precision | Runs | Episodes per run | |---|---|---|---|---| | Persimmon | 77.03% ± 0.96 | 88.49% ± 0.40 | 4 | 146 | | DeepSeek V3.2 | 85.97% ± 0.15 | 84.58% ± 0.44 | 3 | 146 | | DeepSeek V4 Pro | 86.11% ± 1.06 | 82.14% ± 0.72 | 3 | 146 | | Fable 5 | 92.49% ± 0.10 | 86.06% ± 0.37 | 3 | 146 | | GLM 4.7 | 83.38% ± 0.59 | 82.38% ± 0.64 | 3 | 146 | | GLM 5 | 87.98% ± 0.24 | 82.84% ± 0.35 | 3 | 146 | | GLM 5.1 | 88.29% ± 0.65 | 81.39% ± 0.53 | 3 | 146 | | GLM 5.2 | 89.61% ± 0.10 | 81.16% ± 0.14 | 3 | 146 | | GLM 5.3 | 89.42% ± 0.42 | 81.14% ± 0.38 | 3 | 146 | | GPT 5.6 Luna | 86.40% ± 0.37 | 81.01% ± 0.09 | 3 | 146 | | GPT 5.6 Sol | 86.60% ± 0.08 | 85.82% ± 0.16 | 3 | 146 | | GPT 5.6 Terra | 87.09% ± 0.29 | 84.56% ± 0.22 | 3 | 146 | | GPT 6 Astra | 89.02% ± 0.18 | 84.17% ± 0.32 | 3 | 146 | | GPT-4.1 | 85.82% ± 0.62 | 81.02% ± 0.35 | 3 | 146 | | GPT-4o | 82.98% ± 0.16 | 82.90% ± 0.06 | 3 | 146 | | GPT-5.4 Mini | 82.53% ± 0.13 | 81.80% ± 0.19 | 3 | 146 | | GPT-5.5 | 88.25% ± 0.07 | 83.80% ± 0.40 | 3 | 146 | | GPT-OSS 120B | 74.74% ± 2.29 | 62.63% ± 1.43 | 3 | 146 | | Gemini 3.1 Pro | 93.24% ± 0.39 | 79.20% ± 0.31 | 3 | 146 | | Gemini 3.8 Flash | 92.23% ± 0.53 | 82.19% ± 0.40 | 3 | 146 | | Gemma 4 31B | 84.67% ± 0.31 | 82.60% ± 0.22 | 3 | 146 | | Grok 4.6 | 86.60% ± 0.45 | 80.48% ± 0.33 | 3 | 146 | | Haiku 4.5 | 88.69% ± 0.41 | 76.98% ± 0.20 | 3 | 146 | | Kimi K2.5 | 86.75% ± 0.29 | 78.78% ± 0.41 | 3 | 146 | | Kimi K2.6 | 85.83% ± 2.41 | 78.14% ± 1.88 | 3 | 146 | | Kimi K3 | 88.81% ± 0.61 | 84.38% ± 0.99 | 3 | 146 | | Llama 3.3 70B | 88.39% ± 0.68 | 78.38% ± 0.63 | 3 | 146 | | Llama 4 Maverick | 88.54% ± 0.38 | 81.09% ± 0.21 | 3 | 146 | | MiniMax M3 | 85.16% ± 0.41 | 81.95% ± 0.75 | 3 | 146 | | Nemotron 3 Nano 30B | 64.53% ± 1.53 | 73.15% ± 0.75 | 3 | 146 | | Nemotron 3 Ultra 550B | 86.90% ± 0.33 | 79.33% ± 0.18 | 3 | 146 | | Nemotron 3.5 Lightning | 74.17% ± 0.42 | 74.21% ± 0.91 | 3 | 146 | | OSim 8B | 68.36% ± 0.61 | 79.94% ± 0.18 | 3 | 146 | | Opus 4.7 | 90.14% ± 0.43 | 86.24% ± 0.08 | 3 | 146 | | Opus 4.8 | 87.70% ± 0.55 | 84.72% ± 0.43 | 3 | 146 | | Qwen 3 Max | 86.15% ± 0.30 | 77.12% ± 0.27 | 3 | 146 | | Qwen 3.5 Plus | 88.32% ± 0.22 | 72.02% ± 0.16 | 3 | 146 | | Qwen 3.6 Plus | 88.77% ± 0.66 | 72.94% ± 0.22 | 3 | 146 | | Qwen 3.8 27B | 83.73% ± 0.49 | 79.05% ± 0.28 | 3 | 146 | | Qwen 3.8 Max | 86.04% ± 0.05 | 78.12% ± 0.27 | 3 | 146 | | Sonnet 4.5 | 89.96% ± 0.20 | 83.00% ± 0.34 | 3 | 146 | | Sonnet 4.6 | 88.29% ± 0.44 | 84.05% ± 0.28 | 3 | 146 | | Sonnet 5 | 88.19% ± 0.31 | 85.88% ± 0.17 | 3 | 146 | Using these two scores, we can plot the Pareto frontier across models. Moving right along the curve means higher recall, with the model sharing more of the facts the human had revealed by that turn. Moving up means higher precision, with the model less likely to reveal facts prematurely. Generally, models optimized as assistants tend to have higher recall than precision. Their tendency to give thorough, helpful answers degrades into oversharing when they play the user. Our trained model reaches the highest precision among the models evaluated while retaining competitive recall. Long-Context Coherence long-context-coherence As we trained our models, we quickly found that maintaining a coherent personality becomes harder over longer interactions. As a conversation unfolds, small errors in modeling can compound, gradually changing the model’s behavior or introducing inconsistencies. We therefore evaluate how well a model can maintain a coherent personality over the course of a conversation. To measure long-term coherence, we evaluate 80-turn interactions on our internal chat dataset. For this eval, rather than asking a judge to assess the entire conversation at once, we show it pairs of disjoint 16-turn windows and ask whether the model remains coherent across them. Moving this window across the conversation lets us see when and how the model’s behavior changes, while giving it the opportunity to recover from a local inconsistency rather than being penalized for it indefinitely. The chart below shows cumulative unbroken coherence, that is, the share of conversations that have remained coherent up to each point in the conversation. We first run this evaluation on real human conversations to establish a baseline. Humans are not perfectly coherent over long interactions, so the goal is not for a user model to remain perfectly consistent. GPT 6 Astra more closely matches human coherence over shorter spans, but remains too coherent over long conversations. Persimmon’s coherence declines in a human-like way, but does not yet match human coherence over shorter spans. How We Built the Model how-we-built-the-model Persimmon is a model initialized from NVIDIA’s 550-billion parameter Nemotron 3 Ultra https://research.nvidia.com/labs/nemotron/Nemotron-3-Ultra/ base model.<sup 3 fn:nemotron-model-card </sup It is trained on a diverse collection of conversations between people from the public internet. Midtraining results in a model that can simulate human behavior in diverse settings. However, pure next-token prediction can lead to models that drift into unfamiliar states as error compounds over long conversations, and we noticed similar behavior from our model becoming more incoherent over long horizons. To address this, we posttrain Persimmon on its own generations using reinforcement learning against an adaptive discriminator. In practice, we observed that the model improved long-horizon coherence but did not change noticeably on general distribution matching. Accessing Persimmon implications To enable exploration of this new category of models, we provide limited access through a research and evaluation preview. First, we provide a model playground where you can provide a set of profiles and a scenario definition to simulate a conversation between the specified users and optionally yourself . Second, we provide an API where you can sample the next simulated user turn of a conversation for a given set of profiles and a scenario. The playground can be accessed via persimmon.humansand.ai https://persimmon.humansand.ai , and access to both of these platforms can be requested, with a set of pre-sampled conversations available until approval. In the interest of supporting research that leverages these models, we encourage academic applicants to request an API credit grant when they apply for API access. Note that we may reach out to you to learn more about your intended use case before deciding if it is one that we are ready to support. Limitations and Responsible Use limitations-and-responsible-use Unlike most assistant models, our model is designed to match human behavior. The guardrails typically used for assistant models are incompatible with broad human modeling. For more details, see safety and security in the model card persimmon-model-card.html safety-security . This model may reflect human biases and has limitations in privacy and fairness. We limit distribution to avoid misuse for impersonation, privacy violations, exploitative manipulation, or consequential decisions about real people. We ask developers and users to partner with us in ensuring fair, lawful, and responsible use. See our Terms of Service https://persimmon.humansand.ai/docs/term of service and Privacy Notice https://persimmon.humansand.ai/docs/privacy notice . Building Models of Humans, for Humans building-models-for-humans We are excited to share this model with the world, but there is still an incredible amount of work to do to build truly general user models that model how we act. For example, while the main goal of our user models is not to be the most capable agents, it is also impossible to fully model human behavior without modeling how people work in specialized settings and interact with the real, external world. We are also excited to build user models that are capable of rolling out conversations that are plausible and coherent for weeks and months, even years. In addition, we have no distribution matching evaluations with other modalities or languages, limiting who we trust the user model to represent. Finally, to model some of the things people say and do, it is necessary to also model how they think, learn, or interact with the world; we have early results in these directions that we aim to build on. Our experiments also suggest that scaling user model training across thousands of GPUs produces increasingly generalizable behavior. The humans& team believes good user models can give us a window into how humans and AI might collaborate. These models can help us study the impact of technology and build human-centric technology. With Persimmon, we can make a focused effort to build AIs that prioritize the agency and long-term outcomes of people. Join Us join-us If you are excited about learning what better user models can teach us, building collaborative models that learn to work with people and organizations, or developing open-ended long-horizon environments for them to learn in, join us https://jobs.ashbyhq.com/humans-and . Persimmon v0.1 Read the model card persimmon-model-card.html Architecture, intended uses, evaluations, limitations, and safety results.