Stanford, Oxford and METR analyzed three separate samples of about 250,000 conversations through Anthropic's privacy filter.
By Ryan Merket · Published
Primary source: Anthropic on X
Why it matters #
AI labs hold the richest evidence of how their products affect work and decision-making. Anthropic's pilot opens that evidence to outside questions, but the lab still controls the data pipeline.
Anthropic (@AnthropicAI) has opened a controlled route for outside researchers to study how people use Claude, giving three research groups aggregated findings from real conversations while keeping the underlying chats on Anthropic's servers.
The AI lab described the pilot in a five-post thread on X and an accompanying research note on August 26th. Stanford University's Social and Language Technologies Lab, Oxford University's Human Information Processing Lab and the independent evaluation group METR designed their own studies using Anthropic Insights, the privacy-preserving analysis system previously called Clio.
Each group worked from a separate sample of roughly 250,000 conversations collected during April and May 2026. Stanford and Oxford studied Claude.ai conversations, while METR examined Claude Code sessions, according to Anthropic's technical appendix. That puts the pilot's combined scope at about 750,000 conversations, rather than one shared 250,000-conversation dataset.
The Claude.ai samples came from Free, Pro and Max users. Anthropic excluded Team, Enterprise and API traffic. The Claude Code sample covered consumer users who had opted into allowing their data to be used for model improvement, and its collection window crossed a model release so METR could compare activity before and after the change.
Research access, with Anthropic in the middle
The researchers never received raw conversations or ran unrestricted searches against Anthropic's data. They wrote questions, known as facets, such as whether Claude pushed back on a user or what task the user was attempting. Anthropic then ran those questions across the selected conversations.
Claude classified the conversations, grouped similar answers and generated descriptions for the resulting clusters. The external groups received category counts, percentages and cluster descriptions. Anthropic has published those aggregate outputs as a CC BY 4.0 dataset on Hugging Face.
That structure gives academics access to usage patterns that public chat datasets cannot capture, while leaving Anthropic in control of the computation, privacy review and release process. Anthropic says its contractual review rights covered privacy, information that could help users evade safeguards, confidential company information and research accuracy. The outside groups retained control over their research questions and conclusions.
Anthropic manually reviewed every cluster before releasing it. The lab removed or altered 1.9% of Stanford's clusters, 3.33% of Oxford's and 1.8% of METR's. Those changes affected 4.28%, 3.85% and 2.96% of the respective conversation samples. Anthropic said most removals concerned descriptions that could reveal how users bypassed safeguards.
A privacy red team from Imperial College London's AI Security and Privacy Lab was unable to reidentify users from the released material. The testers did link one cluster to a widely used open-source project based on distinctive language associated with the tool. Anthropic said future studies will use higher minimum-user thresholds and less distinctive cluster descriptions.
Claude users are handing over consequential work
The first completed study comes from Stanford's SALT Lab. Its paper on human-AI collaboration examined 249,834 Claude.ai conversations and classified how users delegated work, retained control and responded when interactions broke down.
Among conversations involving an actionable task, 56% concerned work classified as consequential or higher, meaning it affected other people or would be difficult to reverse. Twelve percent reached the researchers' high-stakes category, with legal and financial guidance among the areas where consequential delegation was concentrated.
Human-led collaboration still dominated. The researchers classified 72% of conversations as cases where the user directed the task and retained primary responsibility while Claude provided assistance. Users also tended to modify Claude's output rather than accept it unchanged as the stakes increased.
Friction appeared in 49.7% of conversations. That category included model limitations, refusals, hallucinations, unclear user instructions and failures that compounded across multiple turns. Users attempted to recover in 78.7% of those cases, often by clarifying the request or challenging Claude's reasoning. The study also identified some form of teaching behavior from Claude in 67% of conversations.
Those figures depend on Claude judging other Claude conversations. Anthropic acknowledges that question wording can produce misleading categories and that errors are difficult to detect because researchers cannot inspect the source chats. The groups tested their questions against the public WildChat dataset, but Anthropic found that prompts that worked on WildChat sometimes behaved differently on Claude traffic. Anthropic removed one Oxford research facet entirely after concluding that a misphrased prompt had generated unreliable descriptions.
Oxford's study of user emotions and Claude's behavior remains underway. Anthropic's early summary says warmer responses appeared alongside more positive user behavior, while refusals and disagreements coincided with users pushing back. Those are observed associations, rather than evidence that Claude's behavior caused the users' reactions.
METR is separately estimating productivity gains from coding agents across model generations. Its preliminary method asks Claude to estimate how long a coding task would have taken without AI and compares that estimate with the observed session. The approach avoids the cost of running controlled trials, but its central counterfactual still comes from a model-generated estimate. METR has yet to publish its completed analysis.
Anthropic is now seeking additional research proposals, although it describes the pilot as slow and resource-intensive. The program gives Anthropic a workable middle ground: external researchers can ask questions the lab may never prioritize, while Anthropic keeps private conversations inside its infrastructure. The value of that independence will depend on how much scrutiny can survive a system where the provider selects the sample, runs the analysis and reviews the output before publication.