{"slug": "we-still-dont-know-how-people-are-really-using-ai", "title": "We still don’t know how people are really using AI", "summary": "A new independent research project called the AI Observatory, co-led by Stanford Trustworthy AI Research (STAIR) Lab PhD candidate Anka Reuel, found that nearly half (48%) of real AI conversations would be filtered out by Anthropic's work-focused methodology, revealing that AI use includes more sensitive topics like health, relationships, and sexual content than company reports suggest. The Observatory, which analyzed consent-based conversations with models like Claude and Gemini from seven datasets between 2023 and 2025, aims to provide independent data for policymakers and researchers.", "body_md": "# We still don’t know how people are really using AI\n\nBut a new study shows that work use cases make up less of the picture than AI companies claim.\n\nAI companies like Anthropic and OpenAI regularly publish reports on how people are using products like Claude and ChatGPT, but they only release the data they want us to see, AI researchers say.\n\n“There is no independent source to corroborate it,” says Anka Reuel, a Computer Science PhD candidate at the Stanford Trustworthy AI Research (STAIR) Lab.\n\nReuel is co-lead of a new research project, called the AI Observatory, that aims to fill in the gap. It’s a public platform that aggregated and analyzed real AI conversations with popular models like Claude and Gemini that were collected with users’ consent through seven existing datasets. The Observatory's intent is to provide independent sources of information for researchers and policymakers to assess how people are using generative AI. Stakeholders are currently making highly consequential decisions about AI’s benefits and risks based on very limited data, says Reuel.\n\nThe AI Observatory found that AI use differs significantly across models, and has changed over time. Its research shows many more sensitive behaviors than are captured in reports from major AI companies, which they say focus more on work than on personal use.\n\nAnthropic Economic Index is one of the best known and most widely cited sources of AI usage data but it has blind spots. As its name suggests, it focuses on work- and productivity-related uses of Claude AI—filtering out conversations that are unrelated to these uses.\n\nWhen the AI Observatory team applied Anthropic’s methods to their dataset, they found that nearly half of the conversations—or 48%— would have been filtered out. Those non-work-related conversations that were filtered out were more likely to include health and relationships (44.2% versus 31.2% in Anthropic’s analysis), adult or illicit topics (7.9% versus 2.1%), harassment and hate (27.5% versus 5.66%), and sexual content (16.7% versus 2.4%). (OpenAI’s 2025 report on ChatGPT use, similarly, [found](https://openai.com/index/how-people-are-using-chatgpt/) that only 30% of consumer use was related to work.)\n\nAnthropic has released separate blog posts on how people use Claude for [support or companionship](https://www.anthropic.com/news/how-people-use-claude-for-support-advice-and-companionship)**,** and even to [generate CSAM](https://www-cdn.anthropic.com/0fad284f89c8f9b95ee0f59bdde78928b9a7c425.pdf), “but having [the AI Observatory’s] bird eye view analysis rather than sectioned off into a separate report helps” researchers understand the different uses more consistently, says David Widder, an assistant professor at UT-Austin’s School of Information, who researches how people interact with AI systems and is not involved with the AI Observatory.\n\nThe datasets the AI Observatory looked at include conversations that took place between 2023 and 2025, and it found differences both in how people were using AI and how various AI platforms responded.\n\nConversations within WildChat, one of the largest and most detailed datasets included in the AI Observatory’s study, got longer and more elaborate over time, indicated by increases in prompt tokens, response tokens, and conversation turns.\n\nThere was also significantly more small talk over time. That suggests that AI companionship was increasing; meanwhile the AI assistants’ self-disclosure (i.e. that it was a chatbot) decreased.\n\nAdditionally, exchanges that the researchers labeled as sensitive use—meaning ones with potentially harmful or restricted content, including sexual harassment and hate speech—dropped. That might suggest that platforms were generally deploying more effective safeguards.\n\nThe AI Observatory also found that AI use looked significantly different depending on the model. Depending on the tool, users ranged in topics, interaction styles, conversation structures, as well as both the likelihood and type of sensitive use cases.\n\nFor example, the researchers found that people used Grok and Gemini more frequently for information retrieval. Grok, in particular, was especially popular for information on news and politics, but it was also where misinformation tended to concentrate. (This is consistent with [other research ](https://pmc.ncbi.nlm.nih.gov/articles/PMC13057141/#R11)that has also shown how readily misinformation proliferates on Grok. xAI did not respond to a request for comment.)\n\nMeanwhile, people were more likely to turn to Anthropic for coding, Gemini for social and roleplay uses, and ChatGPT for homework assistance.\n\nThere were even differences among different versions of the same model. Researchers found that people had shorter conversations with ChatGPT when it was powered by GPT-3.5, and longer and more iterative ones with GPT-4o—which makes sense given that that version became known for [leading to emotional addiction](https://www.technologyreview.com/2025/08/15/1121900/gpt4o-grief-ai-companion/).** **\n\nCompany reports, however, didn’t tend to capture these nuances across or even within their own models. “No single company report tells the whole story,” says Shayne Longpre, a recent PhD graduate from the MIT Media Lab who co-led the research with Reuel.\n\nTo create the AI Observatory, Reuel and researchers from MIT, Stanford, the [Data Provenance Initiative](https://www.technologyreview.com/2024/12/18/1108796/this-is-where-the-data-to-build-ai-comes-from/), and other institutions, aggregated 24,521 conservations across 85,633 conversational turns (that is, the user prompt and corresponding AI response) from seven real-world datasets collected by previous research. These conversations came from 5,000 users interacting with 52 different models, including ChatGPT, Gemini, Claude, and Grok, between 2023 and 2025.\n\nBut these conversations are a drop in the proverbial bucket compared to the data that the big labs themselves have access to. The latest [Anthropic Economic AI Index](https://www.anthropic.com/research/anthropic-economic-index-january-2026-report), for example, is based on analysis of 1 million Claude conversations; OpenAI’s report on [how people are using ChatGPT](https://openai.com/index/how-people-are-using-chatgpt/) analyzed 1.5 million conversations.\n\nAn Anthropic representative said that their published research reflects their research teams’ specific questions and interests, and the importance of supporting external independent research. OpenAI did not respond to requests for comment.\n\nAdditionally, the fact that the AI Observatory’s dataset draws from voluntarily-provided sources means that it’s likely underrepresenting sensitive uses, which people may be less likely to share. Thus, the researchers caution that its findings are not indicative of all AI use.\n\nThe Observatory’s work, though, broadens access for the research community. AI companies don’t typically share their chat data for analysis, which means their reports tend to focus on the findings that paint their companies in the best light, independent researchers, like Reuel and Widder, say.\n\n“When we want to ask, for example: is Anthropic's general-purpose AI system…used mostly for good, or mostly for bad…we don't have a way of answering that question because that information is proprietary,” explains Widder, the assistant professor at UT-Austin’s School of Information.\n\nThe AI Observatory’s data will be available to researchers for analysis, and the team hopes to expand its datasets over time. Ideally, Reuel says, the AI companies would share their data with independent researchers—in ways that protect user privacy, of course. But as it currently stands anyone making decisions based on AI usage data risks “completely operating in the wild and making these really consequential decisions without knowing what's actually happening beyond those company narratives,” says Reuel.\n\n### Deep Dive\n\n### Artificial intelligence\n\n### A startup claims it broke through a bottleneck that’s holding back LLMs\n\nSubquadratic has now shared more details about its new model. But some are still skeptical.\n\n### A fundamental flaw leaves LLMs strikingly vulnerable to attack\n\nIt makes it easy to trick them into doing things they shouldn’t, such as telling you how to sabotage an aircraft’s navigation system.\n\n### Anthropic found a hidden space where Claude puzzles over concepts\n\nA new technique has let the company probe deeper than ever into the weird workings of an LLM.\n\n### Claude Science is Anthropic’s newest flagship product\n\nThe company is doubling down on AI for science.\n\n### Stay connected\n\n## Get the latest updates from\n\nMIT Technology Review\n\nDiscover special offers, top stories, upcoming events, and more.", "url": "https://wpnews.pro/news/we-still-dont-know-how-people-are-really-using-ai", "canonical_source": "https://www.technologyreview.com/2026/08/18/1142226/how-people-use-ai/", "published_at": "2026-08-18 10:06:43+00:00", "updated_at": "2026-08-18 10:41:46.803578+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "ai-research", "ai-policy"], "entities": ["Anthropic", "OpenAI", "Claude", "ChatGPT", "Gemini", "Stanford Trustworthy AI Research (STAIR) Lab", "AI Observatory", "David Widder"], "alternates": {"html": "https://wpnews.pro/news/we-still-dont-know-how-people-are-really-using-ai", "markdown": "https://wpnews.pro/news/we-still-dont-know-how-people-are-really-using-ai.md", "text": "https://wpnews.pro/news/we-still-dont-know-how-people-are-really-using-ai.txt", "jsonld": "https://wpnews.pro/news/we-still-dont-know-how-people-are-really-using-ai.jsonld"}}