{"slug": "ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks", "title": "AI Models Can Tell When Researchers Plant Concepts in Their Neural Networks", "summary": "Two of the largest openly available models, Alibaba's Qwen3-235B and Meta's Llama 3.1 405B, flagged researcher-injected concept patterns in their neural networks up to 53.9 percent and 31.7 percent of the time respectively, according to an arXiv study by New York University and Anthropic philosopher Harvey Lederman and University of Texas linguist Kyle Mahowald. In unmodified trials neither model claimed a single injection, and correct naming of the injected concept occurred in only up to 13.9 percent of Qwen trials and 12.9 percent of Llama trials, which the authors attribute to a side effect of the technique rather than genuine knowledge of what was injected. The work follows Anthropic researcher Jack Lindsey's 2025 report, in which the best-performing models detected injections and named the concept in about 20 percent of cases.", "body_md": "[Artificial Intelligence](https://singularityhub.com/category/artificial-intelligence/)\n\n# AI Models Can Tell When Researchers Plant Concepts in Their Neural Networks. Is It Self-Reflection?\n\nSome researchers say the findings show the glimmers of AI introspection—others aren't so sure.\n\nImage Credit\n\nMirella Callage on Unsplash\n\nShare\n\nAI models can produce convincing descriptions of their reasoning, but there's little way to tell whether those explanations reflect what’s actually happening inside. An emerging strand of research, however, suggests some models can detect when their internal activity has been tampered with, potentially suggesting some degree of self-reflection.\n\nWhether AI can introspect has become a controversial question in both AI safety and cognitive science. If models could reliably describe what's going on inside their own neural networks, it would give researchers a powerful new way to check what they’re really up to.\n\nBut there is widespread skepticism, and the idea is hard to test. Large language models are trained on vast amounts of text, much of which contains descriptions of human introspection. So AI has learned to talk convincingly about its inner life, regardless of whether those descriptions are grounded in anything.\n\nHowever, a growing body of research now suggests models can tell when their neural activity has been altered by researchers. While it’s less clear whether they can detect exactly what’s changed, some researchers say the findings indicate at least a limited ability to reflect on their internal states.\n\n“The fact that [an AI model] can notice the anomaly is insane,” Harvey Lederman, a philosopher at New York University and Anthropic, who co-authored [a study](https://arxiv.org/abs/2603.05414) investigating the phenomenon, [told *Quanta Magazine*](https://www.quantamagazine.org/new-research-tests-whether-ai-models-know-their-own-thoughts-20261104). “That is a mind-bending fact. It is wild, and it does seem robust to our various controls.”\n\nLederman’s study was published on the arXiv preprint server in March, while he was still at the University of Texas. The work is a follow-up to [a 2025 report](https://transformer-circuits.pub/2025/introspection/index.html) by Anthropic researcher Jack Lindsey, which pioneered the idea of injecting activity patterns associated with specific concepts into a model’s network.\n\nIn his experiments, Lindsey had found the best performing models could both detect injections and name the concept in question in about 20 percent of cases. Lederman and his colleague Kyle Mahowald, a linguist at the University of Texas, were skeptical. So they decided to test the idea on two of the largest openly available models, Alibaba's Qwen3-235B and Meta's Llama 3.1 405B.\n\nThey told the models that a researcher might be injecting “thoughts” into its mind and to let them know if it noticed anything. In trials where they artificially amplified patterns related to 821 concepts, Qwen flagged injections up to 53.9 percent of the time and Llama in up to 31.7 percent. In trials where they didn’t make any changes, neither model claimed a single injection.\n\nThe models sometimes named the injected concept correctly too—in up to 13.9 percent of trials for Qwen and 12.9 percent for Llama. But the pair found evidence this had little to do with the models knowing what had been injected.\n\nFor a start, wrong guesses bore no relation to the injected concept. Almost three-quarters of Qwen's wrong guesses were simply “apple.” And when the researchers stopped amplifying a concept before the model began answering, correct guesses dropped sharply while detection rates held steady. That suggests the right answers came from a known side effect of the technique: Injecting a concept tends to make a model talk about it whatever the question.\n\nThe researchers concluded that the ability to detect tampering is distinct from whatever enables models to guess the concept. This idea was backed by [a follow-up paper](https://arxiv.org/pdf/2603.21396) from Lindsey and colleagues published the same month that mapped out how anomaly detection works inside Google's open Gemma 3 model. They suggested the mechanism emerges as raw models are trained into chatbots, and that the ability to identify concepts appears to rely on separate processes.\n\n### Be Part of the Future\n\nSign up to receive top stories about groundbreaking technologies and visionary thinkers from SingularityHub.\n\nNot everyone is convinced the findings amount to introspection though. To probe the results, researchers at New York University revamped the experiment by simply telling a model to become obsessed with a particular concept rather than tampering with its network.\n\nThey then subjected Meta's Llama and Google's Gemma models to both concept injection and their new prompting trick and asked the models whether their internals had been interfered with. The models reported “injected thoughts” more often following the prompt than after a genuine injection.\n\nWhen the researchers gave the models a third option to report that the prompt had been manipulated, they were unable to reliably differentiate between the two interventions. A model genuinely inspecting its own internal states should be able to spot that difference, the researchers argue.\n\nThe entire strand of research also raises thorny philosophical questions. Shauli Ravfogel, one of the authors of the New York University paper, points out that all a model’s workings run on its internal states, so almost anything it does could be dressed up as self-reflection. “If we say that this is introspection, the term ‘introspection’ becomes kind of meaningless,” he told *Quanta*.\n\nOthers argue the work cuts both ways. Steve Fleming, a University College London neuroscientist who studies introspection in humans, told *Quanta* the research suggests cognitive scientists may have oversold how clear the distinction between thinking and thinking about thinking really was.\n\nEither way, the work is turning once purely philosophical questions about self-awareness into experimental ones. But we’re likely still a long way from settling the debate about what’s really going on inside the heads of today’s leading AI models.\n\nRelated Articles\n\n## AI Agents Ran a Biology Lab. Humans Popped in to Empty the Trash.\n\n## What Does the ‘I’ in AI Really Mean?\n\n## Science Is Littered With AI Slop. A New Detector Could Filter It Out.\n\nWhat we’re reading", "url": "https://wpnews.pro/news/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks", "canonical_source": "https://singularityhub.com/2026/11/11/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks-is-it-self-reflection/", "published_at": "2026-11-11 20:06:38+00:00", "updated_at": "2026-11-11 20:21:00.067127+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-safety", "large-language-models", "machine-learning"], "entities": ["Harvey Lederman", "Kyle Mahowald", "Jack Lindsey", "Anthropic", "New York University", "University of Texas", "Alibaba", "Meta"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks", "markdown": "https://wpnews.pro/news/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks.md", "text": "https://wpnews.pro/news/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks.txt", "jsonld": "https://wpnews.pro/news/ai-models-can-tell-when-researchers-plant-concepts-in-their-neural-networks.jsonld"}}