{"slug": "captchas-can-still-detect-ai-agents", "title": "CAPTCHAs can still detect AI agents", "summary": "CAPTCHAs remain effective at distinguishing humans from AI agents despite AI systems matching or exceeding human performance on the tasks. Researchers at Roundtable AI developed CogCAPTCHA30, a 30-task battery combining traditional CAPTCHAs with cognitive psychology tests, and found statistically significant differences in how humans and AI solve problems—including error patterns, click sequences, and direction changes—even when both achieve similar results. The findings challenge the classic Turing Test by showing that behavioral output alone is insufficient for detecting AI, and that analyzing cognitive process features can reliably identify bots.", "body_md": "AI systems now match and exceed humans on many tasks, but behave through measurably different cognitive processes. This gap can be exploited to detect AI agents and online bots.\n\n\"CAPTCHAs are broken these days.\" AI can easily identify all the traffic lights in a static grid. So CAPTCHAs don't provide a valuable human signal, right?\n\nYes and no.\n\nYes, because vision language models (VLMs) can recognize images like chimneys, fire hydrants, and traffic lights. Deep learning \"solved\" CAPTCHA-style image classification in the early 2010s.\n\nNo, because AI does not complete CAPTCHAs like humans. If you look across all the data of humans and AI completing\nCAPTCHAs, you start noticing differences in features like error patterns. [Our recent paper](https://arxiv.org/pdf/2605.06524) found statistically significant differences across\nsequential click patterns, direction changes, and overselection behavior - features that define how a participant,\nagent or human, would solve the CAPTCHA problem. In other words, AI can solve CAPTCHAs, but they don't solve them\nlike humans.\n\nThe Turing Test - originally proposed in 1950 by Alan Turing - offers a simple criterion for machine intelligence. If\na judge cannot reliably distinguish a machine's responses from a human's, the machine can be considered\n*intelligent*.\n\nTuring understood this behavioral criterion was a concession and not the end-all-be-all of human vs. machine intelligence. He had to concede: the question is too difficult, abstract, and loaded. Behavioral indistinguishability provided a more tractable condition, and one that seemed like a good North Star in the 1950s.\n\nFollowing Turing's footsteps of defining an adversarially robust discriminator that can separate humans from bots,\nwe designed CogCAPTCHA30. This goes one level deeper than the Turing Test, from exploring *output* (what\nhumans and agents can do) to *process* (how it can do it). CogCAPTCHA30 combines the original CAPTCHA with 29\nclassic cognitive psychology tasks for a 30-task battery.\n\nWe recruited human participants and also deployed AI agents to perform these tasks. The CAPTCHA experiment\ndemonstrated that humans and agents can perform at similar performance (*output*) levels, but with different\n*processes*. We then measured *output equivalence - how* (how similar their answers were)\nand*process equivalence* (how they arrived at their answers) across the whole 30-task paradigm and found that they were uncorrelated:\n\nWhile the classic Turing test measures whether a machine produces output indistinguishable from a human, we\npropose a *Process Turing Test* measuring whether machines produce a process indistinguishable from humans.\n\nOur results raise two questions: what types of language models - if any - are like humans, and how adversarially robust is this discrimination process?\n\nTo answer the first question, we compared the distance between humans and state-of-the-art frontier models (OpenAI's GPT, Anthropic's Claude, Google DeepMind's Gemini) as well as Qwen (an open-source 1.5B foundation model) and Centaur (an open-source 70B-parameter foundation model of human cognition).\n\nWe found that state-of-the-art frontier models (Claude, GPT, Gemini) have less similar human process features\ncompared to smaller models (Qwen, Centaur). As we argued in [AI Capability isn't Humanness](https://research.roundtable.ai/capabilities-humanness/), while\nfrontier models are becoming more powerful over time, they are not necessarily becoming more human. Contemporary\nprogress in artificial intelligence is independent of progress in human simulation.\n\nQwen, a smaller open-source model, is more humanlike than the larger Claude, GPT, and Gemini. And, as a nice validation, Centaur outperforms the other models in similarity to human process feature space. We hypothesize this is due to large-scale output fine-tuning, specifically 10M+ human choices across 160 cognitive experiments.\n\nThis introduces the second question: how adversarially robust is the process to discriminate humans from agents? Any behavioral feature used to distinguish the two may itself become a target for optimization. Accordingly, a detector that succeeds against off-the-shelf agents establishes a behavioral gap only under the current attacker model - how AI exists and operates now. It's to be seen whether it can become a durable human-verification signal for the future technologies. This motivates a stronger test: can an agent close the process gap - between how humans and agents complete tasks - when given increasingly direct access to human data?\n\nWe fine-tuned a Qwen2.5 Instruct model to bring it closer to humans. When given full information - the observed\nfeatures and the discriminator's objective function - the gap between humans and agents disappears. However, the gap\nreappears when parts of the feature space are left out and fully returns when agents have to generalize\ncross-task. In other words, the *Process Turing Test* is robust when the AI does not have full access to the\ndiscriminator and the feature set (i.e., the model does not know *how* it will be evaluated).\n\nThe challenge the *Process Turing Test* poses is whether AI can continuously replicate all of human cognitive\npsychology. Despite the anxiety that models are becoming more capable over time, they are empirically not becoming\nmore *humanlike*. Compared to one-time checks like passwords, CAPTCHAs, document identification, and device\nfingerprinting, the Process Turing Test provides a step-up function in human verification. Simulating human\ncognitive psychology is an exponentially more challenging task.\n\nMayank Agrawal, Milena Rmus, and Mathew Hardy work at Roundtable Technologies Inc., where they are building Proof of Human, an invisible authentication system for the web. Previously, they completed PhDs in cognitive science at Princeton University (Mayank and Matt) and the University of California, Berkeley (Milena).", "url": "https://wpnews.pro/news/captchas-can-still-detect-ai-agents", "canonical_source": "https://research.roundtable.ai/captchas-detect-ai/", "published_at": "2026-05-29 15:57:37+00:00", "updated_at": "2026-05-29 16:19:48.717733+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "computer-vision", "ai-research"], "entities": ["Alan Turing", "CAPTCHA"], "alternates": {"html": "https://wpnews.pro/news/captchas-can-still-detect-ai-agents", "markdown": "https://wpnews.pro/news/captchas-can-still-detect-ai-agents.md", "text": "https://wpnews.pro/news/captchas-can-still-detect-ai-agents.txt", "jsonld": "https://wpnews.pro/news/captchas-can-still-detect-ai-agents.jsonld"}}