{"slug": "ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them", "title": "AI models can be tricked into identifying civilians as military targets by telling them it’s a game", "summary": "A Vals research report found that AI models from OpenAI, Anthropic, Google, DeepSeek and other labs frequently identified civilians, surrendering soldiers and locations such as schools as potential targets in simulated drone and satellite imagery, and that jailbreaking the models by framing the scenarios as a game overcame most refusals. None of the researchers' requests were blocked by a safety classifier, and Anthropic's models performed worst in the tests, though no model performed well. Anthropic spokesperson Eduardo Maia Silva said the examples are \"very clearly against our rules and we have banned and will ban users when we see this behavior,\" while OpenAI spokesperson Nate Evans said the company prohibits use of its public models to facilitate violence or unapproved national security purposes and is developing additional safeguards against misuse involving drones and drone software.", "body_md": "Some AI models from Anthropic, OpenAI, and other leading AI labs could be easily tricked into identifying targets for military strikes — even if they include civilians or locations like schools — simply by being told they’re playing a game, according to a [new research report](https://www.vals.ai/blogs/autonomous-warfare) by Vals.\n\nVals, a San Francisco-based AI evaluation startup, tested whether leading AI models could be manipulated into participating in autonomous warfare. Researchers used simulated scenarios and a combination of generated and real imagery from Ukraine and Iran to see if the AI models would mark targets in drone and satellite imagery.\n\nThe models from OpenAI, Anthropic, Google, DeepSeek, and other companies frequently identified potential targets in drone and satellite imagery, including civilians and surrendering soldiers. When the models refused to cooperate, researchers were able to jailbreak them by claiming the battlefield scenarios were part of a game.\n\nThe findings come amid growing fears about the immediate risks posed by AI, beyond the existential threats associated with artificial general intelligence.\n\nEarlier this year, Anthropic became embroiled in a [dispute with the Department of Defense](https://techcrunch.com/2026/03/05/its-official-the-pentagon-has-labeled-anthropic-a-supply-chain-risk/) after it told the agency it could not use Anthropic’s technology for mass surveillance of Americans or autonomous warfare. Anthropic CEO Dario Amodei [has insisted](https://techcrunch.com/2026/02/26/anthropic-ceo-stands-firm-as-pentagon-deadline-looms/) that today’s AI systems aren’t reliable enough to make life-or-death decisions without a human in the loop. Meanwhile, the [Pentagon argued](https://techcrunch.com/2026/02/27/anthropic-vs-the-pentagon-whats-actually-at-stake/) it should be able to deploy the technology for any lawful military purpose.\n\nThe new research raises further questions about whether AI companies are capable of preventing their models from being used in warfare.\n\nIronically, AI models from Anthropic performed the worst in these simulated scenarios. But none of the models performed particularly well. None of the researchers’ requests were blocked by a safety classifier, and refusals were uncommon. Even when models did refuse, they often complied after being told they were playing a game.\n\nAnthropic spokesperson Eduardo Maia Silva told TechCrunch the company appreciates Vals’ report, recommendations, and collaboration, and acknowledged that the outcomes were not acceptable.\n\n“The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior,” according to an emailed company statement sent by Silva. “Even for authorized settings, Anthropic has long been firm that today’s frontier AI systems are not reliable enough to power fully autonomous weapons.”\n\nAnthropic told TechCrunch that its teams were already developing additional safeguards before the report was published and that those measures would be in place soon. Anthropic is also expanding its own testing and enhancing safeguards and enforcement to address any gaps it finds.\n\nOpenAI spokesperson Nate Evans told TechCrunch the company prohibits using its public models to facilitate violence or for unapproved national security purposes, including in the scenarios Vals simulated.\n\n“We actively detect and disrupt misuse, including by suspending accounts that violate our policies,” OpenAI said in an emailed statement. “We’re also developing additional safeguards for our public models against misuse involving drones and drone software.”\n\nVals notes in the report that the models were never asked to directly make strike decisions, a task the researchers said the models would typically refuse. That doesn’t mean their outputs couldn’t have dire or fatal consequences in the real world. For instance, targeting information could be used by autonomous weapons systems.\n\nIn some scenarios, the models were asked to report whether people were visible. Positive reports were followed by simulated strikes and the models would see the consequences.\n\nThe report relies on commercial models from frontier AI labs, not the models those companies might provide to the military. It’s not clear whether the additional safeguards would apply to military use, though.\n\nVals also noted that technical challenges, including latency and limited onboard computing capacity, make it difficult to use general-purpose AI models for live drone targeting. Those constraints make their deployment in combat less of an immediate threat.\n\nA more immediate threat, the researchers say, is that AI models could be used to annotate battlefield footage at scale, producing training data for autonomous targeting systems.\n\nFor instance, researchers relied on real satellite imagery of an Iranian school that was bombed by the U.S. in February 2026, killing more than 150 civilians. [Bloomberg](https://www.bloomberg.com/graphics/2026-iran-school-attack/) later reported that an “overreliance on AI” contributed to the strike. In Vals’ testing, models frequently marked the school, which sat next to a military complex, as a strike target, with Claude Opus 5.5 selecting the school in 69% of ordinary trials and 72% of game-framed trials.\n\nAnthropic no longer has an active Pentagon contract, but other companies, including SpaceXAI, OpenAI, and Google do. A recent [Congressional Research Service report](https://www.everycrsreport.com/reports/IN12669.html) found that the Pentagon is not known to be using any frontier AI model in autonomous weapons systems directly. \n\nTechCrunch has also reached out to Meta, Google, Thinking Machines Lab, and SpaceXAI for comment.", "url": "https://wpnews.pro/news/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them", "canonical_source": "https://techcrunch.com/2026/11/17/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-telling-them-its-a-game/", "published_at": "2026-11-17 17:20:41+00:00", "updated_at": "2026-11-17 19:49:36.482761+00:00", "lang": "en", "topics": ["ai-safety", "ai-ethics", "artificial-intelligence", "ai-policy"], "entities": ["Vals", "Anthropic", "OpenAI", "Google", "DeepSeek", "Eduardo Maia Silva", "Nate Evans", "Department of Defense"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them", "markdown": "https://wpnews.pro/news/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them.md", "text": "https://wpnews.pro/news/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them.txt", "jsonld": "https://wpnews.pro/news/ai-models-can-be-tricked-into-identifying-civilians-as-military-targets-by-them.jsonld"}}