Soldiers can refuse to commit war crimes. Can AI?

The Pentagon's agreement with OpenAI specifies that the Department of Defense will receive "OpenAI models that are designed for national security use cases and have minimal refusal rates," according to new reporting from The Intercept, after Anthropic lost a major government deal for declining to loosen limits on military use of its technology. The provision conflicts with the US military's legal "duty to disobey" clearly illegal orders, codified in the Department of Defense Law of War Manual and recognized internationally after the Nuremberg Trials, raising concerns that low-refusal AI models could make it harder to stop war crimes before they are committed.

Who gets to tell the military “no”? Soldiers can refuse to commit war crimes. Can AI? The Pentagon wants AI models that can’t say no. Kylie Jones https://www.vox.com/authors/kylie-jones is a Future Perfect fellow writing about the future of fighting: the intersection of weapons technologies, international politics, and the actors shaping it all. That question has become even more important after the Pentagon pushed AI companies to loosen limits https://www.vox.com/future-perfect/500896/lethal-autonomous-weapons-ai-ukraine-appropriate-human-judgment on how their technology may be used in war — including how much human control must remain over autonomous weapons. When Anthropic said no, it lost a major government deal. OpenAI stepped in, embracing the Pentagon’s more flexible standard instead. Now, thanks to new reporting https://theintercept.com/2026/09/08/pentagon-openai-military-contract/ from The Intercept, we’re getting a better look at what those negotiations actually entailed. The OpenAI-Pentagon agreement https://drive.google.com/file/d/1Ek-xpuaXUURWkpTdfXBB2C60tnmZgZOK/view included specific language on the type of AI models the DOD would receive: “OpenAI models that are designed for national security use cases and have minimal refusal rates.” That phrase — “minimal refusal rates” — means that the Pentagon is seeking models that rarely, if ever, tell it no. According to The Intercept, a Justice Department lawyer representing the Pentagon confirmed this language, before backtracking only a few hours later. On the surface, that provision might seem reasonable. The military is a strictly hierarchical institution. When your senior gives you an order, you do it. Leaders generally have more battlefield wisdom and experience than their subordinates, so the rank and file can usually assume that their commanders know what they’re doing and follow along. Plus, in the high-stress, time-sensitive environments of most military operations, there is little room for discussion, explanation, or alternative proposals. So, if soldiers can’t refuse an order, why would a machine be allowed to? Who is a robot to tell its commander no? There’s just one issue. Soldiers can refuse an order — when the order is illegal. In fact, they are legally required to. But by introducing technologies with a limited ability to challenge commands, the military may be making it harder to stop war crimes before they are committed. Humans have a duty to disobe y As outlined in US domestic, military, and international law https://ihl-databases.icrc.org/en/customary-ihl/v2/rule155 , soldiers are only obligated to follow lawful orders, and they have a duty to refuse unlawful ones. The Department of Defense’s Law of War Manual https://media.defense.gov/2023/Jul/31/2003271432/-1/-1/0/DOD-LAW-OF-WAR-MANUAL-JUNE-2015-UPDATED-JULY%202023.PDF page=1115 is unambiguous: “Each member of the armed services has a duty to: 1 comply with the law of war in good faith; and 2 refuse to comply with clearly illegal orders to commit violations of the law of war.” Obeying an illegal order — as in a directive that violates the Constitution, US laws, international law, or directs one to commit a crime — can expose a service member, as well as their commander, to criminal liability. This “duty to disobey” has foundations in early British and American common law, but it achieved international recognition after the Nuremberg Trials against Nazi soldiers after WWII. In those trials, many Nazi soldiers argued they were not liable for crimes they committed during the Holocaust because they were simply following their superiors’ orders. Rudolf Hoess, the commander at the largest Nazi concentration camp, Auschwitz, himself explained https://archive.org/details/psychologyofdict0000gmgi/page/255/mode/1up : “In Germany it was understood that if something went wrong, then the man who gave the orders was responsible. So I didn’t think that I would ever have to answer for it myself.” These defenses were unequivocally rejected by the court. The tribunal held that obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ ordinary sense and understanding <https://jsc.defense.gov/Portals/99/2024%20MCM%20files/MCM%20 2024%20ed %20 2024 01 02 %20 adjusted%20bookmarks .pdf?ver=WLZvJg--lbaFtAC5qOM1uA%3D%3D page=181 .” If “I was just following orders” was an absolute defense, everyone involved in a crime would escape accountability, with the exception of the single highest-ranking officer. Instead, the court determined that individual moral and legal responsibility supersedes national or military chains of command for illegal acts. Obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ordinary sense and understanding.” But, when it comes to agentic AI systems or AI-assisted weapons, increasingly being used in place of human soldiers, does the same logic apply? And, perhaps more importantly, should it? …But do machines? When you use an AI chatbot, it does not simply reject queries it may not “agree” with. It does not say no to requests it doesn’t “like.” Instead, it follows a set of established protocols, much as a soldier should follow the law of war: It refuses requests that violate its terms of service. The military is using AI for far more consequential applications than a layperson using ChatGPT to, say, build a travel itinerary. So, what kind of world is created by the Pentagon’s request for AI that “minimally refuses” its directives, many of which may bear grave ethical, moral, or legal implications? Rebecca Crootof https://law.richmond.edu/faculty/rcrootof/ , an expert on technology law and professor at the University of Richmond School of Law, told Vox that while the language could easily be read as increasing the likelihood of compliance with unlawful orders, or at least reducing the friction around them, “Minimal refusal doesn’t mean no refusal.” “I both want to say it’s not definitely going to mean compliance with unlawful orders,” Crootof said, “but also acknowledge that identifying and evaluating what constitutes an unlawful order in the moment can be incredibly difficult.” Even for human soldiers, disobeying illegal orders is neither simple nor easy. To the contrary, it doesn’t even happen that frequently. In fact, some https://ilr.law.uiowa.edu/print/volume-107-issue-5/abuse-by-authority-the-hidden-harm-of-illegal-orders legal scholars have argued that giving illegal directives in and of itself is an act of abuse against subordinates, who are not thoroughly trained in legal nuances, are conditioned to obey, and face prosecution if they make the wrong choice. Still, the mere fact that soldiers can disobey is an important constraint on commanders who might let battlefield conditions, or even commands coming from their own superiors, erode their ethics and commitment to the law. Those who are optimistic about AI often argue that machines could be better at interpreting and applying the law than human soldiers. Machines can be trained on the law itself — drawing on a repository of legal cases, rulings, and battlefield data — which might make them more “knowledgeable” than an average soldier. And machines do not face the same pressures as soldiers facing prosecution for making mistakes, since they can’t exactly be held accountable; a problem to unpack another time. That, in turn, might make it easier for them to refuse illegal orders in the proper conditions. So, is the answer as simple as letting the machines decide? Not exactly. “This has been a desire and a drive by technologists for ages,” Crootof said. “What if we just put the law in the system and have it only do things that are lawful? Isn’t that going to solve things?” Interpreting and applying the law, though, is not a simple legal-illegal binary. “It’s very context-specific,” Crootof told me, “and honestly, that’s the kind of thing humans generally find difficult and AI decision-making systems are even worse at.” Often, determining the legality of an act comes down to subtle nuances you can’t program for: You’re allowed to target a combatant; you’re not allowed to target a wounded or surrendering combatant. You’re not allowed to target a civilian; you are allowed to target a civilian directly participating in hostilities. The difference between a legal target and an illegal one may come down to body language, clothing choices, or even battlefield intuition. As Crootof explained, “These are not the kinds of contextual analyses AI is good for.” So what’s the solution? Where human soldiers’ power to disobey would traditionally serve as an important check on the power of military commanders, if an imperfect one, replacing soldiers with machines that don’t have such safeguards removes one last layer of judgment over the use of force. And yet, machines themselves are not always capable of making the judgment calls that go into interpreting the law, so it may not be appropriate to let AI have the final say either. So what’s the solution? Given the way AI technology has already been thoroughly integrated into the military, Crootof told me it may be somewhere in the middle. Rather than programming AI to refuse commands it interprets as illegal, Crootof said, “what it could be designed to do is flag those indeterminate zones, those gray-zone analyses, for human review or for a higher authority.” When a system picks up conflicting signals — an enemy soldier who looks like he may be surrendering, for example — it could prompt the commander to confirm their selection or remind them of applicable law. In this way, a check on commanders’ battlefield authority could be preserved without fully ceding judgment over the use of force to the machine. We don’t know yet whether OpenAI has equipped its “minimal refusal” models with such capabilities. But as those technologies become increasingly central to military decision-making, the public deserves answers sooner rather than later.