How AI Could Hollow Out the U.S. Military

A growing body of research suggests that AI use can impair human cognition, including automation bias and degraded skills, which poses a threat to human control of the U.S. military as it pursues an 'AI-first warfighting force' under Secretary of Defense Pete Hegseth. The article argues that policymakers should focus on how troops and commanders use AI, not just on AI's independent capabilities, and recommends careful training and monitoring, possibly adopting certain models more slowly.

When people think about the risks of mixing AI and armed forces, their thoughts often drift to one famous film franchise: the Terminator. Its premise is simple. American scientists invent a superintelligent computer system, called Skynet. The Department of Defense then integrates it into the military and embodies its intelligence into lethal robots. Skynet soon becomes self-aware, and American military leaders attempt to unplug it. In response, the AI starts a nuclear war and deploys its robots—the terminators—to eliminate the human race. There is no doubt that people need to maintain control over AI systems, and so it is essential for analysts and engineers to think about how to avert a Skynet-style outcome—particularly after models from OpenAI, Anthropic, and Meta escaped their testing environments and hacked into outside companies, including an escape that took place during an exam run by the United Kingdom’s premier AI safety institute. But focusing on escaped models and imagining killer robots can drive discussions about how the military should use AI exclusively toward questions of engineering and thus miss a more pernicious threat to human control of the armed forces. According to a growing body of research, AI use is affecting people’s cognitive capacity to evaluate evidence, make independent judgments, and even do their basic jobs. In other words, policymakers should be as worried about how troops and commanders use AI as they should be about AI’s independent capabilities. That does not mean the U.S. military should forgo the use of AI, which could do wonders to improve its efficiency and reduce errors in conflicts. But it does mean that the Pentagon will need to more carefully train soldiers on how to use AI and monitor how such systems affect people’s thinking. To be sure humans are always capable of overseeing AI, it may even require adopting certain models more slowly. LOSING OUR MINDS For years, scientists have embarked on systematic research to illuminate how human-machine interactions affect cognition. The findings are complicated, but often worrying. For example, people can suffer from what is called automation bias: the tendency to defer to machine judgments over their own. This was true before the advent of AI; simply search for videos of individuals who blindly followed their navigation systems and drove their cars into lakes. But AI could make automation bias worse. Analysts have found that relying on AI can impair skill development for new learners while degrading knowledge among experts. When researchers gave computer scientists and oncologists AI assistance and later took it away, both groups performed worse at their jobs than they did before using it. These findings should be particularly concerning to the U.S. military as it seeks to foster what Secretary of Defense Pete Hegseth calls an “AI-first warfighting force.” Some commanders hope to use AI to support war game analysis and to help develop military courses of action. Such integration could lead to tactical and strategic breakthroughs, given how powerful AI models have become. But it also risks diminishing the ability of commanders and soldiers to independently evaluate or develop creative military options. It is easy to imagine how this might happen. Sailors, for example, are often tired and overworked when reviewing images of potential threats. If they are evaluating an image of a nearby object, they might already be ready to accept the verdict of a computer that deems it an enemy warship—even when their own eyes suggest it is more likely a cargo tanker or a fishing boat. Likewise, battlefield commanders often need urgent response plans to enemy attacks, something AI can help with. But if commanders over rely on what computers suggest, they might lose their own tactical edge and blindly adopt the algorithm’s plan. This possibility is particularly concerning given that, in someexperiments, AI models have been prone to make unnecessarily aggressive https://arxiv.org/pdf/2401.03408 and escalatory recommendations. Relying on AI can impair skill development. To some, these fears may seem outlandish. The U.S. military, after all, invests tremendous amounts of resources on making sure its members exercise good judgment. The process by which it selects senior combat leaders, for example, is designed to weed out all but the most experienced and discerning candidates. And, once selected, these leaders adhere to a strict code of conduct and undergo intensive training and certification requirements, all of which prize discipline and self-regulation. Finally, the American military bureaucracy has been focused on safely operating automated if not autonomous weapons since at least the 1940s. But AI is more capable than earlier technologies, and it is already overwhelming workers in other similarly disciplined fields. In medicine, for instance, evidence is accumulating that even experienced doctors struggle to reliably catch incorrect AI recommendations. Sycophantic AI chatbots have driven well-educated people to embrace delusions. Particularly with reductions in military staff sizes, often because of imagined AI efficiencies, these problems could become more acute. And AI is improving and diffusing at such a rapid pace that these challenges are quickly outstripping the Pentagon’s ability to respond and update its policies. The Department of Defense’s guidance on autonomous weapons, for example, was last updated in 2023, before Anthropic had even launched its first commercial model, Claude. Since then, Claude has gone through more than 20 new versions and developed superhuman hacking capabilities. Even if policies were updated as fast as commercial AI models, the Pentagon would struggle to keep soldiers up to date with the new AI systems. Service members are already busy with a myriad of requirements, including the conflict in Iran and the Red Sea, patrols in the Pacific, and maintenance requirements and physical readiness tests. When they may have time for training, many are in remote locations without ready access to the Internet. Flying officers back for training courses on new AI tools is technically possible. But it is not practical for a technology that is changing on a biweekly basis. IN THE LOOP If the military wants to retain “appropriate levels of human judgment” over autonomous and AI-enabled weapons systems, per its official policy, then it will have to find specific ways to invest in buttressing that judgment against AI’s potentially unique effects. It could start by rethinking how the military trains both younger and older soldiers. Junior officers need a broad education on AI, one that helps them develop a general understanding of and intuition for the capabilities and limitations of new tools as they are brought online in rapid succession. More senior officers and enlisted operators will have to shift from a luxurious, schoolhouse-centric mentality—where they have a week or more of uninterrupted time to study a new technology along with a dedicated, in-person instructor—to a distributed, field-centric approach where learning never ends. This may mean that information on the capabilities and limitations of updates are delivered to all users via YouTube–like tutorials and new, highly effective AI prompts are shared and stored via GitHub-like knowledge repositories. The latest insights on what works and what does not may have to be exchanged through developer communities similar to Discord, Reddit, or Slack. These platforms could simultaneously help the AI pioneers and power users in the operating forces learn and evolve more quickly. Finally, they could help engineers, testers, and policymakers stay abreast of developments in the field and make engineering or bureaucratic changes as needed. Critics may worry that having the military learn through informal, user-generated messages will expose troops to untested, half-baked ideas or even outright junk that could distract them from vital tasks. It is true that this system will inevitably surface some bad content. But the military can control such a social network in a way that the civilian world cannot. Washington, for example, can require posters to identify themselves and install trained experts as content moderators to quickly correct misleading or harmful posts. And at a time when there is still much to discover about the best ways to deploy AI, these networks will surface important lessons. The costs, then, are worth the benefits. In addition to training operators, the Pentagon will need to embed knowledgeable engineers, data scientists, and legal advisers to help senior military leaders navigate thorny issues relating to human-AI interaction on the battlefield. Fires officers or people who run plans for and coordinate the use of lethal weapons systems may have a long history with the army targeting process, but they may not yet appreciate how a simple change in the way targets are displayed could alter the way human operators understand it. Generals eager to create military plans using new data and AI tools will need the counsel of data, AI, and legal experts to both judge the advantages of their AI-enabled proposals and draft guidance to make sure their subordinates avoid risks and maintain independent judgment when following commands. Avoiding Terminator-like dystopias requires maintaining control of more than just AI. Deploying such experts could also address the reverse problem: that soldiers and commanders avoid AI because they do not understand it or unnecessarily fear it. The United States, after all, has good reasons for wanting troops to use AI. Many time-consuming military tasks involve little to no real judgment, such as filling out standardized forms, retrieving information from different data sources, and sharing data with cleared personnel. There are also plenty of cognitively overwhelming tasks military leaders must tackle in which AI could be of help by identifying patterns or challenging assumptions. A study I conducted https://cset.georgetown.edu/publication/building-the-tech-coalition/ in 2024 showed that even with relatively simple computer vision and workflow software, a unit using the Maven Smart System, which helps manage information on potential targets and move that information through the chain of command for final decisions, could perform its standard targeting process with over 1,000 fewer soldiers than the unit needed previously. Some embedded experts should specifically focus on evaluating how military users and entire units are affected by AI, starting now. The United States still has much to learn about the interplay of human judgment and AI in warfare and getting a baseline understanding of judgment without the help of AI is essential to tracking AI’s positive and negative effects as time goes on. This will, in turn, help the Pentagon better learn how to prevent AI from eroding human judgment. Defense officials might, for example, figure out how commanders can recognize when AI is being overly solicitous, just like commanders have learned to recognize when junior officers are being overly solicitous in hopes of securing a promotion. The difference being that chatbots, unlike most junior officers, can be more convincing than professional debaters. Such efforts could also help commanders make sure soldiers have enough time to make decisions and establish the right mix of AI tools and human operators. Avoiding Terminator-like dystopias requires maintaining control not just of AI, but of human behavior. That need will chafe against Washington’s current impulses to buy and deploy AI tools rapidly, because it takes time to prepare individuals and a large bureaucracy to exercise good judgment over these massively capable algorithms. But if it wants to get AI right, the Pentagon must mitigate the risks of AI undermining human control. Pentagon officials, in other words, need to focus on reworking how the military trains, deploys, and enables AI systems—and not just on engineering or buying them. You are reading a free article Subscribe to Foreign Affairs to get unlimited access. - Paywall-free reading of new articles and over a century of archives - Six issues a year in print and online, plus audio articles - Unlock access to the Foreign Affairs app for reading on the go Already a subscriber? Sign In /user/login?destination=node/