(1/3) The Dangers of LLMs

A mini-series on LLM dangers warns that real-time deepfakes are already causing over a billion dollars in losses, with CIA Director John Ratcliffe calling frontier AI 'akin to digital nuclear weapons' and the NSA issuing a public statement on the threat. The author argues that LLMs will entrench existing power structures while enabling cheaper dissent suppression and more destructive attacks by smaller groups.

A mini-series, continuing my efforts to document the AI landscape as of July 2026 The industrial revolution gave us machine guns, and eventually we added the atom bomb to our arsenal. Right now we're facing the threat of "LLMs as machine guns." Machine guns do a great job of entrenching existing interests. It still takes economic and industrial power to field these weapons, so the biggest players retain their edge. Governments and other major organizations can still generally field more than the average individual, however. That means large organizations cement their power, and it becomes even cheaper to silence dissent. The first advantage goes to whoever gets the technology first: that's almost always going to be the biggest existing players. The second advantage goes to whoever has the raw wealth and manufacturing capabilities to exploit this, i.e. the biggest existing players. And third, dispersal eventually grants violent individuals and smaller groups significantly more destructive power, without much corresponding defense. A handful of machine guns lets you do a destructive attack on a facility, but it's still not enough to fend off a tank. A tank still isn't enough to fend off an airstrike. Airstrikes are nothing compared to ICBMs. This one isn't even slightly hypothetical: Currently LLMs can already produce real-time deep-fakes. I'd recommend checking out a couple of examples here Video deep-fake of Elon Musk https://x.com/heynavtoor/status/2037158617519186144 Gender Swapping https://x.com/zarqXBT/status/2075930873258516583 . Seeing it for yourself really sums it up better than I ever could - and again, this is real time. You can join a video chat with these, and most people aren't even aware that's possible yet. These scams are already a problem today source https://fortune.com/2025/12/27/2026-deepfakes-outlook-forecast/ , with estimates already placing the damage at more than a billion dollars. Right now scams are mostly still following the same old scripts, and simply faking the voice. But LLMs can adapt dynamically. They can make small talk to build trust, respond to objections, and modulate their tone to match the mood. This gets even more concerning if LLMs improve at "cold reading" detailed introduction https://gwern.net/doc/www/www.greaterwrong.com/da31ddd923084bc2655a17b8c2606acc78214034.html or persuasion see below . This is the one area where defense really shines, and Glasswing highlights that: thousands of security vulnerabilities got fixed before they could be exploited, but only by tightly controlling who had early access. But it's important not to write this off. Y2K was a big nothing as well, for exactly the same reasons: we fixed a bunch of bugs, and those fixes give us receipts for exactly how bad things could have gone. I want to emphasize that major security organizations are waking up to this risk: In rare public remarks, CIA Director John Ratcliffe announces trio of internal changes he says amounts to the “fundamental reshaping of the CIA’s entire approach to technology.” Also says it’s not “misplaced” to refer to frontier AI as “akin to digital nuclear weapons.” The NSA is also worried, and has released a public statement source https://www.nsa.gov/Press-Room/News-Highlights/Article/Article/4523810/five-eyes-cyber-security-agencies-statement/ You don't often get the CIA and the NSA worrying about a private company threatening national security simply by building a good product. The rest of these capabilities are currently "hypothetical": we've seen the edges of them, and it would be surprising if they take more than a year or two to develop into major problems. I expect we will see at least one major crisis within the next year, and a lot of concern about at least some of these capabilities during the 2028 election. If none of these are salient by the election, it's probably a good omen for slow timelines, and lower safety risks. Conversely, a crisis in any of these domains suggests we might also see rapid development in other dangerous domains. This post is a discussion of "LLMs as machine guns", i.e. what the short-term risks look like to me as of July 2026. In future posts I'll discuss more advanced threats - this is meant to represent the minimum set of problems, ones we would expect even if AI is a "normal technology". Labs are already taking a number of precautions around Chemical, Biological, Radiological, and Nuclear CBRN capabilities. I don't think this is security theater. Future "CBRN-3" capabilities would mean a model can significantly help violent individuals navigate these fields without needing technical expertise. Existing models are already at CBRN-2. I want to emphasize that these capabilities are dangerous primarily for their ability to fuel violent individuals. The US already has nuclear missiles and so forth, so CBRN-3 doesn't actually advance their capabilities at all. Nation-states have the tools to police CBRN-3 threats, but terrorists still have a huge window to inflict huge casualties. One of the more central LLM capabilities is their ability to parse and search through huge quantities of text and speech. We're already seeing them develop an eerie ability to identify individuals using both expected and unexpected data - Jack has discovered: "not only can Fable identify me from my writing, it can identify me from my reading." source https://x.com/tracewoodgrains/status/2074442842823618747 Individuals have very little ability to enact a surveillance state of their own. The people who stand to benefit the most are the ones that already have the power and resources to collect the information, and just need the price of analysis to drop. The state has a huge advantage, since they often already control communications infrastructure. On their own, drones are a bit unrelated to LLMs - they mostly use custom neural nets, dedicated hardware, and so on. But that's expensive and hard to get your hands on. Conversely, if LLMs can learn to play video games, they can probably also be used as a cheap, readily available drone pilot. Outside of mere piloting, drone warfare amplifies a lot of the other risks - a rogue agent with a well piloted drone can drastically expand the threat range of newly developed CBRN capabilities. The surveillance state can easily escalate into using drone enforcers. And of course, unlike human pilots, a locally-hosted open source model can't quit, and Pliny has repeatedly demonstrated that the morality core snaps off quite easily. With the right breakthroughs, an individual could deploy LLM "officers" supervising a swarm of drones - thus taking humans even further out of the loop. A recent study found AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses. The AI’s advantage was tied to its ability to rapidly communicate a large number of claims that can easily be fact-checked. After coaching, expert humans performed evenly against an AI constrained to respond at human speeds and with human-length messages. Without that constraint, experts still underperform even after practicing against the AI for a few hours before their formal rematch For the full paper: https://arxiv.org/html/2606.16475 https://arxiv.org/html/2606.16475 I want to draw attention to two big advantages AI has: Patience - An AI will happily continue a conversation for 50 turns across a month of back-and-forth. It will address every single issue and objection you raise, and it can provide citations for all of those. But it is also emotionally patient: it will never get frustrated, or think you're dumb, or mind revisiting a previous argument. You can swear at it all you like, and it will respond in whatever ways the designer thought were most engaging. Quantity - Once you get a Persuasive AI, you don't have to pick and choose who to target it at. You can fire up a horde of instances and have it be persuasive on every level: writing academic papers, swaying social media, targeted campaigns at key journalists and politicians, etc. The other major domain that was hotly contested is the ability for AI to predict events better than a prediction market or a human "super-forecaster". The general consensus here is that AI seems to be about on par with the former milestone. If linear progress continues, it's expected to pass the second milestone within a year - quite possibly within the next six months. For much greater detail: https://www.astralcodexten.com/p/the-ai-superforecasters-are-here https://www.astralcodexten.com/p/the-ai-superforecasters-are-here This one is especially scary if it stacks with Persuasion, since it would presumably then mean the AI is very good at forecasting what strategies will be successful on a tailored per-target level, and forecasting which audiences and individuals are most important to sway. It also introduces a nasty question: did the AI predict any given event? Or was that prediction a manipulation to ensure it happened? There's an important theme here: Even when individuals are empowered, it's primarily destructive rather than defensive. Starting a plague is easier than curing it. Hacking is easier than security. Truth drowns in a sea of artificial propaganda. In these areas, the benefits overwhelmingly accrue to bad actors, at the expense of everyone else. The exception proves the rule: the one place where defenders gained an advantage, cyber-security, was primarily because the defenders had early access - and that's only possible via centralized control. The question isn't whether we trust the government with these powers - they can take them if they want. It might be killing the goose, but they still get whatever golden eggs have already been laid. The question is whether we want these in the hands of dictators and terrorists. The Fable take-down source https://www.anthropic.com/news/fable-mythos-access established that we can quickly take models offline even when there's disagreement about the risks. Coordinating that across multiple national jurisdictions will get much trickier. Open-weight models let the genie out of the bottle irrevocably - they continue to live on a thousand hackers' laptops even after the downloads are taken offline. If you're opposed to centralization, I think you have an uphill political battle in front of you. The status quo suits all the most powerful players, and that means stronger and more credible opposition to decentralization efforts. They have every reason to oppose making those dangerous capabilities more widely available, and "safety" is a popular rallying cry. Part of the problem here is that currently a lot more people are "LLM pilled", but still not "AGI pilled", and we're still quite far from "ASI pilled" being mainstream. Short-term, politics is going to be focused on issues with clear studies and evidence for the concerns, not hypothetical extrapolations. Similarly, we're finally starting to see the risk profile of "LLMs" - a lot of people saw the broad picture coming, but we now have a very specific sense of which domains have become threats https://www.lesswrong.com/posts/ZuBb7Rjgajssasozr/the-llm-revolution-so-far and where they still struggle https://www.lesswrong.com/posts/ELYgXKWJaZdxbEYNm/current-limitations-of-llms compared to humans. Of course, all of this changes when we start to see The Dangers of AGI - but that's for tomorrow. P.S. For all that I think the evidence points this way, I'm also the sort of person who uses Ubuntu, Firefox, and LibreOffice. But I wrote this on a Windows laptop. Tradeoffs.