{"slug": "1-3-the-dangers-of-llms", "title": "(1/3) The Dangers of LLMs", "summary": "A mini-series on LLM dangers warns that real-time deepfakes are already causing over a billion dollars in losses, with CIA Director John Ratcliffe calling frontier AI 'akin to digital nuclear weapons' and the NSA issuing a public statement on the threat. The author argues that LLMs will entrench existing power structures while enabling cheaper dissent suppression and more destructive attacks by smaller groups.", "body_md": "A mini-series, continuing my efforts to document the AI landscape as of July 2026\n\nThe industrial revolution gave us machine guns, and eventually we added the atom bomb to our arsenal. Right now we're facing the threat of \"LLMs as machine guns.\"\n\nMachine guns do a great job of entrenching existing interests. It still takes economic and industrial power to field these weapons, so the biggest players retain their edge. Governments and other major organizations can still generally field more than the average individual, however. That means large organizations cement their power, and it becomes even cheaper to silence dissent.\n\nThe first advantage goes to whoever gets the technology first: that's almost always going to be the biggest existing players. The second advantage goes to whoever has the raw wealth and manufacturing capabilities to exploit this, i.e. the biggest existing players. And third, dispersal eventually grants violent individuals and smaller groups significantly more destructive power, without much corresponding defense.\n\nA handful of machine guns lets you do a destructive attack on a facility, but it's still not enough to fend off a tank. A tank still isn't enough to fend off an airstrike. Airstrikes are nothing compared to ICBMs.\n\nThis one isn't even slightly hypothetical:\n\nCurrently LLMs can already produce real-time deep-fakes. I'd recommend checking out a couple of examples here ([Video deep-fake of Elon Musk](https://x.com/heynavtoor/status/2037158617519186144)) ([Gender Swapping](https://x.com/zarqXBT/status/2075930873258516583)). Seeing it for yourself really sums it up better than I ever could - and again, this is real time. You can join a video chat with these, and most people aren't even aware that's possible yet. These scams are already a problem today ([source](https://fortune.com/2025/12/27/2026-deepfakes-outlook-forecast/)), with estimates already placing the damage at more than a billion dollars.\n\nRight now scams are mostly still following the same old scripts, and simply faking the voice. But LLMs can adapt dynamically. They can make small talk to build trust, respond to objections, and modulate their tone to match the mood.\n\nThis gets even more concerning if LLMs improve at \"cold reading\" ([detailed introduction](https://gwern.net/doc/www/www.greaterwrong.com/da31ddd923084bc2655a17b8c2606acc78214034.html)) or persuasion (see below).\n\nThis is the one area where defense really shines, and Glasswing highlights that: thousands of security vulnerabilities got fixed before they could be exploited, but only by tightly controlling who had early access.\n\nBut it's important not to write this off. Y2K was a big nothing as well, for exactly the same reasons: we fixed a bunch of bugs, and those fixes give us receipts for exactly how bad things could have gone.\n\nI want to emphasize that major security organizations are waking up to this risk:\n\nIn rare public remarks, CIA Director John Ratcliffe announces trio of internal changes he says amounts to the “fundamental reshaping of the CIA’s entire approach to technology.”\n\nAlso says it’s not “misplaced” to refer to frontier AI as “akin to digital nuclear weapons.”\n\nThe NSA is also worried, and has released a public statement ([source](https://www.nsa.gov/Press-Room/News-Highlights/Article/Article/4523810/five-eyes-cyber-security-agencies-statement/))\n\nYou don't often get the CIA and the NSA worrying about a private company threatening national security simply by building a good product.\n\nThe rest of these capabilities are currently \"hypothetical\": we've seen the edges of them, and it would be surprising if they take more than a year or two to develop into major problems. I expect we will see at least one major crisis within the next year, and a lot of concern about at least some of these capabilities during the 2028 election.\n\nIf none of these are salient by the election, it's probably a good omen for slow timelines, and lower safety risks. Conversely, a crisis in any of these domains suggests we might also see rapid development in other dangerous domains.\n\nThis post is a discussion of \"LLMs as machine guns\", i.e. what the short-term risks look like to me as of July 2026. In future posts I'll discuss more advanced threats - this is meant to represent the *minimum* set of problems, ones we would expect even if AI is a \"normal technology\".\n\nLabs are already taking a number of precautions around Chemical, Biological, Radiological, and Nuclear (CBRN) capabilities. I don't think this is security theater. Future \"CBRN-3\" capabilities would mean a model can significantly help violent individuals navigate these fields without needing technical expertise. Existing models are already at CBRN-2.\n\nI want to emphasize that these capabilities are dangerous primarily for their ability to fuel violent individuals. The US already has nuclear missiles and so forth, so CBRN-3 doesn't actually advance their capabilities at all. Nation-states have the tools to police CBRN-3 threats, but terrorists still have a huge window to inflict huge casualties.\n\nOne of the more central LLM capabilities is their ability to parse and search through huge quantities of text and speech. We're already seeing them develop an eerie ability to identify individuals using both expected and unexpected data - Jack has discovered: *\"not only can Fable identify me from my writing, it can identify me from my reading.\" *([source](https://x.com/tracewoodgrains/status/2074442842823618747))\n\nIndividuals have very little ability to enact a surveillance state of their own. The people who stand to benefit the most are the ones that already have the power and resources to collect the information, and just need the price of analysis to drop. The state has a huge advantage, since they often already control communications infrastructure.\n\nOn their own, drones are a bit unrelated to LLMs - they mostly use custom neural nets, dedicated hardware, and so on. But that's expensive and hard to get your hands on. Conversely, if LLMs can learn to play video games, they can probably also be used as a cheap, readily available drone pilot.\n\nOutside of mere piloting, drone warfare amplifies a lot of the other risks - a rogue agent with a well piloted drone can drastically expand the threat range of newly developed CBRN capabilities. The surveillance state can easily escalate into using drone enforcers.\n\nAnd of course, unlike human pilots, a locally-hosted open source model can't quit, and Pliny has repeatedly demonstrated that the morality core snaps off quite easily. With the right breakthroughs, an individual could deploy LLM \"officers\" supervising a swarm of drones - thus taking humans even further out of the loop.\n\nA recent study found AI systems were reliably more persuasive than expert humans, even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses.\n\nThe AI’s advantage was tied to its ability to rapidly communicate a large number of claims that can easily be fact-checked. After coaching, expert humans performed evenly against an AI constrained to respond at human speeds and with human-length messages.\n\nWithout that constraint, experts still underperform even after practicing against the AI for a few hours before their formal rematch\n\nFor the full paper: [https://arxiv.org/html/2606.16475](https://arxiv.org/html/2606.16475)\n\nI want to draw attention to two big advantages AI has:\n\n**Patience **- An AI will happily continue a conversation for 50 turns across a month of back-and-forth. It will address every single issue and objection you raise, and it can provide citations for all of those. But it is also emotionally patient: it will never get frustrated, or think you're dumb, or mind revisiting a previous argument. You can swear at it all you like, and it will respond in whatever ways the designer thought were most engaging.\n\n**Quantity** - Once you get a Persuasive AI, you don't have to pick and choose who to target it at. You can fire up a horde of instances and have it be persuasive on every level: writing academic papers, swaying social media, targeted campaigns at key journalists and politicians, etc.\n\nThe other major domain that was hotly contested is the ability for AI to predict events better than a prediction market or a human \"super-forecaster\". The general consensus here is that AI seems to be about on par with the former milestone. If linear progress continues, it's expected to pass the second milestone within a year - quite possibly within the next six months.\n\nFor much greater detail: [https://www.astralcodexten.com/p/the-ai-superforecasters-are-here](https://www.astralcodexten.com/p/the-ai-superforecasters-are-here)\n\nThis one is especially scary if it stacks with Persuasion, since it would presumably then mean the AI is very good at forecasting what strategies will be successful on a tailored per-target level, and forecasting which audiences and individuals are most important to sway.\n\nIt also introduces a nasty question: did the AI predict any given event? Or was that prediction a manipulation to ensure it happened?\n\nThere's an important theme here: **Even when individuals are empowered, it's primarily destructive rather than defensive.** Starting a plague is easier than curing it. Hacking is easier than security. Truth drowns in a sea of artificial propaganda. In these areas, the benefits overwhelmingly accrue to bad actors, at the expense of everyone else.\n\nThe exception proves the rule: the one place where defenders gained an advantage, cyber-security, was primarily because the defenders had early access - and that's only possible via centralized control.\n\nThe question isn't whether we trust the government with these powers - they can take them if they want. It might be killing the goose, but they still get whatever golden eggs have already been laid.\n\nThe question is whether we want these in the hands of dictators and terrorists. The Fable take-down ([source](https://www.anthropic.com/news/fable-mythos-access)) established that we *can* quickly take models offline even when there's disagreement about the risks. Coordinating that across multiple national jurisdictions will get much trickier. Open-weight models let the genie out of the bottle irrevocably - they continue to live on a thousand hackers' laptops even after the downloads are taken offline.\n\nIf you're opposed to centralization, I think you have an uphill political battle in front of you. The status quo suits all the most powerful players, and that means stronger and more credible opposition to decentralization efforts. They have every reason to oppose making those dangerous capabilities more widely available, and \"safety\" is a popular rallying cry.\n\nPart of the problem here is that currently a lot more people are \"LLM pilled\", but still not \"AGI pilled\", and we're still quite far from \"ASI pilled\" being mainstream. Short-term, politics is going to be focused on issues with clear studies and evidence for the concerns, not hypothetical extrapolations.\n\nSimilarly, we're finally starting to see the risk profile of \"LLMs\" - a lot of people saw the broad picture coming, but we now have a very specific sense of [which domains have become threats](https://www.lesswrong.com/posts/ZuBb7Rjgajssasozr/the-llm-revolution-so-far) and [where they still struggle](https://www.lesswrong.com/posts/ELYgXKWJaZdxbEYNm/current-limitations-of-llms) compared to humans.\n\nOf course, all of this changes when we start to see The Dangers of AGI - but that's for tomorrow.**P.S.** For all that I think the evidence points this way, I'm also the sort of person who uses Ubuntu, Firefox, and LibreOffice. But I wrote this on a Windows laptop. Tradeoffs.", "url": "https://wpnews.pro/news/1-3-the-dangers-of-llms", "canonical_source": "https://www.lesswrong.com/posts/tcNExzquru6or6R8H/1-3-the-dangers-of-llms", "published_at": "2026-07-21 12:01:47+00:00", "updated_at": "2026-07-21 12:30:28.892480+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-policy"], "entities": ["CIA", "John Ratcliffe", "NSA", "Elon Musk", "Glasswing"], "alternates": {"html": "https://wpnews.pro/news/1-3-the-dangers-of-llms", "markdown": "https://wpnews.pro/news/1-3-the-dangers-of-llms.md", "text": "https://wpnews.pro/news/1-3-the-dangers-of-llms.txt", "jsonld": "https://wpnews.pro/news/1-3-the-dangers-of-llms.jsonld"}}